Real time on Apple Silicon (MLX): first audio ~140 ms plain / ~210 ms with a voice direction on an M4 Pro
Sharing in case anyone else is trying to stream Breeze-TTS-2 on a Mac.
Starting from the MLX backend in northcraftfoundries' streaming server (itself built on mlx-audio's port), we got it streaming comfortably faster than real time on an M4 Pro, with directed lines as fast as plain ones and quality closer to bf16 than the community 8-bit build:
| Build (M4 Pro, 48 GB) | First audio, plain / directed | RTF | Memory |
|---|---|---|---|
| community 8-bit (mxfp8) | 212 / 297 ms | 0.58 | 13 GB |
| bf16 | 380 / 456 ms | 0.86 | 15 GB |
| mixed: bf16 backbone + int8 depth decoder, plus the changes below | 139 / 207 ms | 0.62 | 8.9 GB |
(24 lines, two in three with a voice direction at CFG 2; "first audio" includes any buffer needed so playback never stalls.)
What made the difference:
Quantize only the depth decoder. On Apple Silicon the frame loop is memory-bound: each frame reads the backbone once and the depth decoder 15 times, so the depth decoder is ~3/4 of the bytes. Keeping the backbone in bf16 matters for voice directions: CFG multiplies the backbone's quantization error (mxfp8's backbone KL is ~3× higher on directed lines than plain ones), and in a blind listening test the 8-bit build lost on 7 of 10 lines while bf16 and the mixed build tied.
A streaming bug in mlx-audio's Qwen3-TTS codec decoder, which Breeze uses: the transposed-conv bias was added twice at every chunk boundary, so streamed audio depended on the chunk size (up to 0.2 of full scale in an utterance's first frames at 2-frame chunks). Fixed upstream: https://github.com/Blaizzy/mlx-audio/pull/1003
First frames sent one by one (they used to wait for three frames), each frame compiled with
mx.compile, and a cap on MLX's buffer cache.Code, benchmark and quality tool (KL against bf16 on the same tokens): https://github.com/Shoofio/breeze-tts2-fast-streaming-api (
scripts/start_breeze_mac.sh --precision mixed).Or the ready-made checkpoint (same model, verified tensor-for-tensor): https://huggingface.co/Shoof/Gust-MLX
Found and built with Claude Opus 5.5. Happy to compare notes if you're doing the same.
Added two reference points, measured with the same voice, lines and metrics:
- Starting point: BreezeBlue's own PyTorch code (bf16,
model.generate) on the same M4 Pro's GPU: RTF 2.26 on plain lines, more than twice slower than real time, and no streaming (their streaming runtime is CUDA-only), so a one-sentence line takes ~6 s before you hear anything. 11 GB. - For scale: a power-capped RTX 3090 (250 W of 350 W) running BreezeBlue's CUDA fast path measured 87 / 160 ms first audio (plain / directed), RTF 0.50, in our setup (through a small server wrapper that re-processes the reference on every line).
So on the Mac: from RTF 2.26 and no streaming, to RTF 0.62 with first audio 139 / 206 ms. Both rows are now in the tables on the repo and on the model card.
Added side-by-side samples: 4 public voices (2 male, 2 female, LibriSpeech, CC BY 4.0), each with a different voice direction, RTX 3090 running BreezeBlue's code vs this build on an M4 Pro, same text, direction and seed: https://huggingface.co/Shoof/Gust-MLX#samples
Thanks for sharing this and for your work on bringing Breeze-TTS-2 to Apple Silicon! We really appreciate the optimizations, the upstream codec fix, and the side-by-side audio samples. Great to see the community making local streaming on Mac more accessible!
Thanks for sharing this and for your work on bringing Breeze-TTS-2 to Apple Silicon! We really appreciate the optimizations, the upstream codec fix, and the side-by-side audio samples. Great to see the community making local streaming on Mac more accessible!
Thank you!
Appreciate the model a lot and all of the work you do, keep up the great work, you guys! ❤️