Add NeuTTS-2E CoreML conversion (Qwen3 LM + NeuCodec decoder) - #79
Open
Alex-Wengg wants to merge 5 commits into
Open
Add NeuTTS-2E CoreML conversion (Qwen3 LM + NeuCodec decoder)#79Alex-Wengg wants to merge 5 commits into
Alex-Wengg wants to merge 5 commits into
Conversation
Converts neuphonic/neutts-2e (emotional English TTS, Qwen3 236M backbone) and the NeuCodec FSQ decoder to CoreML: - LM-Prefill (macOS14), LM-Decode pass-through-KV (macOS14) and stateful StateType-KV (macOS15) mlpackages, fp16 with fp32 norm/softmax ops. Tied embedding/LM-head shared through one parameter so const-dedup stores the [217232, 512] vocab matrix once per package. - NeuCodec decoder with flexible code length (RangeDim). FSQ dequant is a precomputed [65536, 8] embedding gather (fp16 pass corrupts integer arithmetic above 2048). Replicates the upstream bs_roformer5 RoPE quirk (torchtune rope applied to [b, h, t, d] rotates by head index, constant over time), which is also what keeps the length axis flexible. Vocos same-padding ISTFT reimplemented as IDFT 1x1 convs + ConvTranspose1d overlap-add. - gen-pytorch-ref.py (seeded PyTorch reference), inference.py (full CoreML synthesis + teacher-forced parity replay). Parity: LM CoreML fp16 max-abs-diff 0.024/0.081 vs torch fp32 with argmax match; stateful decode bit-identical to pass-through. Codec CoreML SNR 40-47 dB vs PyTorch across lengths. Teacher-forced replay keeps 98.7% of reference tokens inside the CoreML top-50 support; final audio 41.5 dB vs the PyTorch waveform. Sampled decode 13 ms/token (~77 tok/s vs 50 needed for real-time), codec 2.7-9.6x RT on M5 Pro. torch pinned <2.8: newer torch traces shape unpacking, repeat_interleave and shape[-1]//2 into aten::Int/floor_divide nodes coremltools 9.0 cannot fold; wrappers use static ints + expand/reshape GQA for the same reason.
Same-sentence benchmark vs FluidAudio engines (fluidaudiocli --metrics, parakeet round-trip WER, M5 Pro): Supertonic-3 44x / KokoroAne 6.3x / PocketTTS 4.8x / NeuTTS-2E 1.06x RTFx; NeuTTS-2E and KokoroAne both 0% WER. Also records the discovery that the hosted StyleTTS2 sized BERT/diffusion mlmodelc bundles are missing model.mil (long texts fail with corruptedModel).
- inference.py --stream: upstream 25-frame windowed overlap-add vocoding over the flexible-length CoreML codec, ref codes as initial lookback. TTFA ~650 ms at every text length, inter-chunk ~340 ms (500 ms budget), 0% WER (identical transcription to batch). - Compute-unit sweep: codec is ~2x faster on CPU_AND_NE (22-27x RT) than GPU despite RangeDim — corrected the docstring claim; LM decode best on GPU (9.2 ms/tok warm; CPU_AND_NE breaks the coremltools state API; CPU_ONLY still real-time). inference.py now defaults LM=ALL, codec=NE. - Multi-length benchmark matrix (5/14/53 words x 3 runs) vs KokoroAne, PocketTTS, Supertonic-3, StyleTTS2; NeuTTS-2E batch 1.3-1.7x RT, stream 1.4-1.5x RT, 0% WER on medium/long incl. the 23s paragraph. - StyleTTS2 corruptedModel root cause: FluidAudio downloader drops model.mil from sized mlmodelc bundles (HF assets are complete); cache patched via curl for the benchmark.
convert-lm.py --max-len 1024 pair: stateful decode 7.0 ms/tok pure (143 tok/s) vs 9.1 at M=2048; end-to-end batch ~2.0x RT, streaming TTFA 554 ms / inter-chunk 303 ms, verbatim ASR. inference.py caps generation at the decode state capacity; compress-lm.py added (int8 halves disk, no speed change). Measured dead-ends recorded in README: int8 no speedup (decode is dispatch-latency-bound, not bandwidth-bound); pure-fp16 RMSNorm overflows (0/50 top-50 overlap); in-model top-k head is net slower and hits a CoreML kernel bug (topk over >2^17-wide intermediate returns indices mod 131072, values correct, fp16+fp32, chunking doesn't help; op is correct on a standalone input-fed model). ANE rejects the decode graph (ANECCompile -14), matching the Qwen3-0.6B finding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Model
neuphonic/neutts-2e — emotional English TTS (7 emotions × 4 fixed speakers): Qwen3ForCausalLM 236M backbone (28L, hidden 512, 12Q/4KV, head_dim 128, vocab 217232 with tied embeddings) emitting NeuCodec FSQ speech tokens, decoded to 24 kHz by the NeuCodec decoder (FSQ dequant → ResNet/transformer vocos backbone → ISTFT head).
Published artifacts: FluidInference/neutts-2e-coreml (6 mlpackages + speaker refs + NeuTTS Open License v1.0).
What's here
models/tts/neutts-2e/coreml/: self-contained uv project —gen-pytorch-ref.py(seeded PyTorch reference),convert-lm.py(prefill + pass-through decode macOS14 + StateType stateful decode macOS15,--max-lenvariants),convert-codec.py(flexible-length codec via RangeDim),compress-lm.py(int8, size-only),inference.py(end-to-end CoreML synthesis,--streamwindowed vocoding,--teacher-forceparity replay),src/wrappers.Validation
neucodec.decode_code(fp32); CoreML fp16 40–47 dB across lengths.Deviations / findings (documented in README)
repeat_interleave/shape[-1]//2into aten::Int nodes coremltools 9.0 cannot fold).[b, h, t, d]→ rotation by head index, constant over time; replicated (this is also what keeps the codec length axis flexible).