feat(tts): add NeuTTS-2E backend (emotional English TTS, Qwen3 + NeuCodec)

[Alex-Wengg]

What

New TtsBackend.neuTts — neuphonic/neutts-2e: an autoregressive Qwen3 236M backbone emits NeuCodec speech tokens (50/s, top-k 50 / temp 1.0 sampling, seeded), decoded to 24 kHz audio by the NeuCodec decoder. Seven emotions × four fixed speakers — the only backend with emotion control. Models come from FluidInference/neutts-2e-coreml (converted in mobius, see FluidInference/mobius#79).

swift run fluidaudiocli tts --backend neutts \
  --text "I can't believe it's finally here!" --voice emily --emotion happy

How

• NeuTtsBpeTokenizer — minimal byte-level BPE encoder for the HF tokenizer.json (Qwen2 scheme: split regex read from the JSON, GPT-2 bytes_to_unicode, merge-rank loop). First BPE encoder in the repo; parity-tested against transformers-generated fixtures (NeuTtsTokenizerTests, skipped when the asset isn't cached).
• NeuTtsPrompt — upstream chat-template port: NFKC + curly-quote normalization, <|TEXT_PROMPT_START|>…<|EMOTION|>…<|SPEECH_GENERATION_START|>{ref codes} layout, contiguous speech-token id mapping.
• NeuTtsSynthesizer — prefill (768-token window) → seed 56 per-layer fp16 MLState buffers from the fp32 prefill KV → per-token stateful decode until <|SPEECH_GENERATION_END|> → NeuCodec vocode. Runs on the caller's actor via #isolation; requires macOS 15+/iOS 18+ (MLState).
• Compute placement (measured in mobius): LM on GPU (the ANE compiler rejects the decode graph, ANECCompile -14), codec on .cpuAndNeuralEngine (~2× faster than GPU despite the flexible code-length axis).
• Speaker reference JSONs are fetched individually (the repo walk only descends into required .mlmodelc bundles).

Verification (M5 Pro)

• Three speaker/emotion combos (emily/happy, paul/neutral, sophie/sad) transcribe verbatim via parakeet-tdt-v3.
• 1.67× RT (5.76 s audio in 3.45 s inference) — faster than the Python coremltools host (1.5× at the same M2048 config). Cold model load 4.8 s.
• Underlying CoreML parity chain (vs PyTorch fp32: argmax match, 98.7 % top-50 containment over a 301-token replay, codec 40–47 dB SNR) documented in the mobius project.

Notes

• Perth watermarking is upstream host-side postprocessing and intentionally not implemented; noted in the model card.
• Follow-ups: streaming windowed vocoding (~550 ms TTFA, validated in the Python host), the faster M=1024 model pair for short utterances.

🤖 Generated with Claude Code

[github-actions]

ASR Benchmark Results ⚠️

Status: Some benchmarks failed (see logs)

Parakeet v3 (multilingual)

Dataset	WER Avg	WER Med	RTFx	Status
test-clean	%	%	x	⚠️
test-other	%	%	x	⚠️

Parakeet v2 (English-optimized)

Dataset	WER Avg	WER Med	RTFx	Status
test-clean	%	%	x	⚠️
test-other	%	%	x	⚠️

Streaming (v3)

Metric	Value	Description
WER	%	Word Error Rate in streaming mode
RTFx	x	Streaming real-time factor
Avg Chunk Time	s	Average time to process each chunk
Max Chunk Time	s	Maximum chunk processing time
First Token	s	Latency to first transcription token
Total Chunks		Number of chunks processed

Streaming (v2)

Metric	Value	Description
WER	%	Word Error Rate in streaming mode
RTFx	x	Streaming real-time factor
Avg Chunk Time	s	Average time to process each chunk
Max Chunk Time	s	Maximum chunk processing time
First Token	s	Latency to first transcription token
Total Chunks		Number of chunks processed

_{Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming}

_{files per dataset • Test runtime: • 07/25/2026, 06:32 PM EST}

_{RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time
Processing time includes: Model inference on Apple Neural Engine, audio preprocessing, state resets between files, token-to-text conversion, and file I/O
Example: RTFx of 2.0x means 10 seconds of audio processed in 5 seconds (2x faster than real-time)}

Expected RTFx Performance on Physical M1 Hardware:

• M1 Mac: ~28x (clean), ~25x (other)
• CI shows ~0.5-3x due to virtualization limitations

_{Testing methodology follows HuggingFace Open ASR Leaderboard}

[github-actions]

VAD Benchmark Results

❌ Benchmark failed - no results generated

[github-actions]

Parakeet EOU Benchmark Results ❌

Status: Benchmark failed (see logs)
Chunk Size: ms
Files Tested: /

Performance Metrics

Metric	Value	Description
WER (Avg)	%	Average Word Error Rate
WER (Med)	%	Median Word Error Rate
RTFx	x	Real-time factor (higher = faster)
Total Audio	s	Total audio duration processed
Total Time	s	Total processing time

Streaming Metrics

Metric	Value	Description
Avg Chunk Time	s	Average chunk processing time
Max Chunk Time	s	Maximum chunk processing time
EOU Detections		Total End-of-Utterance detections

_{Test runtime: • 07/25/2026, 06:32 PM EST}

_{RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O}

[github-actions]

Supertonic3 Smoke Test ❌

Check	Result
Build	✅
Model download (incl. VectorEstimatorVariants/ int4 buckets)	❌
Model load	❌
Synthesis pipeline (--ve-variant int4)	❌
Output WAV	❌ (NaN KB)

_Runtime:

_{Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf.}

[github-actions]

PocketTTS Smoke Test ❌

Check	Result
Build	✅
Model download	❌
Model load	❌
Synthesis pipeline	❌
Output WAV	❌ (NaN KB)

_Runtime:

_{Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon.}

[github-actions]

Sortformer High-Latency Benchmark Results

ES2004a Performance (30.4s latency config)

Metric	Value	Target	Status
DER	0.0%	<35%	✅
Miss Rate	0.0%	-	-
False Alarm	0.0%	-	-
Speaker Error	0.0%	-	-
RTFx	0.0x	>1.0x	⚠️
Speakers	0/0	-	-

_{Sortformer High-Latency • ES2004a • Runtime: N/A • 2026-07-25T22:34:47.045Z}

[github-actions]

Offline VBx Pipeline Results

Speaker Diarization Performance (VBx Batch Mode)

Optimal clustering with Hungarian algorithm for maximum accuracy

Metric	Value	Target	Status	Description
DER	NaN%	<20%	⚠️	Diarization Error Rate (lower is better)
RTFx	NaNx	>1.0x	⚠️	Real-Time Factor (higher is faster)

Offline VBx Pipeline Timing Breakdown

Time spent in each stage of batch diarization

Stage	Time (s)	%	Description
Model Download	NaN	NaN	Fetching diarization models
Model Compile	NaN	NaN	CoreML compilation
Audio Load	NaN	NaN	Loading audio file
Segmentation	NaN	NaN	VAD + speech detection
Embedding	NaN	NaN	Speaker embedding extraction
Clustering (VBx)	NaN	NaN	Hungarian algorithm + VBx clustering
Total	NaN	100	Full VBx pipeline

Speaker Diarization Research Comparison

Offline VBx achieves competitive accuracy with batch processing

Method	DER	Mode	Description
FluidAudio (Offline)	NaN%	VBx Batch	On-device CoreML with optimal clustering
FluidAudio (Streaming)	17.7%	Chunk-based	First-occurrence speaker mapping
Research baseline	18-30%	Various	Standard dataset performance

Pipeline Details:

• Mode: Offline VBx with Hungarian algorithm for optimal speaker-to-cluster assignment
• Segmentation: VAD-based voice activity detection
• Embeddings: WeSpeaker-compatible speaker embeddings
• Clustering: PowerSet with VBx refinement
• Accuracy: Higher than streaming due to optimal post-hoc mapping

_{🎯 Offline VBx Test • AMI Corpus ES2004a • NaNs meeting audio • NaNs processing • Test runtime: N/A • 07/25/2026, 06:36 PM EST}