feat(tts): Inflect v2 (Micro/Nano) CoreML backend - #823
Conversation
Ultra-tiny VITS-family English TTS (9.4M / 4.0M params, 24 kHz). Ships FluidInference/inflect-v2-coreml: a fixed-shape encoder + duration predictor and 8 frame-bucketed HiFiGAN synthesizers per variant. Everything stochastic or dynamically shaped runs host-side in Swift (duration expansion, prior sampling), so both CoreML graphs are deterministic. - InflectManager (actor): text + ipa entry points, phoneme chunking, inter-chunk pauses, edge fade. - InflectSymbols: keithito/espeak symbol table + token encoder. Iterates Unicode scalars (a syllabic n̩ is two symbols, not one grapheme) and reproduces intersperse(cleaned_text_to_sequence(...)); golden-vector tested against the upstream Python. - InflectSynthesizer: encoder -> host expand+sample z_p -> smallest fitting bucket -> trim. InflectNoise: seedable Box-Muller Gaussian. - English frontend shared with StyleTTS2 (Misaki lexicon + BART G2P, espeak-approximated); synthesize(ipa:) bypasses it for faithful espeak input. espeak-ng itself (GPL) is not vendored. Validated end-to-end on M5 Pro: micro IPA and nano text both round-trip through Parakeet to the input text. Unit tests cover the symbol encoder (golden vectors), noise determinism, and repo/model wiring.
PocketTTS Smoke Test ✅
Runtime: 1m42s Note: PocketTTS uses CoreML MLState (macOS 15) KV cache + Mimi streaming state. CI VM lacks physical GPU — audio quality and performance may differ from Apple Silicon. |
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 35.0s diarization time • Test runtime: 2m 29s • 07/25/2026, 06:41 PM EST |
Sortformer High-Latency Benchmark ResultsES2004a Performance (30.4s latency config)
Sortformer High-Latency • ES2004a • Runtime: 2m 51s • 2026-07-25T22:42:32.572Z |
Supertonic3 Smoke Test ✅
Runtime: 0m27s Note: CI VMs lack a physical Neural Engine; the ANE-bucketed VectorEstimator falls back to CPU here. This validates download + variant resolution + synthesis, not ANE residency/perf. |
Offline VBx Pipeline ResultsSpeaker Diarization Performance (VBx Batch Mode)Optimal clustering with Hungarian algorithm for maximum accuracy
Offline VBx Pipeline Timing BreakdownTime spent in each stage of batch diarization
Speaker Diarization Research ComparisonOffline VBx achieves competitive accuracy with batch processing
Pipeline Details:
🎯 Offline VBx Test • AMI Corpus ES2004a • 1049.0s meeting audio • 131.2s processing • Test runtime: 2m 25s • 07/25/2026, 06:44 PM EST |
Parakeet EOU Benchmark Results ✅Status: Benchmark passed Performance Metrics
Streaming Metrics
Test runtime: 1m47s • 07/25/2026, 06:48 PM EST RTFx = Real-Time Factor (higher is better) • Processing includes: Model inference, audio preprocessing, state management, and file I/O |
VAD Benchmark ResultsPerformance Comparison
Dataset Details
✅: Average F1-Score above 70% |
ASR Benchmark Results ✅Status: All benchmarks passed Parakeet v3 (multilingual)
Parakeet v2 (English-optimized)
Streaming (v3)
Streaming (v2)
Streaming tests use 5 files with 0.5s chunks to simulate real-time audio streaming 25 files per dataset • Test runtime: 11m13s • 07/25/2026, 07:00 PM EST RTFx = Real-Time Factor (higher is better) • Calculated as: Total audio duration ÷ Total processing time Expected RTFx Performance on Physical M1 Hardware:• M1 Mac: ~28x (clean), ~25x (other) Testing methodology follows HuggingFace Open ASR Leaderboard |
1 participant
Summary
Adds Inflect v2 as a FluidAudio TTS backend — ultra-tiny VITS-family English TTS (Micro 9.4M / Nano 4.0M params, 24 kHz mono). Companion to the mobius conversion (mobius #80); models published at FluidInference/inflect-v2-coreml.
Design
Two-stage fixed-shape CoreML split with everything stochastic or dynamically shaped on the host (so both graphs are deterministic):
encoder— TextEncoder + duration predictor (fixed 512-token axis).z_p = m + noise·exp(logs)·noise_scale, smallest-fitting frame bucket.synthesizer_f<N>— reverse coupling flow + HiFiGAN, 8 frame buckets {256…2048}; audio trimmed to exact length.New module
TTS/Inflect/:InflectManager(actor, text +ipa:paths, chunking, pauses, edge fade),InflectSynthesizer,InflectModelStore(lazy bucket loading),InflectSymbols(keithito/espeak symbol table + encoder),InflectNoise(seedable Box–Muller).Phonemizer (same tradeoff as StyleTTS2)
Inflect trained on espeak-ng IPA. espeak-ng is GPL-3.0 and can't ship in an Apache framework, so the text path reuses the shared Misaki-lexicon + BART G2P frontend (
StyleTTS2Phonemizer), which approximates espeak IPA. Callers with real espeak feed IPA directly viasynthesize(ipa:)for faithful output. No new dependencies; stays Apache-2.0.Validation
End-to-end on M5 Pro (real HF download → encoder → host expansion → synthesizer), Parakeet round-trip:
--phonemes(faithful IPA):"The quick brown fox … over the lazy dog.""Hello from the tiny inflect model."— exact.Benchmarks (mobius harness, MiniMax-English): micro 25.7 ms p50 / 245× RTFx / WER 1.43%; nano 13.2 ms / 460× / 1.92%.
Unit tests: symbol encoder golden vectors (incl. syllabic-consonant scalar handling), noise determinism + distribution, repo/model wiring. (Local XCTest is unavailable here; tests run in CI.)
CLI