A local-first, real-time audio processing engine built with Rust and Tauri.
Sibyl started as a personal project: "How do I build a 1:1 coach that listens to my calls and prompts me with guidance in real-time?" The answer turned out to be a deep dive into audio engineering, model inference, and all the invisible plumbing between a microphone and useful output.
On the surface, "capture audio, send it to a model" sounds straightforward. In practice, there's a surprising amount of engineering required: platform-specific audio capture, voice activity detection, chunking, buffering, dual-stream mixing, mel spectrograms, sentence healing, context windows, garbage filtering — and that's before you even get to the model.
This repository is the result of those hours. It's shared publicly so others can bootstrap from it — whether you're building a conversation coach, speech-to-text for form fields, a transcription tool, or anything that needs local audio → on-device model processing.
- Cross-platform audio capture — microphone via
cpal, system audio via WASAPI (Windows) and ScreenCaptureKit (macOS). No virtual audio cables needed. - Voice Activity Detection — Silero VAD neural model, tuned for natural speech (800ms silence threshold to avoid splitting mid-sentence).
- Audio preprocessing pipeline — AGC normalization, mel spectrograms, 160ms chunk alignment, dual-stream mixing.
- On-device inference — ONNX Runtime with CUDA, DirectML, CoreML, and OpenVINO execution providers. Currently running Gemma 3n E2B.
- Memory and context management — Sentence healing (SegmentBuffer), rolling context windows, in-memory vector store with semantic search.
- A working Tauri app — Rust backend + Next.js frontend, ready to extend.
Everything runs locally. No cloud. No API keys for core functionality.
This section walks through the engineering layers from microphone to useful output. Each layer solves a problem that isn't obvious until you hit it. If you're building something with real-time audio, this is the map of what you'll encounter.
The assumption: Call some API, get audio samples.
The reality: Every platform does it differently. Sample rates vary by device. You need to handle mono/stereo conversion. And if you want to capture what the other person is saying (system audio), that's an entirely separate subsystem.
What Sibyl does:
- Microphone capture via
cpal— cross-platform, but you still need to negotiate sample rates and channel counts - Windows system audio via WASAPI loopback — captures any audio playing through speakers/headphones
- macOS system audio via ScreenCaptureKit — Apple's native API for audio tap
- Everything resampled to 16kHz mono f32 — the format models expect
Code: src-tauri/src/audio/mod.rs (mic), audio/loopback.rs (system audio)
The assumption: Stream all audio continuously; the model will figure out what's speech.
The reality: Sending silence wastes compute. But more importantly, you need to know when someone stops talking to make decisions — commit a transcript segment, trigger analysis, or know when to interject. You also can't split on every tiny pause or you'll fragment sentences mid-thought.
What Sibyl does:
- Silero VAD — a ~2MB neural network that classifies audio frames as speech or silence
- 200ms minimum speech before triggering (filters coughs, clicks)
- 800ms minimum silence before declaring end-of-speech (preserves natural pauses like "I think... maybe we should...")
- VAD events drive the entire downstream pipeline: when to process, when to commit to memory, when to analyze
Code: src-tauri/src/audio/vad.rs
The assumption: Capture everything into one buffer.
The reality: If you're processing a conversation, you need to know who is talking. Traditional diarization (speaker identification) is its own ML problem. But if you capture the mic and system audio as separate streams, you get speaker separation for free — mic is you, system audio is them.
What Sibyl does:
- Dual-stream capture: mic (seller) + loopback (customer) run as independent audio streams
AudioMixersynchronizes them into time-aligned 160ms chunks- Each chunk is tagged with its source — no ML-based diarization needed
- Streams are mixed when needed for the model but kept separate for speaker attribution
Code: src-tauri/src/audio/mixer.rs
The assumption: Feed PCM samples into the model.
The reality: Most audio models don't consume raw waveforms. They expect mel spectrograms — a frequency-domain representation that's closer to how humans perceive sound. You also need to handle quiet audio (some mics output very low amplitude), apply windowing functions, and get the spectrogram parameters exactly right for your model.
What Sibyl does:
- AGC (Automatic Gain Control) — normalizes quiet audio to target RMS of 0.1, up to 10x gain
- Mel spectrogram computation — 128 mel bins, 1024-point FFT, 160-sample hop length (10ms at 16kHz)
- Preemphasis filtering — boosts high frequencies (coefficient 0.97) per Gemma 3n requirements
- All parameters sourced from the model's
preprocessor_config.json— get one wrong and you get garbage output
Code: src-tauri/src/onnx/audio.rs, src-tauri/src/agents/nuance.rs (AGC via normalize_audio())
The assumption: Load model, call predict(), get text.
The reality: Large models (11GB+ for Gemma 3n E2B) need careful loading — you can't just read them into memory at once. You need to select the right execution provider for the hardware (CUDA? DirectML? CoreML?). The model has multiple components (audio encoder, token embedder, decoder) that need to be orchestrated. And autoregressive text generation is a loop, not a single call.
What Sibyl does:
- Hardware detection — auto-detects CPU features, GPU vendor/VRAM, NPU presence, recommends execution provider
- Background model loading —
ModelLoaderAgentloads asynchronously on startup so the UI stays responsive, with progress events (reads in 16MB chunks) - Multi-component pipeline — audio encoder (1.3GB) → token embedder (5.1GB) → decoder (4.8GB), each loaded into separate ONNX sessions
- Autoregressive generation — token-by-token loop with temperature sampling, top-k filtering, and EOS detection
- Feature flags — compile with
--features onnx-cudaoronnx-directmloronnx-coremloronnx-openvino
Code: src-tauri/src/onnx/ (engine, multimodal pipeline), src-tauri/src/hardware/ (detection), src-tauri/src/agents/model_loader.rs (background loading)
The assumption: Model returns a nice transcript.
The reality: The model outputs text fragments per audio chunk — not complete sentences. Sometimes it hallucinates. Sometimes it produces chatbot-style responses ("How can I help you today?") instead of transcriptions. You need to heal sentence boundaries, filter garbage, and maintain a rolling context window so the model has continuity between chunks.
What Sibyl does:
- SegmentBuffer (sentence healer) — accumulates text fragments, only commits when it sees sentence-ending punctuation (
.,?,!) or VAD silence. Prevents storing "I think we should" without "go with option B." - Garbage filtering — detects and suppresses chatbot-style model responses that aren't real transcriptions
- Rolling context window — last 10 segments are kept as working memory for continuity
- Minimal transcription prompt — just "Transcript:" with temperature 0.05 and top-k 10 to keep the model focused
Code: src-tauri/src/agents/memory_bridge.rs (SegmentBuffer, MemoryBridge)
The assumption: Store the text, done.
The reality: If you want to do anything intelligent with the conversation — find relevant context, detect patterns, search past sessions — you need semantic embeddings and a vector store. And you need to decide what goes into long-term memory vs. what stays in the short-term rolling window.
What Sibyl does:
- Dual memory — Working Memory (rolling 10-segment buffer, ~30s) + Vector Store (full session, semantic search)
- fastembed — generates 384-dimension embeddings (all-MiniLM-L6-v2) for semantic similarity
- Hybrid search — combines recency (last 5 segments) with relevance (top 3 semantic matches) and a 3-minute sliding window
- VAD-triggered commits — segments flow from SegmentBuffer → MemoryBridge → Vector Store only on speech boundaries
Code: src-tauri/src/lance/ (vector store, embeddings), src-tauri/src/agents/memory_bridge.rs
This is where Sibyl becomes a coaching app specifically. Everything below this line is domain logic built on the layers above. If you're building something different (speech-to-text for forms, a transcription tool, a meeting assistant), you'd replace this layer while reusing Layers 1-7.
What Sibyl does at this layer:
- Temperature tracking — EMA-smoothed "heat" metric (0.0 calm → 1.0 heated) for conversation tone
- Signal detection — buying signals, objections, hesitation, agreement (each with urgency scores 1-10)
- Coaching triggers — fires interventions on high temperature, urgent signals, or signal accumulation
- Flash Alerts — real-time coaching prompts displayed as a teleprompter overlay
- Post-session report cards — grades, emotion distribution, key moments, recommendations
- RAG — semantic search over playbooks and conversation history for context-aware coaching
Code: src-tauri/src/coaching/, src-tauri/src/agents/analyst.rs, src/components/
┌─────────────────────────────┐ ┌─────────────────────────────┐
│ Next.js Frontend │ │ Rust Backend │
│ (SSG, runs in Tauri) │◄───►│ (Tauri shell) │
│ │ IPC │ │
│ Teleprompter / Coaching UI │ │ Audio ─► VAD ─► ONNX │
│ Session history / Reports │ │ Agents ─► Memory ─► Coach │
└─────────────────────────────┘ └─────────────────────────────┘
Rust modules (src-tauri/src/):
| Module | Purpose |
|---|---|
audio/ |
Mic capture, WASAPI/SCK loopback, mixer, VAD |
onnx/ |
ONNX Runtime sessions, mel spectrograms, multimodal pipeline, streaming |
agents/ |
Scribe (transcription), Analyst (pattern detection), MemoryBridge, ModelLoader |
coaching/ |
Temperature tracking, signal detection, coaching triggers |
lance/ |
In-memory vector store, fastembed embeddings |
hardware/ |
CPU/GPU/NPU detection, execution provider selection |
db/ |
SQLite for sessions, transcripts, artifacts |
commands/ |
Tauri IPC handlers (frontend ↔ backend bridge) |
Frontend (src/):
| Path | Purpose |
|---|---|
app/page.tsx |
Home — model status, session history, search |
app/session/new/ |
Active session with audio capture |
app/session/[id]/ |
Session review with chat, report card |
app/teleprompter/ |
Real-time coaching overlay |
app/settings/ |
Mode config, system info |
lib/hooks.ts |
useModelLifecycle, useAudioCoaching |
lib/tauri.ts |
Type-safe Tauri command wrappers |
components/ |
FlashAlert, TemperatureBar, ReportCard, ASCIIProgressBar |
- Node.js >= 20, pnpm >= 8
- Rust (stable toolchain)
- CMake (for native dependencies)
- A GPU is recommended but not required (CPU fallback works)
# Clone
git clone https://github.com/YOUR_USERNAME/sibyl.git
cd sibyl
# Install JS dependencies
pnpm install
# Run frontend only (UI development)
pnpm dev
# Run full app (Rust + Next.js)
pnpm tauri devCargo and CMake must be in PATH. If you've installed Rust to a non-default location:
export PATH="/c/Dev/.cargo/bin:$PATH"
export CMAKE="/c/Program Files/CMake/bin/cmake.exe"# NVIDIA (default)
pnpm tauri dev # uses onnx-cuda
# AMD / Intel Arc (Windows)
pnpm tauri dev -- -- --features onnx-directml --no-default-features
# Apple Silicon
pnpm tauri dev -- -- --features onnx-coreml --no-default-features
# Intel NPU
pnpm tauri dev -- -- --features onnx-openvino --no-default-featuresOn first launch, Sibyl downloads the Gemma 3n E2B FP16 ONNX model (~11GB) from HuggingFace. This is automatic — the ModelLoaderAgent handles download and loading in the background with progress events sent to the UI.
If you want to build something other than a coaching app, here's what to keep and what to replace:
| Building... | Keep | Replace/Extend |
|---|---|---|
| Speech-to-text for forms | Layers 1-6 (audio → transcript) | Layer 8 (output to form fields instead of coaching) |
| Meeting transcriber | Layers 1-7 (audio → memory) | Layer 8 (export transcript instead of coaching) |
| Voice-controlled app | Layers 1-5 (audio → model) | Layers 6-8 (intent detection instead of transcription) |
| Your own coach/assistant | Everything | Customize coaching prompts and triggers |
The key reusable pieces are in src-tauri/src/audio/ (capture + VAD) and src-tauri/src/onnx/ (preprocessing + inference). These are the layers that take the most time to get right from scratch.
This is a working prototype. The audio pipeline, model inference, and memory system are functional. The coaching UI has real-time alerts, temperature visualization, and post-session reports.
Working:
- Dual-stream audio capture (mic + system audio)
- Voice activity detection with tuned thresholds
- Gemma 3n E2B inference via ONNX Runtime (CUDA verified)
- Sentence healing and garbage filtering
- In-memory vector store with semantic search
- Real-time coaching with Flash Alerts
- Post-session report cards
In Progress:
- Coach REPL (interactive query interface)
- KV cache optimization for faster inference
- macOS ScreenCaptureKit integration