feat(skychat): voice audio backend (PulseAudio) + terminal spectrogram demo#3416
Merged
Conversation
0pcom
force-pushed
the
feat/skychat-voice-audio
branch
from
8cd7eb4 to
289a9ec
Compare
…m demo Real audio for voice calls on Linux, cgo-free, plus a "see the audio" CLI view. pkg/skychat/voice/pulse_linux.go — capture (Source) + playback (Sink) via github.com/jfreymuth/pulse (pure-Go PulseAudio/PipeWire, no cgo). NewMicSource captures the mic, or the default sink's MONITOR (system output) so a call/demo can carry internal audio with no mic, like a PulseAudio monitor source. Sample rate is a parameter (calls: 48 kHz; spectrogram: 24 kHz). A blocking sample-ring paces the reader at the capture rate; non-linux stub returns ErrAudioUnsupported (malgo Windows/macOS is a follow-up). pkg/skychat/voice/spectrogram — the FFT/magnitude/color core, vendored+adapted from the operator's audioprism-go (pkg/spectrogram). Copied, not imported as a module, because audioprism-go back-imports skywire-utilities (a cycle); its go-dsp fork also drags a broken renamed transitive dep into `go mod tidy`, so the FFT + window functions are reimplemented locally (dsp.go, radix-2) — the package now needs only the stdlib. No replace directives. Unit-tested (sine → correct peak bin). cmd/skywire-cli/.../voice_spectrogram.go — `cli skychat voice spectrogram [--monitor]`: a live scrolling tcell spectrogram of local audio. Deliberately MIRRORS the audioprism-go tcell UI so it looks the same — 24 kHz capture, overlap-stepped FFT columns (one per StepSize), a 2048x1024 circular history downscaled to the terminal, vertical axis 0-12 kHz linear with low freq at the bottom, same MagnitudeToPixel colormap. Test asserts a 3 kHz tone peaks at the matching row. tcell/v2 already vendored. Only new dep: jfreymuth/pulse (pure Go). Validated live: monitor capture of a tone → FFT peak at the right bin, and the tcell view renders (screenshot). Build + lint + tests clean.
0pcom
force-pushed
the
feat/skychat-voice-audio
branch
from
289a9ec to
bc7d216
Compare
…t answer Makes a real 1:1 call carry audio, gated safely. - Real audio is OPT-IN via SKYWIRE_VOICE_AUDIO=mic|monitor (default: silent media + auto-answer, unchanged — safe for headless visors, a call leaks no audio). When enabled, init_voice wires the pulse Source/Sink and switches the manager to EXPLICIT-ANSWER so a visor never streams its microphone to a caller without an explicit answer. Falls back to silence if no audio device. - voice.Manager: ManualAnswer mode + a ringing registry. An inbound invite RINGS (parks the conn) until Answer(id)/Decline(id) or a 45s ring timeout, instead of the immediate OnIncoming decision. Ring callback notifies (logs) the incoming call. Incoming() lists ringing calls. Unit-tested: ring→answer establishes the call; ring→decline fails the caller. - Visor RPC + CLI: VoiceAnswer / VoiceDecline / VoiceIncoming → `cli skychat voice answer|decline|incoming`. (API interface + rpc server + rpc_client + mock + proxy stubs.) Build + lint + tests clean.
Collaborator
Author
|
✅ Live audio-path validation (real PulseAudio, on a PipeWire host): A captured the system monitor while a 2000 Hz tone played, streamed it through a voice So the real capture→session→playback audio path works end-to-end. Signaling, ring/answer, and the dmsg/skynet transport are covered by unit tests + the file-sharing validation. Remaining: two-panel in-call spectrogram, Opus, malgo (Win/Mac), wasm WebAudio. |
Adds the in-call "see the audio" view. Since call audio is captured/played on
the visor (server-side), the visor taps it and the CLI polls it:
- voice.Manager Visualize mode (pkg/skychat/voice/tap.go): tee the session's
Source (sent) and Sink (received) into small ~1s rings; CallAudio(id) serves
the recent PCM. Audio path untouched. Enabled when SKYWIRE_VOICE_AUDIO is set.
Unit-tested (tap captures the sent tone).
- Visor RPC VoiceCallAudio(id) → {Sent, Recv []int16} (+ API/rpc_client/mock/
proxy).
- `cli skychat voice spectrogram --call <id>`: polls the visor and draws two
side-by-side spectrograms (left: sent, right: received) with the same
audioprism-matched renderer, rate-aware (48 kHz call audio). Standalone
`voice spectrogram [--monitor]` unchanged.
Build + lint + tests clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Real audio for voice calls on Linux — cgo-free — plus a "see the audio" CLI view (the terminal stand-in for a video feed on a call).
Audio backend —
pkg/skychat/voice/pulse_linux.goCapture (
Source) + playback (Sink) viagithub.com/jfreymuth/pulse— a pure-Go PulseAudio/PipeWire client, no cgo, no system libs to link.NewMicSource(false)= mic;NewMicSource(true)= the default sink's monitor (system output), so a call or the demo can carry internal audio with no mic, like a PulseAudio monitor source. A blocking sample-ring paces the reader at the capture rate. Non-Linux builds get a stub returningErrAudioUnsupported(malgo Windows/macOS is a follow-up).Spectrogram DSP —
pkg/skychat/voice/spectrogramThe FFT / magnitude / colormap core, vendored + adapted from the operator's audioprism-go (
pkg/spectrogram). Copied rather than imported as a module because audioprism-go back-importsskywire-utilities(a cycle), and its go-dsp fork drags a broken renamed transitive dep (mjibson→madelynnblue) intogo mod tidy. So the FFT + window functions are reimplemented locally (dsp.go, radix-2) — the package now needs only the standard library. No replace directives. Unit-tested: a sine at a bin-aligned frequency peaks at exactly that FFT bin.CLI —
cli skychat voice spectrogram [--monitor]A live scrolling tcell spectrogram of local audio — the CLI's "see the audio" view (
tcell/v2was already vendored). The same renderer feeds a two-panel sent/received view during an actual call (next).Only new dependency:
jfreymuth/pulse(pure Go). No cgo, no replace, stdlib-only DSP.Validated live on a PipeWire host: monitor-captured a 1500 Hz tone → FFT peak at bin 32 (~1500 Hz) with the expected Hann-window spread.
go build ./...+golangci-lint+go test ./pkg/skychat/voice/...all clean.Next: wire the pulse Source/Sink into
initVoiceso calls carry real audio; tap sent/received PCM into a two-panel spectrogram; Opus codec; malgo cross-platform; wasm WebAudio/WebCodecs path.