Word-level timestamps from the RNNT emission trace - #1
Conversation
A branch: "main" dependency makes the package unresolvable for any dependent that pins mlx-swift by version — SPM refuses to mix branch and version requirements for the same package. .upToNextMinor(from: "0.31.3") keeps the package usable as a library dependency. Package.resolved is regenerated to match: pinned at 0.31.3 (swift-tools 5.12, the newest line buildable below Swift 6.3 — 0.31.5+ manifests require 6.3, so toolchains older than that resolve 0.31.3/0.31.4 via SPM tools-version gating, while newer toolchains may pick up to 0.31.6). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The greedy decoder already records the encoder frame index of every emitted token in RNNTGreedyStepTrace; this groups that trace into timestamped words: - GigaAMProTranscription and GigaAMResult gain words: [GigaAMProWord] (text, start, end in seconds). - Tokens group at sentencepiece "▁" boundaries; punctuation pieces from the e2e model attach to the preceding word; <unk> becomes a standalone ⁇ word, matching how decode renders it in the text. - Word start = first token's frame, end = last token's frame + 1. Frame duration = mel hop (10 ms) x 4 conformer subsampling = 40 ms, exposed as GigaAMProModel.encoderFrameDuration (derived, not hardcoded). - SentencePieceTokenizer.piece(forTokenId:) added for piece lookup without a full decode. - Unit tests for the grouping (pure function, no model files needed); README section + roadmap checkbox. No extra inference cost: timestamps come from the same trace the text decode consumes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Two notes surfaced by an adversarial review while preparing this PR — both pre-existing in the repo, not introduced here, leaving them as a heads-up:
|
Implements the word-level timestamps roadmap item, in two commits: a packaging fix and the feature itself (with unit tests and README docs).
Word timestamps
RNNTGreedyDecoderalready records the encoder frame index of every emitted token inRNNTGreedyStepTrace— this PR groups that existing trace into words, so timestamps cost no extra inference:GigaAMProTranscriptionandGigaAMResultgainwords: [GigaAMProWord](text,start,endin seconds) — available both from the low-level model API and fromGigaAMRecognizer.transcribe.▁boundaries; punctuation pieces emitted by the e2e model attach to the preceding word;<unk>becomes a standalone⁇word, matching howdecoderenders it in the text.GigaAMProModel.encoderFrameDuration(derived from the mel config and manifest sample rate, not hardcoded).SentencePieceTokenizer.piece(forTokenId:)added for piece lookup without a full decode.Tests/GigaAMKitTeststarget covering boundaries, punctuation,<unk>, blank/control skipping, mid-word chunk starts, and the frame math (swift test: 7/7). README gets a «Пословные таймстемпы» section and the roadmap checkbox.This mirrors the technique the official gigaam python package uses for
transcribe(..., word_timestamps=True)(salute-developers/GigaAM#65).Validated in a real macOS app on Russian call recordings: timestamps line up with Silero-VAD segment boundaries, and the text output is unchanged (words are built from the same trace the text decode consumes).
Packaging fix (first commit)
Package.swiftdepended on mlx-swiftbranch: "main", which SPM refuses to resolve alongside dependents that pin mlx-swift by version — making GigaAMKit unusable as a library dependency in most real projects. Changed to.upToNextMinor(from: "0.31.3"); the API surface used here (MLX/MLXNN ops) is stable in 0.31.x.Package.resolvedis regenerated to match, pinned at 0.31.3 — the newest mlx-swift line buildable below Swift 6.3 (0.31.3/0.31.4 manifests are swift-tools 5.12; 0.31.5+ require 6.3, and SPM tools-version gating lets newer toolchains resolve up to 0.31.6 while older ones stay on 0.31.x-early). Happy to adjust the range or split this into a separate PR if you prefer.API compatibility notes
GigaAMProTranscription's memberwise init is internal, so its added field is not source-breaking for consumers.GigaAMResult.initgainswordswith a default value — existing call sites compile unchanged. One caveat: the synthesizedCodablenow expects awordskey, so JSON of the old shape would need a migration step if anyone persists results; say the word and I'll add adecodeIfPresentinitializer.