Skip to content

Word-level timestamps from the RNNT emission trace - #1

Open
pantafive wants to merge 2 commits into
kruatech:mainfrom
pantafive:timestamps
Open

Word-level timestamps from the RNNT emission trace#1
pantafive wants to merge 2 commits into
kruatech:mainfrom
pantafive:timestamps

Conversation

@pantafive

@pantafive pantafive commented Aug 5, 2026

Copy link
Copy Markdown

Implements the word-level timestamps roadmap item, in two commits: a packaging fix and the feature itself (with unit tests and README docs).

Word timestamps

RNNTGreedyDecoder already records the encoder frame index of every emitted token in RNNTGreedyStepTrace — this PR groups that existing trace into words, so timestamps cost no extra inference:

  • GigaAMProTranscription and GigaAMResult gain words: [GigaAMProWord] (text, start, end in seconds) — available both from the low-level model API and from GigaAMRecognizer.transcribe.
  • Tokens group at sentencepiece boundaries; punctuation pieces emitted by the e2e model attach to the preceding word; <unk> becomes a standalone word, matching how decode renders it in the text.
  • Word start = first token's frame, end = last token's frame + 1. Frame duration = mel hop (10 ms) × 4 conformer subsampling = 40 ms, exposed as GigaAMProModel.encoderFrameDuration (derived from the mel config and manifest sample rate, not hardcoded).
  • SentencePieceTokenizer.piece(forTokenId:) added for piece lookup without a full decode.
  • The grouping is a pure function with a new Tests/GigaAMKitTests target covering boundaries, punctuation, <unk>, blank/control skipping, mid-word chunk starts, and the frame math (swift test: 7/7). README gets a «Пословные таймстемпы» section and the roadmap checkbox.

This mirrors the technique the official gigaam python package uses for transcribe(..., word_timestamps=True) (salute-developers/GigaAM#65).

Validated in a real macOS app on Russian call recordings: timestamps line up with Silero-VAD segment boundaries, and the text output is unchanged (words are built from the same trace the text decode consumes).

Packaging fix (first commit)

Package.swift depended on mlx-swift branch: "main", which SPM refuses to resolve alongside dependents that pin mlx-swift by version — making GigaAMKit unusable as a library dependency in most real projects. Changed to .upToNextMinor(from: "0.31.3"); the API surface used here (MLX/MLXNN ops) is stable in 0.31.x. Package.resolved is regenerated to match, pinned at 0.31.3 — the newest mlx-swift line buildable below Swift 6.3 (0.31.3/0.31.4 manifests are swift-tools 5.12; 0.31.5+ require 6.3, and SPM tools-version gating lets newer toolchains resolve up to 0.31.6 while older ones stay on 0.31.x-early). Happy to adjust the range or split this into a separate PR if you prefer.

API compatibility notes

  • GigaAMProTranscription's memberwise init is internal, so its added field is not source-breaking for consumers.
  • GigaAMResult.init gains words with a default value — existing call sites compile unchanged. One caveat: the synthesized Codable now expects a words key, so JSON of the old shape would need a migration step if anyone persists results; say the word and I'll add a decodeIfPresent initializer.

pantafive and others added 2 commits August 5, 2026 10:36
A branch: "main" dependency makes the package unresolvable for any dependent
that pins mlx-swift by version — SPM refuses to mix branch and version
requirements for the same package. .upToNextMinor(from: "0.31.3") keeps the
package usable as a library dependency.

Package.resolved is regenerated to match: pinned at 0.31.3 (swift-tools 5.12,
the newest line buildable below Swift 6.3 — 0.31.5+ manifests require 6.3, so
toolchains older than that resolve 0.31.3/0.31.4 via SPM tools-version gating,
while newer toolchains may pick up to 0.31.6).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The greedy decoder already records the encoder frame index of every emitted
token in RNNTGreedyStepTrace; this groups that trace into timestamped words:

- GigaAMProTranscription and GigaAMResult gain words: [GigaAMProWord]
  (text, start, end in seconds).
- Tokens group at sentencepiece "▁" boundaries; punctuation pieces from the
  e2e model attach to the preceding word; <unk> becomes a standalone ⁇ word,
  matching how decode renders it in the text.
- Word start = first token's frame, end = last token's frame + 1. Frame
  duration = mel hop (10 ms) x 4 conformer subsampling = 40 ms, exposed as
  GigaAMProModel.encoderFrameDuration (derived, not hardcoded).
- SentencePieceTokenizer.piece(forTokenId:) added for piece lookup without a
  full decode.
- Unit tests for the grouping (pure function, no model files needed); README
  section + roadmap checkbox.

No extra inference cost: timestamps come from the same trace the text decode
consumes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pantafive

Copy link
Copy Markdown
Author

Two notes surfaced by an adversarial review while preparing this PR — both pre-existing in the repo, not introduced here, leaving them as a heads-up:

  1. gigademo/.../tokenizer.model is committed as real bytes into an LFS-attributed path. Any clone with git-lfs installed shows a permanent phantom modified diff for it (the clean filter wants to turn it into a pointer), and a checkout without git-lfs would break the checksums.sha256 expectation if the blob ever becomes a pointer. Either dropping the LFS attribute for this small file or re-committing it through LFS would resolve the ambiguity.

  2. README declares Swift 5.10+, but every resolvable mlx-swift version requires newer tools (0.31.3/0.31.4 → swift-tools 5.12; 0.31.5+ → 6.3). The effective minimum toolchain is 5.12 even before this PR (the previous branch: "main" pin was 6.3-only). This PR's lockfile pins 0.31.3 to keep the floor as low as mlx-swift allows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant