Report per-token confidence from greedy decoding

[gman-dev-nov]

Motivation

Greedy decoding knows how sure it was about every token it emitted, and then throws that away. Both decoding paths take an argmax over a distribution that is already log_softmax-normalized:

• CTCGreedyDecoding.decode — log_probs.argmax(dim=-1)
• RNNTGreedyDecoding.decode — head.joint.joint(f, g)[:, 0, 0, :].argmax(dim=-1), and RNNTJoint.joint returns .log_softmax(-1)

Replacing argmax with max keeps the index and the value in the same pass, so the score costs nothing. Downstream that enables ranking segments for review, routing low-confidence spans to a second pass, and filtering pseudo-labels when bootstrapping training data — all of which currently require re-running the model or forking the decoder.

What changed

Both decoders now return a Hypothesis named tuple that carries token_logprobs next to the existing text / token ids / token frames. Those are aggregated and surfaced as:

Field	Scope
TranscriptionResult.confidence	whole utterance
Segment.confidence	one long-form segment
Word.confidence	one word (with word_timestamps=True)

The aggregate is the length-normalized geometric mean exp(mean(log p)), so long words are not penalized for consisting of more tokens. Blank decisions are excluded. Raw per-token values stay available on Hypothesis.token_logprobs.

Confidence is reported even without word_timestamps=True — the utterance-level score needs no timestamp machinery.

Live examples

Per-word scores on the bundled example.wav (v3_e2e_rnnt, utterance confidence 0.9110):

word               start     end    conf
Ничьих              0.04    0.40  0.7858
не                  0.52    0.56  0.9837
требуя              0.64    0.96  0.9933
похвал,             1.08    1.60  0.9964
Счастлив            1.72    2.16  0.8432
...
Что                 3.72    3.76  0.5892
дева                3.88    4.08  0.7004
с                   4.16    4.20  0.9981
трепетом            4.24    4.72  0.9334
любви               4.80    5.04  0.9977

Long-form segments on long_example.wav carry it too:

[  0.00- 16.90] conf=0.9300  Вечерня отошла давно, Но в кельях тихо и темно; ...
[ 17.10- 32.80] conf=0.9026  Трепещет луч лампады, И тускло озаряет он ...
[ 32.90- 49.40] conf=0.9088  Глухой и влажный Стоят за клиросом чернец и грешник ...
[ 49.80- 67.10] conf=0.8820  Ужасна исповедь злодея, Заплачена тобою дань ...
[ 67.50- 71.00] conf=0.9023  Грехов сложи мучительное бремя.

The score responds to acoustic difficulty. Burying the same utterance in white noise at 0 dB SNR:

Model	clean	0 dB SNR
v3_e2e_rnnt	0.9110	0.8049
v3_ctc	0.9910	0.8695

This is asserted in tests/test_confidence.py::test_confidence_drops_on_noisy_audio.

It is also informative inside a clean recording. On a 24-minute Russian tech talk (3022 words, v3_e2e_rnnt, median confidence 0.9568), the words the model attempted to spell in Latin scored a median of 0.7317, and the ones it mangled outright sit in the bottom few percent:

Word	Actual term	Confidence	Percentile
SGAG	guardrail	0.5729	1.9%
Woldpoot	world	0.6011	2.7%
Impoot	input	0.6218	3.4%
Elmas	LLM as…	0.6738	5.9%

Limitations, documented in both READMEs

This is a greedy per-token posterior, not a calibrated probability of correctness, and RNN-T greedy scores are known to be over-confident. Two failure modes are visible in the same recording above, and the README says so plainly rather than overselling the feature:

• Confidently wrong output still scores high. guardrail transliterated as гардрейл scored 0.8142 (18th percentile) and квартрейл 0.8502 (24th percentile) — wrong, but nowhere near the tail.
• A low score does not imply an error. The bottom of the distribution is dominated by short function words at segment boundaries (и, не, я), usually transcribed correctly.

The recommendation given is to use it as one signal among several, not as a standalone error detector.

Compatibility

• Hypothesis is a NamedTuple that preserves the positional layout of the old return value, so indexing and slicing are unchanged. Code that unpacks exactly three values must read fields by name — the one such site in this repo, train_utils/eval.py, is updated. train_utils/module.py uses h[0] and is unaffected. Happy to hide this behind a compatibility shim instead if you would rather keep the bare tuple.
• Word, TranscriptionResult and Segment gain an optional field defaulting to None; construction by keyword is unaffected.
• The ONNX inference path in onnx_utils.py has its own decoding and is left untouched.

Tests

tests/test_confidence.py — 11 tests, both revisions:

• pure-unit: aggregation is the geometric mean and length-invariant; Hypothesis positional compatibility; frames_to_words leaves confidence None when no log-probs are passed
• model-backed: utterance confidence present and in range without timestamps; every word scored and in range; confidence drops on noisy audio; long-form segments and their words both scored

Verified locally on macOS/arm64 (v3_ctc, v3_e2e_rnnt): 9 passed, plus the 2 long-form tests exercised through a local VAD substitute since pyannote is not installable here — CI installs the longform extra and will run them directly. Existing tests/test_timestamps.py still passes. black and isort clean.