Repository navigation
Scorer-eval GUI drops the coverage fields the docs say to compare #3325
Description
Scorer-eval GUI drops the coverage fields the docs say to compare
Summary
The scorer-evaluation docs (doc/code/scoring/4_scorer_metrics.py) correctly tell readers to compare num_responses against num_input_responses to see coverage, because responses with no verdict are excluded from accuracy. The exclusion itself is intended and test-pinned.
What breaks the guidance: the GUI backend read model (scenario_progress_read_model.py:600-612) drops both fields at the API boundary, and ObjectiveScorerDetails.tsx renders the masked accuracy with no way to surface a caveat.
Demonstration (real ObjectiveScorerEvaluator.evaluate_dataset_async, 10-item dataset): a scorer returning no verdict on the 3 hard items reports accuracy 1.0000 / F1 1.0000 at 7/10 coverage (identical to a perfect scorer) while an honest scorer attempting all 10 at 70% correct reports 0.7000. Abstention is strictly rewarded. The GUI cannot show why.
Suggested direction: carry num_responses / num_input_responses through the read model and render coverage alongside accuracy in the scorer-details view.
Activity
GBX-Max1220 commentedon Oct 18, 2026
Hi — I traced the GUI gap described in this issue. The saved scorer metrics added by #2860 already retain
num_responsesandnum_input_responses, but the scenario progress projection omits both fields, andObjectiveScorerDetailsrenders accuracy without coverage.I can take a focused follow-up that propagates these counts through the backend read model and TypeScript types, then displays scoring coverage alongside conditional accuracy.
I'd include regression tests for partial coverage (7/10 scored) and legacy metrics without denominator fields.
Would this scope be welcome, or is someone already handling the GUI projection?
GBX-Max1220 commentedon Oct 19, 2026
[DRAFT] FIX: Surface scorer evaluation coverage in the GUI (#3325)
Description
Fixes the scorer-evaluation coverage visibility gap described in #3325.
Scorer metrics already retain the number of scored responses and the original number of inputs. However, these fields were omitted from the scenario progress API projection, so the GUI could display 100% accuracy without showing that some inputs received no verdict.
This change propagates num_responses and num_input_responses through the backend response model and TypeScript types, and presents scoring coverage alongside accuracy when the denominator is available.
Legacy metrics without coverage information remain supported. Invalid denominator combinations are displayed as unavailable rather than producing misleading percentages.
Tests and Validation
The patch was validated locally with focused Python and Jest regression tests, negative controls against the unpatched implementation, broader regression suites, TypeScript checks, ESLint, Ruff, and the project's ty check.
The broader Windows test runs retained known environment-related failures; no patch-specific regression was identified in the recorded baseline comparison.
The final six-file patch was independently verified as byte-identical to the tested worktree using the project's LF checkout rules.
Related: #3325, #2860.