Land the lecture evaluation system (benchmark plugin 0.3.0: rubric v2, skill wired)

[mmcky]

Lands @xuanguang-li's complete evaluation system for lecture code rewrites — the deliverable #4 was waiting on, developed and validated on QuantEcon/lecture-python.myst#717 and QuantEcon/lecture-python.myst#654. Two commits, deliberately split: commit 1 is the package as delivered, authored by @xuanguang-li; commit 2 is path/plumbing integration plus docs.

What lands

Where	What
references/EVALUATION_FRAMEWORK.md	The standard in prose: 7 weighted dimensions, numeric anchors, structural checklists, verdict bands, worked HIGH/LOW examples
scripts/scoring/	The standard as code: rubric.py (evidence → score, deterministically — never hand-typed), score.py (CLI), EVIDENCE_TEMPLATE.json (the judgement contract: measured numbers + cited true/false answers)
scripts/calibration/	The shared aiyagari Bellman benchmark pinning the "25× as-used = score 5" anchor
references/examples/ge_arrow/	Complete worked evaluation: 2.85/5 mixed/wash (#717)
references/examples/markov_asset/	Complete worked evaluation: 2.25/5 net regression — build-breaking bug (#654; reported there)

The two worked cases double as the regression baseline: the skill must reproduce their verdicts. Verified in this PR — both scorecards regenerate byte-identically from the new layout via python scripts/scoring/score.py references/examples/<lecture>.

Integration (commit 2)

• score.py takes a lecture directory path (runs from any cwd); ge_arrow scripts converted to local imports matching the markov_asset idiom
• run_all.py writes a provenance stamp (results/env.json: python/platform/library versions) — the seed of the PROJECT: QuantEcon benchmarking programme (code performance & execution) meta#335 shared result + environment-descriptor schema
• All relative links in the framework and reports rewritten for the new layout (no dangling references)
• scripts/README.md rewritten around the three-step contract (measure → record evidence → score); SKILL.md procedure now points at the real engine and templates, with the worked cases as calibration anchors
• benchmark plugin 0.1.0 → 0.2.0, marketplace kept in sync; validate.py green

Design note

Per-lecture measurement scripts are adapted templates, not a fixed harness — adapting them to each lecture's examples and call sequence is exactly the step the skill automates, while scoring stays deterministic. This supersedes the "generalised CLI" idea in #4 item 2.

🤖 Generated with Claude Code

Update 2026-07-22 — docs merged in, rubric v2, skill wired (plugin 0.3.0)

This PR now also carries three follow-ups so the whole system can be tested from this branch before landing on main:

1. PR Docs: skill usage, repo setup, and validated triage mode #6 squash-merged in (2aea122): user/developer docs, the benchmark user guide, validated triage mode, the design-review record (reviews/), and the markov_asset corrections of record.
2. Rubric v2 (719e825) — the four engine quick-wins from PLAN: evaluation rubric v2 — enforce couplings, verdict vocabulary, instrument fixes (from the three-way design review) #7 that survived the three-way design review, each validated against the committed evidence files: safety couplings enforced in score_all (derived logic&design bug-cap; correctness 1/2 gates the verdict band — the review's honest-evidence 4.2 hole now gates to net regression), the no-conversion verdict (reconciles review mode with triage), a one-flip sensitivity stamp (robust/fragile with deciding flips), and K-repeat as-used (median of 3 fresh-process runs with contested-band annotation).
3. Skill wiring (a36a744): evaluations run in the user's workspace (benchmark-eval/<lecture>/) with the plugin read-only at CLAUDE_PLUGIN_ROOT; preconditions and the extraction/replay diff check added to the procedure.

Re-validation: both totals are unchanged (ge_arrow 2.85, markov_asset 2.25); both verdicts now lead with no-conversion, markov_asset's additionally gated at net regression. ge_arrow stamps fragile (the review's demonstrated flips, listed in the scorecard); markov_asset stamps robust — the v2 gate absorbs the one-concept band flip the review demonstrated. Band movements are documented as deliberate v2 changes in both reports. Plugin 0.2.0 → 0.3.0, marketplace in sync, validate.py green.

[Copilot]

Pull request overview

This PR lands the benchmark plugin’s lecture-acceleration evaluation framework, including a prose rubric plus a deterministic scoring engine and two worked evaluation baselines (ge_arrow, markov_asset) that can be regenerated from evidence files.

Changes:

• Adds a deterministic scoring engine (scripts/scoring/) that computes dimension scores + weighted totals from per-lecture evidence.json.
• Adds a shared efficiency calibration benchmark (scripts/calibration/bellman_bench.*) and updates plugin/marketplace versions to 0.2.0.
• Adds the evaluation framework document plus two fully-worked evaluation example directories (scripts, results, evidence, reports) used as regression anchors.

Reviewed changes

Copilot reviewed 42 out of 43 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
README.md	Updates plugin status messaging for benchmark.
benchmark/skills/review-acceleration/SKILL.md	Replaces placeholder procedure with the new 3-step evidence→score workflow and calibration anchors.
benchmark/scripts/scoring/score.py	Adds the scoring CLI that reads <lecture>/evidence.json and writes <lecture>/results/scorecard.json.
benchmark/scripts/scoring/rubric.py	Implements the rubric as code (weights, thresholds, checklists, verdict bands).
benchmark/scripts/scoring/EVIDENCE_TEMPLATE.json	Introduces the evidence schema template for new evaluations.
benchmark/scripts/README.md	Documents the scripts layout and end-to-end evaluation workflow.
benchmark/scripts/calibration/bellman_bench.py	Adds the shared high-end efficiency calibration benchmark.
benchmark/scripts/calibration/bellman_bench.json	Commits the benchmark output pinning the “score 5” efficiency anchor.
benchmark/references/examples/markov_asset/scripts/static_metrics.py	Adds markov_asset static metrics extraction used in scoring evidence.
benchmark/references/examples/markov_asset/scripts/smoke_test.py	Adds a runnable smoke test for old/new markov_asset implementations.
benchmark/references/examples/markov_asset/scripts/run_all.py	Adds markov_asset pipeline orchestration + provenance stamp + shared scoring invocation.
benchmark/references/examples/markov_asset/scripts/model_old.py	Adds extracted baseline NumPy implementation under evaluation.
benchmark/references/examples/markov_asset/scripts/model_new.py	Adds extracted candidate JAX implementation under evaluation (verbatim).
benchmark/references/examples/markov_asset/scripts/check_equivalence.py	Adds markov_asset equivalence checking (float32 vs x64) and patched-call-option comparison.
benchmark/references/examples/markov_asset/scripts/benchmark.py	Adds markov_asset warm scaling benchmark.
benchmark/references/examples/markov_asset/scripts/as_used_total.py	Adds markov_asset as-used (cold-inclusive) end-to-end timing script.
benchmark/references/examples/markov_asset/results/static_metrics.json	Adds captured metrics output used by the evidence/scorecard.
benchmark/references/examples/markov_asset/results/scorecard.json	Adds computed scorecard output for markov_asset baseline.
benchmark/references/examples/markov_asset/results/scaling.json	Adds captured scaling results for markov_asset baseline.
benchmark/references/examples/markov_asset/results/equivalence_x64_True.json	Adds captured x64 equivalence results for markov_asset baseline.
benchmark/references/examples/markov_asset/results/equivalence_x64_False.json	Adds captured float32-as-shipped equivalence results for markov_asset baseline.
benchmark/references/examples/markov_asset/markov_asset_REPORT.md	Adds the worked markov_asset evaluation report.
benchmark/references/examples/markov_asset/evidence.json	Adds the worked markov_asset evidence file consumed by the scorer.
benchmark/references/examples/ge_arrow/scripts/sweep_bench.py	Adds ge_arrow sweep benchmark used by the evaluation pipeline.
benchmark/references/examples/ge_arrow/scripts/static_metrics.py	Adds ge_arrow static metrics extraction used in scoring evidence.
benchmark/references/examples/ge_arrow/scripts/run_all.py	Adds ge_arrow pipeline orchestration + provenance stamp + shared scoring invocation.
benchmark/references/examples/ge_arrow/scripts/model_old.py	Adds extracted baseline NumPy implementation under evaluation.
benchmark/references/examples/ge_arrow/scripts/model_new.py	Adds extracted candidate JAX implementation under evaluation (verbatim).
benchmark/references/examples/ge_arrow/scripts/cold_start.py	Adds cold-start latency measurement for ge_arrow evaluation.
benchmark/references/examples/ge_arrow/scripts/check_equivalence.py	Adds ge_arrow equivalence checking across all lecture examples.
benchmark/references/examples/ge_arrow/scripts/benchmark.py	Adds ge_arrow benchmark (as-used, warm, scaling) used in scoring evidence.
benchmark/references/examples/ge_arrow/scripts/as_used_total.py	Adds ge_arrow as-used (cold-inclusive) end-to-end timing script.
benchmark/references/examples/ge_arrow/results/sweep.json	Adds captured sweep benchmark results for ge_arrow baseline.
benchmark/references/examples/ge_arrow/results/static_metrics.json	Adds captured metrics output used by the evidence/scorecard.
benchmark/references/examples/ge_arrow/results/scorecard.json	Adds computed scorecard output for ge_arrow baseline.
benchmark/references/examples/ge_arrow/results/equivalence.json	Adds captured equivalence results for ge_arrow baseline.
benchmark/references/examples/ge_arrow/results/benchmark.json	Adds captured benchmark results for ge_arrow baseline.
benchmark/references/examples/ge_arrow/ge_arrow_REPORT.md	Adds the worked ge_arrow evaluation report.
benchmark/references/examples/ge_arrow/evidence.json	Adds the worked ge_arrow evidence file consumed by the scorer.
benchmark/references/EVALUATION_FRAMEWORK.md	Adds the evaluation framework specification, anchors, and workflow.
benchmark/.claude-plugin/plugin.json	Bumps benchmark plugin version to 0.2.0.
.gitignore	Adds standard Python/macOS ignore patterns.
.claude-plugin/marketplace.json	Updates marketplace version to 0.2.0.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mmcky]

Detailed logic review completed — all 17 scripts line-by-line, plus scripted cross-checks of every evidence value against its results-file source. Full write-up now lives in the PR as benchmark/references/examples/README.md (the logic-check and provenance document for the canonical examples).

Verdict: logic sound, methodology fair, every traceable number consistent. Fairness properties verified: block_until_ready on all JAX timings, _clear_cache() for cold trials, medians over repeats, identical call sequences per side in the as-used replays, the markov_asset bug patched only where timing is otherwise impossible (disclosed in docstring and output record), and a non-strawman NumPy baseline in the calibration (same vectorised algorithm, agreement to 1e-14).

Fixed in 433462b (mechanical, no scored value changes; both scorecards still regenerate byte-identically):

	Fix
M2	The headline as-used metric lived only on the console — run_all.py now persists the fresh-process scripts' JSON into results/as_used.json (+ cold_start.json for ge_arrow) with the derived as_used_speedup, as the docstrings already claimed
m4	ge_arrow static-metrics had a duplicated @ pattern double-counting concept token hits (informational metric; regenerated 110 → 105)
m5	markov_asset used statements_for_one_asset where the template/engine vocabulary is statements_for_one_result (values unchanged)

For @xuanguang-li's consideration (rubric/design territory — deliberately not changed here):

1. M1: n_prerequisite_concepts (the readability driver, weight 0.25 — the heaviest input in the system) and statements_for_one_result are hand-curated judgements encoded in the measurement scripts, disclosed as such. When the skill adapts templates to a new lecture it will author these lists, so they arguably belong in evidence.json as cited judgement slots — one citation per claimed concept, the same discipline the structural checklists enforce — rather than script constants.
2. m3: ge_arrow's equivalence.json doesn't stamp the x64 regime and is overwritten between regimes; markov_asset's equivalence_x64_{bool}.json + _meta pattern is the better template — worth back-porting on the next revision.
3. n6: sweep_bench's old side computes only α while the new one-call API forcibly computes everything — faithful to each API as used, and arguably the point (the API's cost is real), but worth a sentence in the script docstring.

Also still open from the earlier Copilot round: the matches_under_x64 guard semantics (see the reply on that thread) — same category, the standard's author's call.

🤖 Generated with Claude Code

[mmcky]

Final review completed (31-agent adversarially-verified pass: integration correctness traced path-by-path, measured duplication analysis, doc-overlap mapping, installed-plugin packaging, fresh-eyes diff review, and an execution agent that ran every documented command). Fixes landed in eacd5fe; both scorecards still regenerate byte-identically and every documented command works as written.

Two additional items for @xuanguang-li (found by the fresh-eyes pass; your files, your call):

1. The framework's HIGH-anchor table doesn't match the committed calibration data. EVALUATION_FRAMEWORK.md §Dim-3 quotes 1664/69 ms (24×) and 29.3/1.16 s (25×), but the committed scripts/calibration/bellman_bench.json records 2955/112 ms (26.4×) and 54.29/2.28 s (23.8×) — absolute timings differ ~1.8×, well beyond the ±15% run-to-run band, i.e. the prose table and the committed JSON come from different runs/machines. The speedups agree (~24–26×, so the anchor itself is sound); options are updating the table to the committed values or noting the table is from the original validation machine.

2. The jax 0.4.x pins in the docstrings are drifting — four places pin it while environments move on (the machine here now has jax 0.10.1). The new per-run env.json stamp (with steps_failed for partial runs) is the durable answer; the prose pins could soften to "the env recorded in results/env.json".

Deferred simplification bundle (measured, verified outcome-preserving, but touching your scripts' structure — proposed for whenever a third example lands or the next re-measurement, whichever comes first):

Consolidation	Measured	Est.
Shared run_steps.py runner (run_all is now ~89% identical across examples and fully lecture-independent)	78/88 LCS-identical lines	−50 LOC now, −70/lecture after
markov_asset's bug-patched call_option exists as a renamed clone in two scripts — the equivalence-verified copy is not the timed copy	AST-identical after alpha-renaming	one patched_call.py sibling, −16 LOC
ge_arrow's four lecture economies hand-built at four sites	~53 LOC	one economies.py sibling, −30 LOC
static_metrics AST core shared — best bundled with the M1 (prerequisite-concepts into evidence.json) decision	49–50 LCS lines, already diverging	conditional on M1

Deliberately not proposed: sharing the median-timer/results-path/allclose helpers — measured as a net negative (couples the templates the adaptation model wants independent).

🤖 Generated with Claude Code