Docs: skill usage, repo setup, and validated triage mode#6
Merged
Conversation
- benchmark/README.md: the plugin's user guide — review mode (session
walkthrough, report format), triage mode ("should this lecture be
converted?"), manual pipeline quickstart, plugin map
- SKILL.md: triage-mode section (the prospective subset: baseline
as-used total, pattern match against the calibrated poles, crossover
check, readability-cost forecast, and the weight-algebra decision
rule)
- docs/using-skills.md: consumer guide — setup paths, invocation forms,
report-first expectations, troubleshooting
- docs/developing-skills.md: contributor guide — layout, conventions,
dev loop, versioning, squash-merge/stacking and external-author
attribution patterns
- README.md: documentation index
Triage mode is empirically validated before being documented: blind
triage using only baseline-side data (fresh runs: ge_arrow 0.028s,
markov_asset 0.087s; committed calibration: aiyagari pattern 54.3s)
reproduces all three known verdicts (don't convert / don't convert /
convert), and correctly cannot predict conversion-quality defects
(markov_asset's build bug) — that scope limit is documented with it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a set of repo + plugin documentation pages covering (1) how to install/invoke QuantEcon skills, (2) how to develop/validate plugins in this repository, and (3) how to use the benchmark evaluation skill in both review and validated triage mode.
Changes:
- Adds top-level documentation entrypoints in
README.mdand introduces two new guides:docs/using-skills.mdanddocs/developing-skills.md. - Documents
benchmarkplugin usage (review + triage modes) inbenchmark/README.md. - Extends the
review-accelerationskill procedure to include a triage mode section.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Adds a Documentation section linking to the new guides and the benchmark plugin user guide. |
| docs/using-skills.md | New end-user guide for setup, invocation, expectations, and troubleshooting. |
| docs/developing-skills.md | New contributor guide for repo layout, conventions, validation workflow, versioning, and PR flow. |
| benchmark/skills/review-acceleration/SKILL.md | Adds a “Triage mode” procedure and decision rule rationale to the skill doc. |
| benchmark/README.md | New benchmark plugin user guide describing review vs triage, manual pipeline quickstart, and repo map. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Addresses Copilot review on #6: the manual-install snippet is three commands, not two; the benchmark map row's link text now matches its target (scripts/README.md) with the engine path named in the description instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The evidence-file + engine pattern applies to skills that aggregate judgements into scored verdicts; findings-list skills need only cited claims. developing-skills.md gets the three-tier discipline; CATALOG gains the principle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two independent design critiques of the evaluation system (a fresh unframed session; a 36-agent adversarial workflow with steelman defense) were merged in reviews/. Three of our own claims were falsified by execution and are corrected here: - markov_asset's lecture DOES build in notebook order: a stale global err masks the stray err.throw(), silently disabling the checkify stability validation (worse than a crash, but not a build failure). Erratum prepended to the REPORT; wording corrected in examples README, SKILL.md, plugin README; correction posted on lecture-python.myst#654 - "mirrors the lecture exactly": both reference replays deviate from the lectures' construction patterns; certification corrected - "medians over repeats": false for the as-used totals (single pass per side); fairness-audit wording corrected; triage validation noted as in-sample reviews/ holds the independent report and the merged synthesis: which critiques survived the steelman defense (convention-only safety couplings; readability instrument inverting its own exemplars; the review/triage mode contradiction; band-label semantics; noise vs resolution) and which were defended (ratio form, weighted total, min() shape, per-consequence billing), plus the ranked v2 plan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
9 tasks
The evaluation findings briefly posted to lecture-python.myst#654 were withdrawn; the PR will receive one authoritative evaluation after the rubric-v2 revision and a full skill run, rather than a comment-and-correction trail. Erratum and merged review updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #5 (retargets to main when it merges) — documentation for the evaluation skill and the repo in general, written for @xuanguang-li to review and test alongside the system itself.
What's here
benchmark/README.md — the plugin's user guide: the two modes (review a conversion PR / triage whether a lecture is worth converting), what a session and its report look like, the manual pipeline quickstart, and a map of the plugin. Deliberately pointer-based: weights/anchors/bands stay canonical in EVALUATION_FRAMEWORK.md.
Triage mode — documented only after being tested. Blind triage using baseline-side data alone (fresh measurements this machine: ge_arrow total 0.028s, markov_asset 0.087s; committed calibration: aiyagari pattern 54.3s) reproduces all three known verdicts, and the scope limit is part of the doc: triage cannot predict conversion-quality defects like markov_asset's build bug. The decision rule falls out of the rubric weights: efficiency (0.15) can gain at most +0.30 weighted while readability (0.25) losing two bands costs −0.50, so a conversion that costs meaningful readability cannot break even on speed alone. Also added as a section in SKILL.md.
docs/using-skills.md — for authors/reviewers: the three setup paths (settings.json opt-in, manual install, CI), invocation forms, the report-first/never-silently-edit contract, troubleshooting.
docs/developing-skills.md — for contributors: layout, conventions (verb-first names, deterministic-before-LLM, no cross-doc duplication, self-contained plugins), the dev loop (
claude --plugin-dir,validate.pywith negative testing), versioning, and the squash-merge/stacked-branch + external-author attribution patterns from this PR series.All relative links verified resolving;
validate.pygreen.@xuanguang-li — the useful test here: follow benchmark/README.md cold (both the manual pipeline and the triage recipe) and note anywhere the docs and your intent diverge.
🤖 Generated with Claude Code