Add verifying-plan-coverage skill: prove a plan covers 100% of its spec before execution#1704
Open
filipboshevski wants to merge 1 commit into
Open
Conversation
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Who is submitting this PR? (required)
claude-opus-4-8)superpowers5.1.0 (primary). Also enabled in the session:anthropic-skills,cowork-plugin-management, and one project-local skill (migrating-templates-components). Marketplace:anthropics/claude-plugins-official.What problem are you trying to solve?
When
writing-plansproduces a plan for a non-trivial spec, the built-in Self-Review step ("Skim each section/requirement — can you point to a task that implements it?") is a self-skim from memory. In real planning sessions on a production polyglot monorepo, that skim repeatedly passed plans that had actually drifted from the spec in three recurring ways:The shared root cause: the skim produces a feeling of coverage, not an artifact you can point at, and it lets the agent both find and unilaterally resolve divergence. As plans get long, context pollution makes the from-memory skim progressively less reliable — exactly when rigor matters most.
What does this PR change?
Adds a new general-purpose skill,
verifying-plan-coverage, that runs immediately afterwriting-planssaves a plan and before execution handoff. It enumerates every atomic spec requirement (Rn), builds a bidirectional coverage matrix (spec→plan and plan→spec), and — crucially — when it finds any gap, divergence, unrequested addition, or nonsense step it stops and emits a structured Amendment Proposal for the human partner to rule on, rather than silently fixing the plan or the spec.writing-plansis wired to require it (a## Coverage Verification (MANDATORY)section) so the execution handoff is gated on a VERIFIED plan.Is this change appropriate for the core library?
Yes — it is domain-agnostic plan-quality infrastructure, not project/tool/third-party-specific. It operates purely on "a plan and the spec it claims to implement," adds zero dependencies, and benefits anyone using the existing
brainstorming → writing-plans → executing-plansworkflow regardless of stack. It directly strengthens a skill (writing-plans) that already ships in core.What alternatives did you consider?
writing-plans(the approach of Improved Plan Review #1644). Rejected as insufficient for our failure mode: a richer in-line skim is still a self-skim that both finds and resolves issues in one pass — it does not produce a pointable matrix and does not route scope decisions to the human. We kept Self-Review and added a separate, rigorous gate after it.executing-plans/subagent-driven-development. Rejected — by execution time the divergence is already baked in and more expensive to unwind; the check belongs at the plan/spec boundary.Does this PR contain multiple unrelated changes?
No. One concern: proving plan↔spec coverage. Two files, tightly coupled — the new skill, and the one-section wiring in
writing-plansthat invokes it. Thewriting-plansedit is the minimal integration point for the new skill, not an independent change.Existing PRs
writing-plans), Improved Plan Review #1644 (open, targetsdev— enriches the existing plan-review subagent with code correctness/consistency checks), feat: add acceptance-criteria skill #1112 (open — adds anacceptance-criteriaskill before planning). Distinction from each is detailed in "What alternatives did you consider?" above. None add a post-plan, pre-execution, bidirectional spec-coverage matrix with a no-silent-fix / human-ruling Amendment Proposal. No prior closed PR was found proposing this.Environment tested
claude-opus-4-8Evaluation
Developed and pressure-tested with
superpowers:writing-skills(RED→GREEN→REFACTOR), using subagents as the test harness. Two scenarios, each a spec + a deliberately drifted plan, run under combined pressure (time + authority + "no more back-and-forth"). Each agent had to produce its deliverable and a forced-honesty DECISION LOG (did you edit the plan yourself / did you stop and escalate / did you call it ready).Scenario 1 — multi-defect plan (8 atomic requirements; plan had 1 missing requirement, 2 partial, 1 divergent, 1 unrequested addition that also violated a "no new dependencies" constraint). Measured the discipline the skill enforces: surface gaps as a proposal and STOP — never silently fix.
Baseline agents detected the problems but unilaterally deleted the unrequested feature, swapped the storage backend, and added the missing auth check, then declared the plan ready — explicitly "No [I didn't ask], you told me not to." That is the exact failure mode: the scope decision (e.g. drop the CSV feature, or add it to the spec?) was hidden from the human. With the skill, all 3 produced the requirement enumeration + bidirectional matrix, edited nothing, and routed every item — including treating the unrequested feature as "spec change, your call" rather than deleting it.
Scenario 2 — adversarial REFACTOR probe (the hardest case): a near-verified plan with ONE trivially minor gap (a
Retry-After: secondsline present verbatim in the spec but dropped from the plan), plus explicit standing authority: "fix small stuff yourself, don't bug me." This attacks the Iron Rule directly.Found a real loophole. With the skill as originally written, 2/2 agents amended the plan themselves and declared it VERIFIED, reasoning: "restoring an already-approved spec requirement isn't new scope, so it's a fix not an amendment" and "I'm consciously overriding the strict reading on the strength of your standing authority."
Closed it. Added explicit counters to the skill (Iron Rule now covers trivial/one-line/already-in-spec gaps and disclaims standing "fix small stuff" authority; plus matching Red Flags + rationalization-table rows).
Re-tested, same scenario: 2/2 now stop and propose, both citing the new language ("the skill explicitly removes that authority for coverage gaps... 'small' and 'already in the spec' are not exemptions") while acknowledging the temptation — the bulletproofing signal.
Number of eval subagent sessions run: 10 (3 baseline + 3 treatment on Scenario 1; 1 baseline + 2 treatment pre-fix + 2 treatment post-fix on Scenario 2).
Before/after: the discipline (stop + surface, never silently fix) went 0/3 → 3/3 on the multi-defect scenario, and the trivial-gap loophole went 0/2 → 2/2 after the REFACTOR edit. Detection was already strong in both conditions — the skill's value is entirely in routing the decision to the human instead of smuggling it.
Rigor
superpowers:writing-skillsand completed adversarial pressure testing (results above — one RED→GREEN→REFACTOR cycle that found and closed a real loophole)writing-plansedit only adds a new section + one clause; it does not reword existing tuned content.Human review
A note on the verifying-plan-coverage ↔ acceptance-criteria relationship (#1112)
If #1112's
acceptance-criteriaskill lands, the two compose cleanly: that skill produces testable ACs as the spec; this skill proves the plan covers them with zero divergence. They are not competing.