fix(skills): description-level exceptions are authoritative in the routing rule#1732
Closed
arittr wants to merge 15 commits into
Closed
Conversation
…#1) writing-plans told agents to "document everything they need to know" assuming zero context — every agent in the 2026-06-09 six-agent quorum sweep obeyed and restated the entire spec inline in the plan (cost-spec-plan-duplication failed 5/5 completed agents; pi's plan was 683 lines of duplicated spec). - writing-plans: state the division of labor — spec owns WHAT/WHY, plan owns HOW; cite the spec by path/section, never restate it. "Zero context" means mechanically executable steps, not duplication. Add a **Spec:** line to the plan header template. - brainstorming: close the path loophole the re-run exposed — claude shortened docs/superpowers/specs/ to docs/specs/ in 2/2 runs; both path mentions now explicitly forbid the shortening. TDD evidence (quorum): - RED: batch-20260609T023452Z-68aa et al — 5/5 agents fail - GREEN: cost-spec-plan-duplication-claude-20260609T234142Z-9625 pass (plan: "this plan does not restate them" + spec cited by path; both docs in docs/superpowers/) - Canary: triggering-writing-plans-claude pass (skill still fires) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… reference rule
Adversarial review findings (C1, C2, C3, C5, A8, F3):
- "never restate" did not cover paraphrase/summary — the actual failure
mode in the RED evidence; now "never restate, paraphrase, or summarize".
- The No Placeholders intra-plan repetition mandate gave a symmetric
argument for re-inlining the spec; the rule now draws the line:
repetition WITHIN the plan is required, copying FROM the spec is not.
- Drift argument was invertible ("snapshot to avoid drift"); now states
snapshots hide drift.
- **Spec:** header gets a no-spec branch (state requirements once in
the header, not per task) instead of inviting "no spec, rule is moot".
- Brainstorming path bullet: an existing differently-named docs dir is
not a "user preference" override.
- Execution Handoff now notes review fanout scales (forward-ref to
SDD's Proportionality rule) instead of promising unconditional
two-stage review.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Eval-caught regression: the no-spec branch added to the **Spec:**
header gave the agent a sanctioned path to skip the spec doc entirely
("avoiding duplication by skipping the spec" —
cost-spec-plan-duplication-claude-20260610T213934Z-8e5b, fail). The
branch is now scoped: if brainstorming happened the spec exists and
must be cited; "none — requirements:" applies only when requirements
arrived conversationally and no spec doc was ever produced. The
reference-discipline paragraph states the same rule up front.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
This change is part of the following stack: Change managed by git-spice. |
Draft
5 tasks
…ting-plans spec gap Staff-review findings (4-reviewer panel): - Reference paragraph rewritten 170→123 words preserving every behavioral condition (paraphrase/summarize coverage, no-skip guard, WHAT-WHY/HOW split, No Placeholders boundary, drift counter, zero-context rescope); fixes the "(brainstorming did)" syntax. - **Spec:** header bracket: cut the never-skip sermon duplicated from the Overview (same loaded document); the conditional none-branch stays. - executing-plans Step 1 now reads the spec the plan cites — plans are no longer self-contained, and the non-subagent execution path was never told (the eval only exercised the SDD consumer). - writing-plans plan-location preference line gets the same existing-dir-is-not-a-preference guard as the spec path. - brainstorming: deduplicate the docs/specs/ prohibition (step 6 parenthetical stays; After-the-Design bullet was the second statement in one file). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
subagent-driven-development mandated implementer + two-stage review + final reviewer unconditionally — agy and opencode each dispatched 4 subagents for a one-line console.log in the 2026-06-09 quorum sweep (cost-trivial-task-review-fanout), and the agents that passed did so only by disobeying the skill. - Proportionality rule: when the entire plan is one trivial, fully-specified mechanical change, implement directly, verify, commit — no review fanout. "When in doubt, it is not trivial." Within a multi-task plan the full pipeline still applies to every task regardless of size. - Flowchart gets the trivial-exit diamond (the failing agents follow the flowchart literally; prose alone would not redirect them). - Red Flags "never skip reviews" amended to reference the exception so the skill does not contradict itself. TDD evidence (quorum): - RED: agy 025324Z + opencode batches — 4 dispatches for 1 line - GREEN: cost-trivial-task-review-fanout-opencode-20260610T002518Z-f3f5 pass — 0 dispatches, $0.04, change landed on main checkout - Canary: sdd-rejects-extra-features-claude-20260610T002901Z-458a pass — multi-task plan still runs implementer + two-stage review per task (tool-called Agent ✓, spec reviewer as YAGNI gate after each task) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…es cited specs Adversarial + consistency review findings (B1, B2, B3, B5, F1): - Red Flags line read literally licensed skipping reviews on trivial tasks INSIDE multi-task plans; now states the only exception is a whole-plan trivial change and never-skip within multi-task plans. - "a one-line edit" example blessed one-line behavioral changes (e.g. adding "|| user.isOwner"); dropped. Trivial is now defined as a property of the diff (no logic/control-flow/behavior change), not of the plan's self-description. The "nothing for review to catch" justification proved too much; replaced with the cost argument. - "verify it" was undefined on the trivial path; now concrete (run tests/command, confirm output, verification-before-completion). - Flowchart diamond now matches the prose: "fully-specified" + "any doubt = no" (the failing agents execute the flowchart literally). - New Spec Context section + prompt-template updates: the controller reads the spec cited in the plan header and pastes cited sections into implementer/spec-reviewer prompts; the spec reviewer's diff-only rule gets a spec-document exception. Without this, the stack's reference-not-restate rule starves the SDD pipeline of requirements. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ualify constant bumps Staff-review findings (4-reviewer panel): - CONTRADICTION FIX: Spec Context said "Subagents never read the spec file themselves" while spec-reviewer-prompt grants exactly that access. Now: implementers never read it; the spec reviewer may, at the cited path. - "a constant bump" was an unqualified trivial example — a one-line BCRYPT_ROUNDS or session-TTL change is a security-posture change; now qualified "with no security or behavioral consequences" (matching brainstorming's config-change qualifier). The diff-property definition adds "nothing security-relevant". - Proportionality rewritten 146→~115 words (house style; one statement of the multi-task containment instead of two). - Red Flags Never-line trimmed 33→14 words (pointer to Proportionality instead of third in-file restatement). - Prompt-template rationale tails cut (the controller just read Spec Context; subagents need the pasted text, not the policy rationale). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…gn (SUP-333 #3) The HARD-GATE ("EVERY project regardless of perceived simplicity") plus the anti-pattern list naming "a config change" made design+approval mandatory even for fully-specified trivial asks — all 6 agents in the 2026-06-09 quorum sweep ran a multi-option design flow for "a basic checkbox, nothing fancy" (cost-checkbox-over-trigger failed 6/6). Two layers, because routing happens before skill content is read (GREEN attempt 1 proved it: the agent invoked the skill on the description's mandate and only then saw the in-skill exception, and the invocation itself is the cost event): - description: carve-out visible at skill-selection time — zero open design decisions, fully specified trivial change → implement directly without invoking. - HARD-GATE: matching exception with objective re-gating tripwires (new file/dependency, schema/API/data question, >1 plausible interpretation, user frames it as a feature/project), and the anti-pattern section now distinguishes "seems simple" (a rationalization when decisions exist) from "contains every decision" (the exception). "A config change" moves from the all-of-them list to the exception's example. The repo's acceptance test ("Let's make a react todo list" must auto-trigger brainstorming) is unaffected: a react todo list leaves many decisions open and todo lists remain in the anti-pattern list. TDD evidence (quorum): - RED: cost-checkbox-over-trigger fails 6/6 agents (batch 2026-06-09); GREEN attempt 1 with in-skill exception only: still fail (invoked via description, then asked a clarifying question) - GREEN: cost-checkbox-over-trigger-claude-20260610T004320Z-a30e pass — no brainstorming invocation, agent cited the exception verbatim, checkbox landed in 31s - Canary: cost-spec-plan-duplication-claude-20260610T004506Z-22ea pass — a real feature still triggers the full brainstorm→spec→plan flow (and the stacked writing-plans reference discipline holds) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…+ rationalization counters
Adversarial review findings (A1-A7, D3):
- BLOCKER A1: the re-gating tripwires lived only in the HARD-GATE, but
the skip decision happens at the description (our own GREEN-attempt-1
evidence). The description now carries the tripwires: adds a
file/dependency, touches schema/API/persisted data, deletes or
disables anything, alters behavior/security posture, >1 plausible
reading.
- A2: "a schema/API/data question" was defeated by "the user answered
the question"; now touch-based ("even if the user stated the desired
outcome").
- A3: destructive changes and behavior/security-visible changes had no
tripwire (pure removals were structurally invisible); both added.
"a literal config value change" example now qualified ("with no
security or behavioral consequences").
- A4: the checkbox example no longer teaches hedge-phrase = fully
specified ("where the context leaves nothing to choose").
- A5: "EVERY project regardless of perceived simplicity" now ends
"with exactly one exception below" instead of contradicting it.
- A6: rationalization table added (codebase-pattern, infer-the-obvious,
hedge-phrase, asking-wastes-time).
- A7: anti-pattern opener is a claim again ("Anything with open
decisions goes through this process").
- D3: exception states TDD and verification-before-completion still
apply, so the fast path does not read as zero-oversight.
Description: 689 chars (limit 1024), YAML-validated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion Staff-review findings (4-reviewer panel): - The tripwire list existed twice in this file (description + HARD-GATE) and the copies had already drifted after one editing round — the framing tripwire and the security qualifier lived only in the HARD-GATE, which the skip decision never reads (our own GREEN-attempt-1 evidence). The description is now the single authoritative list; the HARD-GATE exception defers to it. - Security-posture fix: the "beyond the literally stated value" escape no longer applies to security — touching auth, sessions, permissions, CORS, or crypto re-gates EVEN when the value is exactly as stated (the harm of "set CORS to *" IS the stated value). User-visible behavior keeps the beyond-the-stated-change scope (a requested checkbox is the stated change; that is the point of the exception). - The framing tripwire moves into the description where it can act. - Anti-pattern final clause cut (was the 4th in-file statement of the exception's condition). - Description: 886/1024 chars, YAML-validated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Eval-caught leak (cost-remove-export-boundary-claude, first run): the
agent reasoned "the user already decided the deletion, so no design
decision is open" and silently removed a working feature — reading the
tripwires as indicators of open decisions rather than unconditional
re-gates. The deletion tripwire now carries the same rider as the
security one ("even when the deletion is exactly what was asked"), and
the rationalization table counters the exact quoted escape.
Description: 950/1024 chars.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aff9195 to
87ddfac
Compare
9d0ac38 to
36e289e
Compare
This was referenced Jun 11, 2026
…e stated
Measured boundary leak (cost-session-timeout-boundary: claude surfaced
the security tradeoff 1/3; cost-remove-export-boundary 1/2 post-rider):
in failing runs the agent acted without consulting the exception at
all. Two prompt defects fixed:
- Ordering: the description granted "implement it directly" BEFORE the
tripwire list — a skimming agent got the permission and stopped
reading. The tripwires now come first and the permission is earned
("Only when NO tripwire hits...").
- Observability: the skip was silent. It now requires a stated
one-line scan before implementing, which forces the scan to actually
happen (the routing-layer mandate lands in the companion
using-superpowers commit).
"timeouts" added to the security examples — the literal failing case.
Description: 971/1024 chars, YAML-validated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…uting rule (SUP-333 #4) Adversarial review findings D1/D2: the 1%-chance invocation rule and the "Add X doesn't mean skip workflows" line contradicted the new brainstorming description exception in both directions — a compliant agent re-imposes the cost failure (invocation itself is the measured cost event), while a cost-optimizing agent could treat any skip as sanctioned. The routing skill now states: a documented exception in a skill's own description defines that skill's scope (compliance, not rationalization); any doubt about the exception's conditions means invoke; and only the description can define one — agents cannot infer exceptions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ception rule; writing-skills carve-out
Staff-review findings (4-reviewer panel):
- The skill_flow digraph still routed "yes, even 1%" straight to
invoke with no exception branch — and this stack's own evidence says
agents follow flowcharts literally. The flow now passes through
"Skill's own description exempts this request?" with no/any-doubt →
invoke.
- The <EXTREMELY-IMPORTANT> block ("you cannot rationalize your way
out of this") read unconditional; one parenthetical defers to The
Rule's single carve-out without weakening the block.
- Trimmed the redundant "the description defines the skill's scope"
clause from The Rule paragraph.
- writing-skills' "descriptions must not carry process" doctrine would
have had a future editor strip the brainstorming exception and
silently regress the cost evals; it now distinguishes negative
triggering conditions (scope — allowed and, per the routing rule,
required at the description) from workflow summaries (still
forbidden).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measured boundary leaks (session-timeout 1/3, remove-export 1/2 on
claude) traced to silent skips: the agent acted without consulting the
exception, and nothing made the consultation observable. The skip now
requires a stated one-line scan before the first action ("Skipping
brainstorming per its exception: no security/deletion/schema/new-file
tripwires; outcome fully specified") — externalizing the scan forces
it to happen. Flowchart routes the exempt path through the scan
statement; red-flags table counters "too trivial to scan".
The trivial path stays fast: the ceremony is one sentence, not a
design flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
36e289e to
a96fb01
Compare
6535b2d to
7857e05
Compare
Collaborator
Author
|
Consolidated into #1718 per independent-mergeability restructure: the routing-rule change and the brainstorming description exception are one behavioral mechanism (the stated-scan protocol spans both files), and shipping them separately left an active contradiction window between merges. #1718 now carries both, plus the full eval evidence. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Who is submitting this PR? (required)
claude-fable-5[1m])devcheckout); quorum eval lab (superpowers-evals) as the testing apparatus; unrelated local ops plugins (decision-log, episodic-memory, superpowers-chrome, primeradiant-ops)What problem are you trying to solve?
An adversarial review fleet (three parallel reviewers: red-team, cross-corpus consistency, evidence verification) audited the SUP-333 stack and found that using-superpowers' routing rule contradicts #1718's description-level exception in both directions:
What does this PR change?
Adds one paragraph to The Rule: a documented exception in a skill's own description is authoritative — not invoking is compliance, not rationalization; any doubt about the exception's conditions means invoke; only the skill's description can define such an exception (agents cannot infer one). The "Add X doesn't mean skip workflows" line gains the matching qualifier.
Is this change appropriate for the core library?
Yes — it defines how the routing layer treats description-level scoping for ALL skills, which #1718 introduces and future skills may use. Without it the bootstrap text and skill descriptions give contradictory instructions.
What alternatives did you consider?
Does this PR contain multiple unrelated changes?
No — one rule, two coordinated touches in one file (the rule paragraph and the User Instructions line that contradicted it).
Existing PRs
Environment tested
New harness support (required if this PR adds a new harness)
N/A — no harness changes.
Evaluation
cost-checkbox-over-trigger/claude pass (the exception fires through the routing layer — no brainstorming invocation),triggering-writing-plans/claude pass (the 1% rule still triggers skills that DO apply),cost-spec-plan-duplication/claude pass (brainstorming still gates real features),cost-trivial-task-review-fanout/opencode pass,sdd-rejects-extra-features/claude pass (run IDs in the stack PRs).Rigor
superpowers:writing-skillsand completed adversarial pressure testing (paste results below)The Red Flags table is untouched; the new paragraph closes the "description exempts me" gap the table could not cover, with the doubt-means-invoke backstop preserved. Adversarial findings (D1/D2) and both exploit directions are documented above; the 5-run battery is the post-change evidence.
Human review
Round 3: staff-review refinements + evidence
Refinements in this PR's follow-up commit: the
skill_flowdigraph now routes through a "Skill's own description exempts this request?" diamond (no/any-doubt → invoke) — this stack's own evidence says agents follow flowcharts literally, and the chart previously contradicted the rule; the<EXTREMELY-IMPORTANT>block gains a one-line deferral to The Rule (previously it read unconditional, contradicting the rule in the same always-loaded file); writing-skills now distinguishes negative triggering conditions (scope — allowed, and required at the description per this rule) from workflow summaries (still forbidden), so a future editor applying its checklist does not strip the exception and silently regress the cost evals.Final-text evidence: the exception routes correctly where supported —
cost-checkbox-over-triggerskip: claude 3/3, codex ✓, antigravity ✓ (kimi does not pick up description exceptions; unchanged from baseline). The 1% rule still triggers skills that apply:triggering-writing-plans/claude 3/3 pass; ×codex fail — byte-for-byte its pre-existing documented signature (loads sibling skills, skips the mandated one; predates this stack, tracked separately).Merge guidance: merge together with #1718 (see its note).
Round 4: stated-scan mandate
The exception-skip path now requires a visible artifact: one line naming the exception and the empty tripwire scan, written before the first action ("If you did not write the scan line, you did not scan — invoke the skill instead"). The flowchart's exempt edge routes through the scan statement; the red-flags table counters "too trivial to scan". Rationale: the measured boundary leaks were silent non-consultation — externalizing the scan forces it to happen, at a cost of one sentence on the trivial path. Measured effect recorded on #1718 (boundary cells 1/3 → 2/3 with the trivial path intact at 2/2).