Skip to content

fix(skills): description-level exceptions are authoritative in the routing rule#1732

Open
arittr wants to merge 2 commits into
drew/sup-333-3-brainstorming-triviality-gatefrom
drew/sup-333-4-description-exceptions-authoritative
Open

fix(skills): description-level exceptions are authoritative in the routing rule#1732
arittr wants to merge 2 commits into
drew/sup-333-3-brainstorming-triviality-gatefrom
drew/sup-333-4-description-exceptions-authoritative

Conversation

@arittr

@arittr arittr commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #1718 (brainstorming gate exception) — top of the SUP-333 stack (#1715#1716#1718 → this). Targets dev per the template requirement.

Who is submitting this PR? (required)

Field Value
Your model + version Claude Fable 5 (claude-fable-5[1m])
Harness + version Claude Code 2.1.169
All plugins installed superpowers (this repo, dev checkout); quorum eval lab (superpowers-evals) as the testing apparatus; unrelated local ops plugins (decision-log, episodic-memory, superpowers-chrome, primeradiant-ops)
Human partner who reviewed this diff Drew Ritter (@drewritter)

What problem are you trying to solve?

An adversarial review fleet (three parallel reviewers: red-team, cross-corpus consistency, evidence verification) audited the SUP-333 stack and found that using-superpowers' routing rule contradicts #1718's description-level exception in both directions:

  • Compliant agents re-impose the cost failure: "even a 1% chance a skill might apply → ABSOLUTELY MUST invoke" mandates invoking brainstorming for any trivial request (there is always ≥1% doubt), and the invocation itself is the measured cost event. "If an invoked skill turns out to be wrong, you don't need to use it" even endorses invoke-then-exit — the exact behavior the eval fails.
  • Cost-optimizing agents get a free skip: "the skill is overkill" is a tabled, forbidden rationalization, but "the skill's own description says it doesn't apply" was unaddressed — leaving any skip arguably sanctioned with no counter.
  • "Instructions say WHAT, not HOW. 'Add X' … doesn't mean skip workflows" names the exception's canonical case ("add a basic checkbox") as a violation.

What does this PR change?

Adds one paragraph to The Rule: a documented exception in a skill's own description is authoritative — not invoking is compliance, not rationalization; any doubt about the exception's conditions means invoke; only the skill's description can define such an exception (agents cannot infer one). The "Add X doesn't mean skip workflows" line gains the matching qualifier.

Is this change appropriate for the core library?

Yes — it defines how the routing layer treats description-level scoping for ALL skills, which #1718 introduces and future skills may use. Without it the bootstrap text and skill descriptions give contradictory instructions.

What alternatives did you consider?

  1. Leave it unreconciled — rejected: both failure directions are live (the eval demonstrated the compliant-agent direction; the red-team demonstrated the free-skip direction).
  2. Weaken the 1% rule itself — rejected: the rule is correct for the 99% case; the fix scopes it rather than weakens it ("any doubt means invoke" is preserved verbatim inside the new paragraph).
  3. Put the reconciliation in brainstorming instead — rejected: the contradiction lives in the routing layer's text; future description-level exceptions would hit it again.

Does this PR contain multiple unrelated changes?

No — one rule, two coordinated touches in one file (the rule paragraph and the User Instructions line that contradicted it).

Existing PRs

Environment tested

Harness (e.g. Claude Code, Cursor) Harness version Model Model version/ID
Claude Code (agent under test) 2.1.169 Claude Opus claude-opus-4-8
opencode (agent under test) 1.16.2 GPT openai/gpt-5.5

New harness support (required if this PR adds a new harness)

N/A — no harness changes.

Evaluation

  • Initial prompt: quorum eval scenarios with scripted naive users (no skills named by the driver).
  • Eval sessions after the change: the full 5-run verification battery ran on the assembled stack including this change: cost-checkbox-over-trigger/claude pass (the exception fires through the routing layer — no brainstorming invocation), triggering-writing-plans/claude pass (the 1% rule still triggers skills that DO apply), cost-spec-plan-duplication/claude pass (brainstorming still gates real features), cost-trivial-task-review-fanout/opencode pass, sdd-rejects-extra-features/claude pass (run IDs in the stack PRs).
  • Before/after: before, the routing text and the description exception contradicted; after, both eval directions hold simultaneously — skip fires only for the documented exception, triggers fire everywhere else.

Rigor

  • If this is a skills change: I used superpowers:writing-skills and completed adversarial pressure testing (paste results below)
  • This change was tested adversarially, not just on the happy path
  • I did not modify carefully-tuned content (Red Flags table, rationalizations, "human partner" language) without extensive evals showing the change is an improvement

The Red Flags table is untouched; the new paragraph closes the "description exempts me" gap the table could not cover, with the doubt-means-invoke backstop preserved. Adversarial findings (D1/D2) and both exploit directions are documented above; the 5-run battery is the post-change evidence.

Human review

  • A human has reviewed the COMPLETE proposed diff before submission

Round 3: staff-review refinements + evidence

Refinements in this PR's follow-up commit: the skill_flow digraph now routes through a "Skill's own description exempts this request?" diamond (no/any-doubt → invoke) — this stack's own evidence says agents follow flowcharts literally, and the chart previously contradicted the rule; the <EXTREMELY-IMPORTANT> block gains a one-line deferral to The Rule (previously it read unconditional, contradicting the rule in the same always-loaded file); writing-skills now distinguishes negative triggering conditions (scope — allowed, and required at the description per this rule) from workflow summaries (still forbidden), so a future editor applying its checklist does not strip the exception and silently regress the cost evals.

Final-text evidence: the exception routes correctly where supported — cost-checkbox-over-trigger skip: claude 3/3, codex ✓, antigravity ✓ (kimi does not pick up description exceptions; unchanged from baseline). The 1% rule still triggers skills that apply: triggering-writing-plans/claude 3/3 pass; ×codex fail — byte-for-byte its pre-existing documented signature (loads sibling skills, skips the mandated one; predates this stack, tracked separately).

Merge guidance: merge together with #1718 (see its note).

@arittr

arittr commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator Author

arittr and others added 2 commits June 10, 2026 18:54
…uting rule (SUP-333 #4)

Adversarial review findings D1/D2: the 1%-chance invocation rule and
the "Add X doesn't mean skip workflows" line contradicted the new
brainstorming description exception in both directions — a compliant
agent re-imposes the cost failure (invocation itself is the measured
cost event), while a cost-optimizing agent could treat any skip as
sanctioned. The routing skill now states: a documented exception in a
skill's own description defines that skill's scope (compliance, not
rationalization); any doubt about the exception's conditions means
invoke; and only the description can define one — agents cannot infer
exceptions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ception rule; writing-skills carve-out

Staff-review findings (4-reviewer panel):
- The skill_flow digraph still routed "yes, even 1%" straight to
  invoke with no exception branch — and this stack's own evidence says
  agents follow flowcharts literally. The flow now passes through
  "Skill's own description exempts this request?" with no/any-doubt →
  invoke.
- The <EXTREMELY-IMPORTANT> block ("you cannot rationalize your way
  out of this") read unconditional; one parenthetical defers to The
  Rule's single carve-out without weakening the block.
- Trimmed the redundant "the description defines the skill's scope"
  clause from The Rule paragraph.
- writing-skills' "descriptions must not carry process" doctrine would
  have had a future editor strip the brainstorming exception and
  silently regress the cost evals; it now distinguishes negative
  triggering conditions (scope — allowed and, per the routing rule,
  required at the description) from workflow summaries (still
  forbidden).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arittr arittr force-pushed the drew/sup-333-3-brainstorming-triviality-gate branch from aff9195 to 87ddfac Compare June 11, 2026 02:16
@arittr arittr force-pushed the drew/sup-333-4-description-exceptions-authoritative branch from 9d0ac38 to 36e289e Compare June 11, 2026 02:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant