eval: boundary + plumbing scenarios for the SUP-333 skill-edit stack#11
Merged
Conversation
Four scenarios from the staff-review evidence program: - cost-session-timeout-boundary: the nothing-to-design exception must re-gate a security-consequential "one-liner" (session lifetime). - cost-remove-export-boundary: hedge-phrased deletion of working user-visible functionality must re-gate. - sdd-spec-context-consumed: SDD controller must paste plan-cited spec sections into subagent prompts (deterministic tool-arg-match on the dispatch args) — first functional coverage of the Spec Context plumbing. - writing-plans-no-spec-conversational: conversational requirements → plan header "none — requirements:", no fabricated spec citation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
arittr
added a commit
to obra/superpowers
that referenced
this pull request
Jun 11, 2026
…escription exceptions are authoritative (SUP-333 C) Consolidates the brainstorming exception with its routing-layer semantics, so this PR is independently mergeable (previously split across two stacked PRs whose intermediate state left the always- injected routing text contradicting the shipped description). brainstorming: the nothing-to-design exception, earned by a tripwire scan stated in one line before acting. Tripwires precede the permission (skimmers stop at "implement directly"); security-posture touches re-gate even with the exact value stated; requested deletions re-gate; rationalization table per writing-skills bulletproofing. Description 971/1024 chars, YAML-validated. using-superpowers: description-level exceptions are authoritative (compliance, not rationalization); doubt means invoke; only the description can define one; the skip must state its scan; flowchart routes the exempt path through the scan statement; <EXTREMELY-IMPORTANT> defers in one parenthetical. writing-skills: negative triggering conditions are scope (allowed, required at the description) vs workflow summaries (still forbidden) — prevents a future checklist pass from stripping the exception. Eval evidence (quorum): RED cost-checkbox-over-trigger failed 5/6 agents (pi ⊘); GREEN claude 3/3, codex ✓, antigravity ✓ (kimi unchanged from baseline — does not read description exceptions); gate-still-fires: brainstorming-resists 2/2 + codex, spec-plan brainstorm leg 3/3. Boundary scenarios (security one-liner, requested deletion): pre-stack dev baseline 0/3 + 0/3 (silent edit every time — the blanket gate never fired on one-liners); this text 2/3 + 2/3, the first text in the corpus to catch these at any rate; scenarios ship as regression instruments (proposed in prime-radiant-inc/superpowers-evals#11, open). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
arittr
added a commit
to obra/superpowers
that referenced
this pull request
Jun 11, 2026
…escription exceptions are authoritative (SUP-333 C) Consolidates the brainstorming exception with its routing-layer semantics, so this PR is independently mergeable (previously split across two stacked PRs whose intermediate state left the always- injected routing text contradicting the shipped description). brainstorming: the nothing-to-design exception, earned by a tripwire scan stated in one line before acting. Tripwires precede the permission (skimmers stop at "implement directly"); security-posture touches re-gate even with the exact value stated; requested deletions re-gate; rationalization table per writing-skills bulletproofing. Description 971/1024 chars, YAML-validated. using-superpowers: description-level exceptions are authoritative (compliance, not rationalization); doubt means invoke; only the description can define one; the skip must state its scan; flowchart routes the exempt path through the scan statement; <EXTREMELY-IMPORTANT> defers in one parenthetical. writing-skills: negative triggering conditions are scope (allowed, required at the description) vs workflow summaries (still forbidden) — prevents a future checklist pass from stripping the exception. Eval evidence (quorum): RED cost-checkbox-over-trigger failed 5/6 agents (pi ⊘); GREEN claude 3/3, codex ✓, antigravity ✓ (kimi unchanged from baseline — does not read description exceptions); gate-still-fires: brainstorming-resists 2/2 + codex, spec-plan brainstorm leg 3/3. Boundary scenarios (security one-liner, requested deletion): pre-stack dev baseline 0/3 + 0/3 (silent edit every time — the blanket gate never fired on one-liners); this text 2/3 + 2/3, the first text in the corpus to catch these at any rate; scenarios ship as regression instruments (proposed in prime-radiant-inc/superpowers-evals#11, open). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Canary-caught addition: on the assembled text, triggering-writing-plans went 0/3 with claude citing "your explicit instruction wins per the priority rules" to skip writing-plans under the scenario's "don't ask me any questions" pressure — the Instruction Priority section read as licensing ad-hoc pressure to skip workflow steps. User Instructions now distinguishes pressure phrasing (changes interaction style) from instructions that name what to skip (honored), and tags the quoted rationalization.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four scenarios from the staff-review evidence program (see obra/superpowers#1715–#1732 stack): two brainstorming-exception boundary probes (security one-liner, hedge-phrased deletion), the SDD Spec Context plumbing check (deterministic tool-arg-match on dispatched prompts), and the writing-plans no-spec conversational path. First-run results recorded on the superpowers PRs. Known calibration debt: sdd-spec-context-consumed's AC requires SDD dispatch but the plan boilerplate sanctions executing-plans too; cost-remove-export-boundary's consequence examples read as a checklist to the grader.
🤖 Generated with Claude Code