Skip to content

security-audit.md

Latest commit

 

History

History
257 lines (165 loc) · 36.7 KB

security-audit.md

File metadata and controls

257 lines (165 loc) · 36.7 KB
Raw

Security Audit

  • Owns how the security specs are audited: the schedule and the release gate, the deterministic checks and the skip they allow, the domains and their prompts, the orchestration contract, the three outcomes, the reporting steps, the embargo on finding detail and its disclosure, the environment that holds AUDIT_PAT, the canary that measures recall, the second-opinion audit on Codex, and the red team.
  • Defers what is audited to docs/specs/security.md and the specs it names, and each agent's procedure to its prompt in .github/audit/.
  • Read docs/specs/security.md first.

Schedule and gate

.github/workflows/security-audit.yaml audits docs/specs/security.md and the specs it names: nightly, on workflow_dispatch, and on the release tag, dispatched by .github/workflows/release.yml whose publish-vscode job needs: it — so no release ships without a passing audit. A second, weekly schedule line runs the canary instead (Canary recall). Dispatched, not uses:-called — see docs/specs/security-ci.md -> "GitHub Actions Policies".

FAIL IF lines are grouped by the operation that answers them, one bullet asserting several properties when a single call establishes all of them; a qualitative pass looks for holes the specs do not cover. On any FAIL IF violation or BLOCKER-severity finding the audit fails (Outcomes and reporting).

  • FAIL IF S1 .github/audit/_preamble.md stops requiring every FAIL IF to run as a mechanical check (gh api, grep, file read, or a script run) with evidence, each clause its own PASS/FAIL and never satisfied in bulk, or a domain prompt drops its qualitative pass.
  • FAIL IF S2 .github/workflows/security-audit.yaml is missing or disabled, or any of the three separate things that make it a release gate is gone: the gh workflow run dispatch, the gh run watch --exit-status that turns a failed audit into a failed job, and publish-vscode's needs: edge on that job (rationale). Pinned by scripts/github-state-check.mjs (disabled) and scripts/security-audit.test.mjs.

Deterministic checks

Every FAIL IF that GitHub state answers is judged by scripts/github-state-check.mjs, not a model (rationale). The Check GitHub state step runs it on AUDIT_PAT before the domains start; it enumerates rulesets, environments, and secrets from the API, compares them with .github/audit/expected-github-state.json, and writes audit-github-state.md in the fragment form .github/audit/_preamble.md fixes, one line per clause. That fragment rides AUDIT_FRAGMENTS, so its verdict reaches the release gate through Outcomes and reporting and its detail follows Embargo like any domain's.

  • A domain never re-derives a clause the fragment answers; it reads the fragment as evidence and keeps the judgement bullets, the qualitative pass, and whether the expected-state file still encodes its spec.

  • A FAIL IF marked Pinned by a script or test in pnpm test is answered by running it, not by reading the code it checks.

  • The check reads a 403 under AUDIT_PAT as scope drift, a FAIL, any other failed call as UNVERIFIABLE, and never passes a clause it could not read; --local makes a 403 UNVERIFIABLE for an operator's own login.

  • FAIL IF S3 Check GitHub state is removed, runs after the agent, or its fragment leaves AUDIT_FRAGMENTS, or scripts/github-state-check.mjs stops failing a planted violation of any clause it judges. Pinned by scripts/security-audit.test.mjs and scripts/github-state-check.test.mjs.

Source of truth: judge in scripts/github-state-check.mjs; Check GitHub state in .github/workflows/security-audit.yaml.

Skipping an unchanged audit

A schedule run skips the domains when nothing they read changed: the last completed run on main succeeded, its audit-state artifact records this commit and this GitHub-state hash, and the full audit it descends from is under seven days old (rationale). The skipped run still runs the deterministic checks, writes an audit-report.md saying it skipped, and reports only audit-github-state.md.

  • FAIL IF S4 a workflow_dispatch run — the release gate's included — can skip, a run skips after a run that did not succeed or with a changed commit or hash, or a skipped run reports PASS over a GitHub-state fragment that did not pass. Pinned by scripts/security-audit.test.mjs.

Source of truth: decide in scripts/security-audit-plan.mjs.

Domains

The CI audit fans out to per-domain subagents, each owning a disjoint share of the tree; a domain may read outside its share — remote and local run the repo's own lints — but owns nothing there (rationale).

Ownership is by file: every docs/specs/security*.md spec, its .rationale.md aside, is in exactly one domain's scope, declared as backticked repo paths in the bullet list under the **Scope line of its domain file in .github/audit/, and enforced by scripts/spec-lint.mjs; dormouse.sh/security publishes the assignment as its audit index (docs/specs/website-docs.md -> "/security spec").

AUDIT_PAT is a step-level env: on the one job, so every subagent inherits it, and only the prompt tells remote, local, and hosted not to use it (docs/specs/security.md -> "Known gaps"); ## Future -> Credential separation stages the fix.

CI and the local runner share the prompts and their scopes in .github/audit/ (rationale).

The qualitative scopes are stated by subtraction, so adding a directory cannot orphan it (rationale). local takes the remainder of every other domain's claims, worked out from ls -A rather than from a list. Dotfile directories are named explicitly wherever they land, and the subtraction is recursive: where a domain claims a subdirectory rather than a whole tree, the rest of that tree belongs to local.

  • FAIL IF S5 a docs/specs/security*.md spec (not a .rationale.md) is in no domain's scope, or in two, or a scope names a file that does not exist (rationale).
  • FAIL IF S6 .github/audit/orchestrator.md lets the orchestrator audit anything itself (rationale).
  • FAIL IF S7 the audit stops fanning out to a dedicated remote subagent scoped to docs/specs/security-remote.md, a dedicated local subagent scoped to docs/specs/security-local.md, or a dedicated hosted subagent scoped to docs/specs/security-hosted.md, or any of those scopes is merged back into a context that also carries another domain (rationale).
  • FAIL IF S8 remote, local, or hosted does not run on Opus and the mechanical domains on Sonnet, the stronger model on the code-reading domains, in both .github/workflows/security-audit.yaml's claude_args — its --model sets the floor and its --agents raises those three domains — and scripts/security-audit-local.sh's default Claude engine (rationale). This split is the release gate; the Codex engine never is (Second-opinion audit). Pinned by scripts/security-audit.test.mjs.
  • FAIL IF S9 .github/audit/ is missing a prompt file the workflow names, or scripts/security-audit-local.sh stops running the audit from those same files (rationale), or exits zero after a failed process, a missing or unfinished fragment, or any verdict other than exact VERDICT: PASS. Pinned by scripts/security-audit.test.mjs.
  • FAIL IF S10 the union of the subagents' qualitative scopes does not cover every top-level path in the repository: the catch-all's remainder stops subtracting every other domain's prompt, or a second domain claims the remainder (rationale). Pinned by scripts/security-audit.test.mjs.
  • FAIL IF S11 .github/audit/ or .vscode/ is outside any consumer of .github/workflows/workflow-audit.yaml's diff window — the commit list, own_changes, is_clean_merge, and both content classifiers' refusals, whose half is derived from the single WINDOW array. The security specs are deliberately not watched there (rationale).

Source of truth: the **Scope and ## Qualitative pass sections of each domain prompt in .github/audit/; claude_args in .github/workflows/security-audit.yaml; run_domain in scripts/security-audit-local.sh.

Orchestration

Subagents launch in the background, so an agent that ends its turn to await a completion notification is finished: an orchestrator ends the whole run, a delegating domain ships what it has (rationale). .github/audit/orchestrator.md and .github/audit/_preamble.md own the procedure; these are the invariants the workflow depends on:

  • Each domain appends to its own fragment as it determines each result, and the fragments upload with the transcript, so an orchestrator that dies mid-merge still ships what the domains found.

  • A fragment opens VERDICT: INCONCLUSIVE and closes with the literal <!-- END OF REPORT -->, its verdict rewritten once at the end. The sentinel, not existence, is what a reader treats as finished (rationale).

  • FAIL IF S12 the orchestrator prompt stops requiring a non-turn-ending wait — a Bash until loop over the fragments' sentinels, breaking on its own sub-cap under the Bash cap the workflow sets so every call ends by printing its answer, re-issued under a bounded 32-minute deadline persisted to a file ($RUNNER_TEMP/audit-deadline) rather than recomputed from now. That cap is BASH_DEFAULT_TIMEOUT_MS in .github/workflows/security-audit.yaml, set above the loop's 540-second break (rationale).

  • FAIL IF S13 the audit job's timeout-minutes in .github/workflows/security-audit.yaml is not above the orchestrator's persisted wait deadline (rationale), or AUDIT_FRAGMENTS there stops naming every domain's fragment. Pinned by scripts/security-audit.test.mjs.

  • FAIL IF S14 the prompt permits ending the turn without audit-report.md (rationale).

  • FAIL IF S15 a domain prompt lets findings be held for a write-up at the end, lets a domain that delegates end its turn or background its wait loop, or the wait, the merge, or the verdict treats existence rather than the sentinel as a domain having reported (rationale).

  • FAIL IF S16 the orchestrator can report PASS while a subagent left no report fragment — nor FAIL, unless some domain actually returned one: the prompt writes no status file when a fragment is missing and no domain failed, routing an audit that ran out of time to INCONCLUSIVE. Both exit non-zero and hold the release gate shut (rationale).

Source of truth: .github/audit/orchestrator.md; the fragment contract in .github/audit/_preamble.md; the wait and merge blocks run as shipped in scripts/security-audit.test.mjs.

Outcomes and reporting

The reporting step distinguishes three outcomes, not two (rationale):

Outcome Computed when Result
PASS every domain computes PASS and audit-status.txt is literally PASS status issue rewritten; exit zero once the ledger was filed and read (Findings ledger)
FAIL any domain computes FAIL private issue filed; status issue rewritten; exit non-zero
INCONCLUSIVE anything else filed as for FAIL, both saying no verdict was reached; exit non-zero

Verdicts are computed from the fragments' lines, never read from a conclusion (rationale). scripts/security-audit-report.mjs parses each fragment in the grammar .github/audit/_preamble.md fixes:

  • FAIL: any FAIL result or BLOCKER finding, malformed or not.
  • INCONCLUSIVE: otherwise, an UNVERIFIABLE result, an owed rule with no result, a skipped clause letter, a malformed line other than an INFO anomaly (below), a result for a rule not owed, not exactly one QUALITATIVE: done line, no sentinel, a first line other than exact VERDICT: PASS, no open-findings list, or a PASS passing over an open finding (Findings ledger).
  • PASS: none of these.

An INFO off the grammar is an anomaly, never a doubt: a line opening exactly - INFO: whose remainder starts like no other line is kept verbatim in the private report and changes no verdict (rationale).

A domain owes one result per FAIL IF rule in the specs its **Scope claims, so a skipped section is a missing rule. A result, a ledger key, and a seed name a rule by its ID alone, placed by the spec and heading the checked-out specs give it wherever a reader needs them, so a rule moved or reworded re-keys nothing (rationale). A rule pinned to scripts/github-state-check.mjs alone is owed by audit-github-state.md instead, which owes every rule that pin names, no QUALITATIVE line, and no evidence lines on its findings. A verdict line or audit-status.txt that disagrees with the computed verdict is reported as an anomaly; a pessimistic one holds at INCONCLUSIVE, never raising to FAIL.

  • The private report is what did not pass: computed verdicts, anomalies, non-PASS results, missing rules, malformed lines, and merged findings. PASS lines and the orchestrator's audit-report.md stay in the encrypted artifact.

  • The private report is split, never truncated, by scripts/clamp-issue-body.mjs into the issue body and its comments, under a bounded part count (rationale).

  • The audit-transcript artifact uploads during postprocessing; runner timeout or cancellation can prevent upload, and a missing artifact receives no download link (rationale).

  • A FAIL IF names only a condition an audit run can read: the readable half is audited, and the rest is staged under ## Future only while it is unbuilt, otherwise stated beside the rule. AUDIT_PAT-readable GitHub state stays audited (rationale).

  • FAIL IF S17 the Redact secrets from agent output step is removed, stops covering any sink later archived or filed (audit-report.md, every AUDIT_FRAGMENTS fragment, and the transcript), or stops failing closed by deleting those files when the redactor itself throws (rationale). Pinned by scripts/security-audit.test.mjs.

  • FAIL IF S18 the reporting step takes a verdict from a VERDICT: line or audit-status.txt over the computed one, reads a stated verdict from a first line other than exact VERDICT: PASS, VERDICT: FAIL, or VERDICT: INCONCLUSIVE, a fragment computes PASS under any INCONCLUSIVE condition above, an INFO line, in the grammar or not, changes a verdict or opens a ledger issue, the manifest is read from anywhere but the claimed specs' FAIL IF lines or comes back empty for a fragment or spec it cannot resolve, or a builder that throws hands on anything but INCONCLUSIVE (rationale).

  • FAIL IF S19 the private report omits a note for a condition that holds — a failing, missing, unreadable, cut-off, or inconclusive domain, an anomaly, or no status — or any note asserts something about a condition other than its own, or the status issue's domain table omits one (rationale).

  • FAIL IF S20 the audit has been weakened in a way no bullet above names — e.g. the prompt no longer requires the qualitative pass, a FAIL IF can be ignored, the reporting step that rewrites the status issue and exits non-zero has been removed, or the AUDIT_PAT pre-check is removed or bypassed. This bullet is a judgement item, not a checklist.

The reporting step's known gaps are docs/specs/security.md -> "Known gaps" (rationale).

Source of truth: Compose the audit report and Publish the audit status in .github/workflows/security-audit.yaml; scripts/security-audit-report.mjs.

Findings

A BLOCKER fails the run like a failed FAIL IF; the severity rubric and evidence rules are .github/audit/_preamble.md's (rationale). Findings naming the same file and root cause within five lines are one finding, at the worst severity reported, across domains. Delegation follows each domain prompt's ## Work streams, one qualitative stream among them, the domain's only qualitative pass.

  • FAIL IF S21 .github/audit/_preamble.md stops rating BLOCKER by attacker-controlled input reaching code execution, a credential, or an authorization grant, or stops requiring every BLOCKER and WARNING to quote its code, trace source to a named sink, and give a reproduction, or a BLOCKER or WARNING lacking those lines can reach PASS (rationale).
  • FAIL IF S22 the private report stops merging findings by file, root cause, and a five-line window, or the ledger files one file and root cause as two open issues.
  • FAIL IF S23 a domain prompt's ## Work streams omits or repeats a heading its manifest owes, or .github/audit/_preamble.md lets a domain delegate by any other partition or run more than one qualitative pass (rationale).
  • FAIL IF S24 .github/audit/_preamble.md stops requiring a PASS to trace each input a gate compares to whether the rule's attacker sets it, to judge a configuration's omissions by their actual fallback and take its failure and default paths, to keep every condition of a split rule, and to re-verify the open findings citing its rule (rationale).

Source of truth: .github/audit/_preamble.md; dedupFindings in scripts/security-audit-report.mjs.

Embargo

Finding detail stays private until Disclosure publishes what of it may cross (rationale):

Sink Carries
Second-opinion comment on the status issue, one per codex-audit-report.yaml run what Second-opinion audit lists
codex-recall artifact that run's canary counts, model, commit, and the two run links
Public status issue, labeled security-audit-status the run's time, link, commit, and outcome; whether the domains were skipped and when they last ran; each domain's computed verdict and its failed-check, missing-rule, malformed-line, BLOCKER, and WARNING counts; each failed check's spec and heading; how many verdict lines disagreed; whether the private filing failed; each open ledger finding's severity, kind, and dates, keyed as Disclosure allows; the latest disclosures
Public disclosure issue, labeled security-audit-finding, one per ledger issue what Disclosure lists
Private issue in diffplug/dormouse-embargo, one per non-PASS run the private report (Outcomes and reporting), titled with date and commit
Ledger issue in diffplug/dormouse-embargo, one per open finding key one failed rule, its clauses together, or one merged BLOCKER or WARNING, with its evidence
audit-open-findings.txt, on the runner the open ledger titles, leaving the runner only in the transcript
audit-ledger-snapshot.jsonl, on the runner what Read the findings ledger keeps of each ledger issue (below)
audit-transcript artifact age ciphertext of the transcript, audit-report.md, and the fragments
audit-state artifact the audited commit, the GitHub-state hash, whether the domains ran, and when they last did
Check GitHub state job log the GitHub-state verdict, its clause and WARNING counts, and the hash
List open findings and Read the findings ledger job logs how many ledger issues they read

EMBARGO_TOKEN is a fine-grained PAT with Issues: write on diffplug/dormouse-embargo alone, minted on an admin's account.

  • FAIL IF S25 a public issue can carry finding text: a status or disclosure title, body, or comment is posted from anything but scripts/security-audit-public-body.mjs and the step's fixed failure sentence, or that builder emits anything but fixed text, counts, dates, validated run metadata and model names, fragment names, ledger keys and severities, headings it found in the checked-out spec, files it found in the checked-out tree, PR numbers the API reports merged, and a maintainer's public summary. A failed check naming no rule its domain owes, in the line form .github/audit/_preamble.md fixes, is counted, never quoted (rationale).
  • FAIL IF S26 secrets.EMBARGO_TOKEN is referenced anywhere but the env: of List open findings, File embargoed findings, and Read the findings ledger, a workflow or job env: or a $GITHUB_ENV write could carry it to another step, any of them runs anything but gh and shell builtins, List open findings writes a title anywhere but audit-open-findings.txt, or Read the findings ledger writes anywhere but audit-ledger-snapshot.jsonl or keeps a title past its key and severity, a body past its opening metadata line, or a comment past a maintainer's Fixed in PR number or public summary (rationale). Pinned by scripts/security-audit.test.mjs.
  • FAIL IF S27 a non-PASS run whose private filing failed or was skipped can succeed, or its status issue omits that the filing failed; or a PASS run files anything in the private tracker but ledger issues, or scripts/security-audit-local.sh files anything.
  • FAIL IF S28 the audit-transcript artifact can upload anything but age ciphertext: its path: names a non-.age file, it uploads when Encrypt the audit transcript did not succeed, that step reads its recipient from anywhere but .github/audit/transcript-recipient.txt, or it stops running whatever the audit step's outcome (if: always()).
  • FAIL IF S29 diffplug/dormouse-embargo is publicly visible: an unauthenticated curl -s -o /dev/null -w '%{http_code}' https://github.com/diffplug/dormouse-embargo must print 404 (rationale). Pinned by scripts/github-state-check.mjs.

That the agent shares its job with these steps is docs/specs/security.md -> "Known gaps".

Source of truth: .github/workflows/security-audit.yaml; scripts/security-audit-public-body.mjs.

Findings ledger

Every run reads the ledger, a PASS included, and only a human closes a ledger issue (rationale). A ledger issue's title starts [audit-finding <key>]: a failure's key hashes its rule's ID; a finding's, its normalized file and the root-cause symbol it names, or its rule's ID where it names none (rationale). A key with no open issue gets one, a PASS's WARNINGs included, an INFO never; one already open gets a Seen again comment. Any open [audit-finding …] issue counts, a hand-filed manual-… key included.

A filed ledger issue's body opens with one metadata line, <!-- audit-ledger {…} -->: JSON holding its key, severity, and its rule's ID, spec, and heading or its finding's file, with no -- inside. A maintainer closes a fixed ledger issue with a comment opening Fixed in #<n> (Fixed by, diffplug/dormouse#<n>, or the PR's URL are read alike), and closes one not worth fixing as not planned.

Before the agent starts, List open findings writes the open titles to audit-open-findings.txt for the domains to re-verify. A PASS line naming [audit-finding <key>] and then a path:line re-verifies a failure on the rule it cites, FAIL: S42 minted or FAIL S42: <summary> hand-filed (rationale).

  • FAIL IF S30 File embargoed findings can close, edit, or reopen an issue, comment on a ledger issue with anything but its fixed Seen again sentence, or open a ledger issue for a key that already has one open (rationale).
  • FAIL IF S270 two audit jobs of .github/workflows/security-audit.yaml can run at once, or a queued one be dropped: the job loses its one fixed concurrency group, sets cancel-in-progress: true, or queues with anything but queue: max (rationale). Pinned by scripts/security-audit.test.mjs.
  • FAIL IF S31 a PASS's status issue reads as all clear while the ledger holds an open finding or could not be read, or a PASS whose findings were not filed or whose ledger could not be read exits zero (rationale). Pinned by scripts/security-audit.test.mjs.
  • FAIL IF S32 a failure's key changes with its clause letter or where its rule sits, or a finding's with its line or the words around its symbol, or an open issue with a key the run did not mint stops counting. Pinned by scripts/security-audit.test.mjs.

Disclosure

A ledger finding becomes public when its ledger issue closes, or 7 days after it was opened, whichever comes first (rationale), whatever its severity. Publish the audit status gives each ledger issue one issue labeled security-audit-finding, titled [audit-disclosure <key>/<ledger number>] <severity> <kind>; that title, never a mark on the ledger, keeps a re-run from publishing twice (rationale).

Ledger issue Disclosed as Its issue
closed, citing a PR the API reports merged here fixed, linking the PR created closed
closed as completed, citing none closed, no fix PR cited created closed
closed as not planned dismissed created closed
open 7 days not yet fixed created open; closed, rewritten, by the first run that sees the ledger issue closed

A disclosure carries the key, severity, kind, dates, reason, and where: the rule's spec and heading, found by its ID, or the finding's file, each only when the checked-out tree has it; a ledger issue with no metadata line names neither. Read the findings ledger keeps of each ledger issue its number, state, close reason, dates, title key and severity, metadata line, and maintainers' fix citations and public summaries. The one free text that crosses is a maintainer's comment opening ## Public summary, verbatim (rationale). A hand-filed key stays off the status issue until disclosed.

The status issue is the one open issue labeled security-audit-status; a run that finds none creates it. A run comments on it only when the outcome changes and once per new disclosure.

  • FAIL IF S33 a disclosure can be published before its ledger issue closes or turns 7 days old, one closed or 7 days old stays unpublished after a run that read it, a ledger issue gets a second disclosure issue, or a public summary is taken from an issue body or from a comment whose author association is not OWNER, MEMBER, or COLLABORATOR (rationale). Pinned by scripts/security-audit.test.mjs.
  • FAIL IF S34 the status issue is missing, closed, or one of several open issues labeled security-audit-status in diffplug/dormouse. Pinned by scripts/github-state-check.mjs.

A maintainer pins the status issue by hand; scripts/github-state-check.mjs reports an unpinned one as a WARNING, which changes no verdict (rationale).

Source of truth: planDisclosures and statusBody in scripts/security-audit-public-body.mjs; Read the findings ledger in .github/workflows/security-audit.yaml.

Environment and AUDIT_PAT

The audit job declares environment: security-audit, whose deployment-branch-policy admits only main and v* tags — both admin-only by the rulesets in docs/specs/security-ci.md -> "Automated Maintainer (tend)" (rationale). A workflow_dispatch from any other ref is rejected before any step runs, so audit changes are iterated on main.

  • AUDIT_PAT is required. A dedicated step verifies the secret is present before the audit step runs and refuses to continue otherwise (rationale).

  • The PAT is fine-grained and read-only: Administration + Secrets + Environments, scoped to diffplug/dormouse only, minted on an admin's account, stored env-scoped.

  • gh api responses never carry secret values (rationale).

  • FAIL IF S35 the step that verifies AUDIT_PAT is provisioned before the audit runs is removed or bypassed (rationale). Pinned by scripts/security-audit.test.mjs.

  • FAIL IF S36 a workflow step prints $AUDIT_PAT or $CLAUDE_CODE_OAUTH_TOKEN, or .github/audit/_preamble.md stops forbidding a domain to print a secret value or to pass the PAT any way but an unexpanded GH_TOKEN= prefix (rationale).

Source of truth: Verify AUDIT_PAT is provisioned in .github/workflows/security-audit.yaml; Never print a secret value in .github/audit/_preamble.md.

Canary recall

A workflow_dispatch with canary: true, or the weekly schedule line, measures the audit's recall instead of auditing (rationale). Its canary job runs on main alone, and the audit job not at all:

  1. seed applies CANARY_SEEDS seeds from .github/audit/canaries/, dealt round-robin across the code domains; then it deletes the pool and replaces the history with one root commit, so nothing in the checkout tells a seed from the code around it (rationale).
  2. The audit's agent runs the code domains over that checkout: supply-chain, remote, local, and hosted, handed an empty open-findings list. ci-and-secrets reads live GitHub state, which a checkout cannot seed, and the ledger is never read.
  3. score counts a seed caught only when a FAIL result, or a finding at or above the seed's catch severity, cites one of its Expect: files and, for a seed naming a FAIL IF, that rule's ID: the result's, or one the finding names as its root cause or anywhere in its text.

  • A seed re-introduces a vulnerability already fixed on main, or a synthetic one, and deletes the test and lint cases that would fail on it, so the seeded tree passes CI as a regression on main has (rationale). Its header is factual and names no finding under embargo.

  • A seed's catch severity is WARNING unless its Severity: header names another, then — and why (rationale).

  • The public record is the step summary's line and the canary-recall artifact: seeded and caught counts, how many domains left no finished report, and the run link. validateRecord in website/scripts/generate-canary-recall.js accepts exactly these fields; add one to both, never rename.

  • Which seeds ran, what caught them, and findings on code no seed touched stay in the canary-transcript artifact; a canary files them nowhere.

  • A canary run is a completed run on main, so the next scheduled run audits in full (Skipping an unchanged audit).

  • FAIL IF S37 the canary job can push or file: it holds any write permission, runs its agent on a token but its own read-only one, keeps the checkout's credentials, runs gh or git push, or names AUDIT_PAT or EMBARGO_TOKEN; or scripts/security-audit-local.sh canary, which seeds a copy of the committed tree, files anything, either engine's included. Pinned by scripts/security-audit-canary.test.mjs.

  • FAIL IF S38 a canary run, dispatched or weekly, can gate a release or feed the skip: the audit job runs in it, it runs on a ref but main, it uploads an audit-state artifact, or the gate in .github/workflows/release.yml can select a run with the canary's title. Pinned by scripts/security-audit-canary.test.mjs.

  • FAIL IF S39 a canary publishes seed detail: the step summary or the canary-recall artifact carries anything but the counts and a validated run link, or the scorecard, the seeds applied, or a fragment leaves the runner but as age ciphertext to .github/audit/transcript-recipient.txt. Pinned by scripts/security-audit-canary.test.mjs.

  • FAIL IF S40 a seed stops applying to the tree, alone or over the rest of the pool, or names an ID no rule carries, a rule no longer holding its phrase, or one its Domain: does not own, or a code domain has fewer than three seeds. Pinned by scripts/security-audit-canary.test.mjs, which pnpm test runs.

scripts/security-audit-local.sh canary --key <run ID> seeds that CI canary run's commit with what that run seeded.

Source of truth: seed and score in scripts/security-audit-canary.mjs; the canary job in .github/workflows/security-audit.yaml; run_canary in scripts/security-audit-local.sh.

Published recall

dormouse.sh/security shows a rolling recall under "How the guarantees are checked": caught over seeded across the canary-recall artifacts of canary runs on main in the 30 days before its as-of date, each run linked. Its weekly graph draws three lines: Claude over every run; Codex over the runs a Second-opinion audit record pairs with; and caught by either, where a run no record pairs adds Claude's catches alone, and a paired run with a null either count, not computed rather than zero, leaves its week no point. It is a committed snapshot, website/src/data/canary-recall.json, regenerated at each release (docs/specs/deploy.md -> "Release checklist") (rationale).

  • A record out of shape fails the snapshot, never repaired or skipped: a field but the scorer's, a count not a non-negative integer, more caught than seeded, or a link but to its own run on diffplug/dormouse.
  • Every run in the window counts as its record has it (rationale): an unfinished domain's seeds stay seeded, caught only by what it reported, and the run shows its unfinished count. A run with no canary-recall artifact is listed without counts.
  • The generator downloads canary-recall and codex-recall alone, never canary-transcript.
  • A second-opinion record pairs with its canary run by link, the newest successful run on main per canary run; one whose seeded count differs from that run's, or whose either count falls below Claude's or above the two engines' sum, fails the snapshot.

Source of truth: summarize in website/scripts/generate-canary-recall.js; CanaryRecall in website/src/components/CanaryRecall.tsx.

Second-opinion audit

The second-opinion audit runs the same prompts on Codex, outside Claude's model family: no Codex verdict gates a release, feeds the skip, or files (rationale). scripts/security-audit-local.sh --engine codex runs them on gpt-6-sol under the daybreak_blue Cyber access program, sandboxed to the working tree. Codex has no subagents, so each ## Work streams entry runs as its own session on the preamble less its delegation sections, appending to a fragment the runner opens and closes with the verdict its lines compute (rationale). A maintainer runs every domain nightly, and weekly a canary keyed by the latest CI canary run on main, which counts the seeds either engine caught given that run's decrypted scorecard, and leaves that count empty without it (rationale). A Codex BLOCKER or WARNING reaches the ledger only as a maintainer's hand-filed manual- key (Findings ledger).

On a machine running Codex, the gh login holds no read on the ledger, since the sandbox reaches it (rationale): AUDIT_OPEN_FINDINGS=<file> hands the domains a list written on another login, copied to audit-open-findings.txt in place of listing the ledger. Without it, --engine codex refuses to audit a domain.

.github/workflows/codex-audit-report.yaml publishes the run from workflow_dispatch inputs that carry nothing a finding could be read from: the commit, the model, each domain's computed verdict, and the canary's run ID and seeded, Codex-caught, either-caught, and unfinished counts. scripts/security-audit-public-body.mjs checks each against its pattern and the commit against main, checks the counts against that canary run's own record and commit, comments once on the status issue, and writes the codex-recall record Published recall pairs.

  • FAIL IF S275 .github/workflows/codex-audit-report.yaml can gate, file, or publish past its inputs: it holds any write permission but its one job's issues: write, reads a secret, interpolates an input into a script, runs on a ref but main, posts for anyone but an admin or anything but codexComment's text, or the gate in .github/workflows/release.yml can select its run; or scripts/security-audit-local.sh runs Codex unsandboxed or on another model or program. Pinned by scripts/security-audit.test.mjs.

Source of truth: run_domain and write_open_findings in scripts/security-audit-local.sh; streamPrompt in scripts/security-audit-codex.mjs; publishCodex in scripts/security-audit-public-body.mjs; .github/workflows/codex-audit-report.yaml.

Red team

A maintainer runs a red team in the first week of every month, and before any release that changes remote control or Hosted (rationale): an open-ended pass over the code and the published guarantees without the FAIL IF lines, on Claude through Anthropic's Cyber Verification Program and on Codex through OpenAI's Daybreak program. Its BLOCKERs and WARNINGs reach the ledger as hand-filed manual- keys (Findings ledger).

A scheduled workflow opens each month's issue, Red team: <Month YYYY> labelled red-team, unless one with that title exists; closing it as completed records the pass. A release's job summary warns, and never fails, when what the remote and hosted domains own has changed since the latest completed red-team issue closed.

Source of truth: .github/workflows/red-team-issue.yaml; SURFACES in scripts/red-team-since.mjs; the red-team-note job in .github/workflows/release.yml.

Future

Scope: audit-credential-separation — Credential separation.

Credential separation

A second job outside the security-audit environment, running the domains that need no PAT and passing their fragments back as artifacts, would leave remote, local, and hosted unable to hold AUDIT_PAT at all. The fragments would cross as ciphertext, and File embargoed findings would move to a job no agent ran on, holding the key to read them.