Skip to content

fix: audit separate crash launch and error capture - #462

Merged
janicduplessis merged 2 commits into
mainfrom
fix/benchmark-crash-audit-461
Sep 6, 2026
Merged

fix: audit separate crash launch and error capture#462
janicduplessis merged 2 commits into
mainfrom
fix/benchmark-crash-audit-461

Conversation

@janicduplessis

@janicduplessis janicduplessis commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

Description

The crash benchmark rejects a successful Stim run when the launch output omits the JavaScript error, even though the next stim logs --errors captures it. The original collector required the token inline for Stim; the current workflow explicitly separates launching from capturing errors.

Solution

Require a successful launch followed by a token-specific runtime error and source location for either arm. Accept current setup commands while still rejecting appended source inspection before diagnosis, and decode displayed shell wrappers consistently in collection and export. After recorded cleanup, retain an integrity-matched, previously verified repair proof without retaining the old audit verdict.

This changes benchmark auditing only, not Stim or benchmark prompts. Genuine timeout and isolation failures remain excluded.

Test plan

  • Re-audited the retained Sol Stim crash attempt: diagnosis 2m42s, Settings 4m37s, valid repair and recording, all original evidence hashes unchanged. No agent rerun.
  • Confirmed control remains invalid after timing out without completing its recording.
  • Regression coverage includes separate launch/log capture, historical setup, forbidden source reads, and cleanup-proof retention with mismatched evidence or another audit failure.

Fixes #461

@janicduplessis
janicduplessis marked this pull request as ready for review September 6, 2026 15:14
@janicduplessis
janicduplessis merged commit d53bebf into main Sep 6, 2026
5 checks passed
@janicduplessis
janicduplessis deleted the fix/benchmark-crash-audit-461 branch September 6, 2026 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Align crash benchmark audit with current launch and setup workflow

1 participant