Skip to content

FIX: record the attack identity on ERROR attack results - #3097

Merged
Richard Lundeen (richlundeen) merged 2 commits into
microsoft:mainfrom
u7k4rs6:fix/error-result-identifier
Oct 11, 2026
Merged

Richard Lundeen (richlundeen) merged 2 commits into
microsoft:mainfrom
u7k4rs6:fix/error-result-identifier

Conversation

@u7k4rs6

Copy link
Copy Markdown
Contributor

Fixes #3096.

The error handler is a separate object with no access to the strategy, so execute_with_context_async now leaves a small builder on the context, and _on_error_async uses it to stamp the same AtomicAttackIdentifier the completed paths record. The eval hash gets auto-stamped on save like for any other row.

It's built lazily, and only on the error path. If building it fails, it logs at debug and saves the error row without it, as before, so an odd target identifier can't block saving the error.

Scenario runs still enrich only completed results with seed info, so their error rows get the plain attack identity, which is enough for class and technique filters. Enriching those too would need the executor to keep input indices for incomplete objectives, so I left that out.

Repro on this branch:

row objective='please fail' outcome=ERROR identifier=set eval_hash=cb56db9a6405
get_attack_results_async(attack_classes=['PromptSendingAttack']) -> ['FAILURE', 'ERROR']
compute_technique_stats_async -> AttackStats(... failures=1, errors=1)

Test: a failing PromptSendingAttack's stored ERROR row is found by the class filter and carries the attack identity and eval hash. It fails on main. tests/unit/executor, scenario, analytics and backend: 5437 passed.

Clear the per-run identifier callback after execution and log builder failures at warning level. Cover builder failures, generic strategies, reused contexts, and compound error identities.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: fcbdea79-a119-453f-a809-3fc2d2cec113
@richlundeen
Richard Lundeen (richlundeen) added this pull request to the merge queue Oct 11, 2026
Merged via the queue into microsoft:main with commit 96077b2 Oct 11, 2026
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: ERROR attack results are saved without the attack identifier

3 participants