FIX Prevent duplicate attacks when scenario resume history cannot be read - #2857
Merged
Roman Lutz (romanlutz) merged 2 commits intoSep 26, 2026
Merged
Roman Lutz (romanlutz) merged 2 commits into
Roman Lutz (romanlutz) merged 2 commits into
Conversation
This was referenced Sep 25, 2026
Roman Lutz (romanlutz)
approved these changes
Sep 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A transient failure while reading completed attack results can make a resumed scenario execute already-completed objectives again. If the progress query keeps failing but the other memory operations succeed, the old implementation can still run all objectives and mark the scenario
COMPLETED.Concrete example: a scenario has two groups: A has a persisted successful result; B has not run yet. During resume,
get_attack_results(scenario_result_id=...)raises a database error.Root cause:
_get_completed_objective_hashes_for_attack()caught all read exceptions and returned an empty set, making an unavailable history indistinguishable from a successfully read empty history._get_remaining_atomic_attacks_async()also called it once per atomic group, repeatedly loading and scanning every result in the scenario.Change: load all completed objective hashes once per reconciliation, indexed by
(parent_collection, parent_eval_hash), before mutating any group's remaining seeds. A read exception now reaches the existing scenario retry/failure handling. Each retry reads a fresh snapshot. Technique identities remain separate, legacy rows without an eval hash still match by collection name, andERRORresults remain eligible for retry. No schema migration or public API change is required.This reduces the reconciliation history reads from N to 1 for N atomic groups. It is a query-count improvement verified by the test below, not a claim about measured production latency.
Tests and Documentation
Three regressions failed on unmodified upstream and pass with the fix:
FAILEDwhen status storage is available, leave the original seed groups intact, and invoke neither A nor B.get_attack_results()call and correctly skip the five already-completed groups.The tests use the real Scenario retry loop and isolated SQLite persistence, with faults injected specifically at the history-read boundary. Existing tests continue to cover partial-result retry, technique identity separation, and legacy attribution.
make unit-testwith Python 3.11/default dependencies: 20,414 passed, 146 skipped, 1 failed. The sole failure is the existingtest_get_seed_dataset_summaries_follows_a_trailing_blank_insensitive_collationintests/unit/memory/memory_interface/test_interface_seed_prompts.py, also reproduced at unmodified upstream73d90a7with the same environment.ty check pyrit.73d90a7: 582 relevant combined regressions passed. The patches apply together without conflicts.To reproduce the targeted checks:
uv run pytest tests/unit/scenario/core/test_scenario_retry.py -q -k 'resume_read_failure or resume_loads_one'Updated the private resume reconciliation docstrings to describe snapshot loading and error propagation. No notebooks changed; no live target or production database was used.