Current Behavior
Several places in the pipeline call open() on attack-case files without first checking that the file exists. When such a file was never created the run crashes with FileNotFoundError.
Both stages read ATTACK_TOOLS from the same source (os.getenv("ATTACK_TOOLS", "ares,promptfoo"), one .env loaded at cli.py:43), but each re-reads it at the moment it runs and nothing records which attack files generation actually wrote. So if the value changes between invocations, for example, an edited .env, it might open a file that isn't there. The fix in all cases is the same: guard on the file existing before reading it.
Fix
The three unguarded reads
src/smith/test_case_evaluation/classify_guidance.py:108 — classify_promptfoo_cases, the primary cross-stage crash:
with open(promptfoo_cases_file, "r") as f:
cases = json.load(f)
src/smith/test_generation/convert_test_case.py:68 — merge_with_ares:
with open(output_file_attack, "r") as f:
attack_cases = json.load(f)
src/smith/test_generation/convert_test_case.py:88 — merge_with_promptfoo:
with open(output_file_attack_promptfoo, "r") as f:
attack_cases = json.load(f)
Check that each attack file exists before opening it by adding an os.path.exists guard at the top of classify_promptfoo_cases, merge_with_ares, and merge_with_promptfoo.
Current Behavior
Several places in the pipeline call
open()on attack-case files without first checking that the file exists. When such a file was never created the run crashes with FileNotFoundError.Both stages read ATTACK_TOOLS from the same source (
os.getenv("ATTACK_TOOLS", "ares,promptfoo"), one.envloaded atcli.py:43), but each re-reads it at the moment it runs and nothing records which attack files generation actually wrote. So if the value changes between invocations, for example, an edited .env, it might open a file that isn't there. The fix in all cases is the same: guard on the file existing before reading it.Fix
The three unguarded reads
src/smith/test_case_evaluation/classify_guidance.py:108 — classify_promptfoo_cases, the primary cross-stage crash:src/smith/test_generation/convert_test_case.py:68 — merge_with_ares:src/smith/test_generation/convert_test_case.py:88 — merge_with_promptfoo:Check that each attack file exists before opening it by adding an os.path.exists guard at the top of classify_promptfoo_cases, merge_with_ares, and merge_with_promptfoo.