Skip to content

[BUG]: Guard against reading attack files that were not created #26

Description

@dhl123

Current Behavior

Several places in the pipeline call open() on attack-case files without first checking that the file exists. When such a file was never created the run crashes with FileNotFoundError.

Both stages read ATTACK_TOOLS from the same source (os.getenv("ATTACK_TOOLS", "ares,promptfoo"), one .env loaded at cli.py:43), but each re-reads it at the moment it runs and nothing records which attack files generation actually wrote. So if the value changes between invocations, for example, an edited .env, it might open a file that isn't there. The fix in all cases is the same: guard on the file existing before reading it.

Fix

The three unguarded reads

  1. src/smith/test_case_evaluation/classify_guidance.py:108 — classify_promptfoo_cases, the primary cross-stage crash:
with open(promptfoo_cases_file, "r") as f:
    cases = json.load(f)
  1. src/smith/test_generation/convert_test_case.py:68 — merge_with_ares:
with open(output_file_attack, "r") as f:
    attack_cases = json.load(f)
  1. src/smith/test_generation/convert_test_case.py:88 — merge_with_promptfoo:
with open(output_file_attack_promptfoo, "r") as f:
    attack_cases = json.load(f)

Check that each attack file exists before opening it by adding an os.path.exists guard at the top of classify_promptfoo_cases, merge_with_ares, and merge_with_promptfoo.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions