[FEATURE/BUGS]: Add Ares case confidence score & guard attack-file reads against missing files & Conduct OPA policy bypasses to generate adversarial test cases

[dhl123]

Summary

This PR supports generating malicious cases to bypass policy, adds confidence score for Ares generated test cases, and adds guard for attack-file reads against missing files

Closes: #24 #25 #26

Changes

New feature: policy-bypass test-case generation

A new pipeline that finds where the policy diverges from the guidance and generates additional adversarial cases for each divergence.

• New package src/smith/policy_agent/policy_analysis/bypass/ — analyze_bypass.py (detect_bypass_vectors), synthesize_cases.py (synthesize_bypass_cases), schema.py (BypassVector/BypassReport).
• cli.py: new generate_bypass_cases() function and --flag bypass_case_generation (detect → synthesize → convert), guarded against a missing/empty policy.
• convert_test_case.py: new convert_bypass_case() (only) routes bypass cases into disallow/ or allow/ with a bypass_test_case prefix.
• .env_template: new vars BYPASS_CASE_FILE, BYPASS_REPORT_DIR.
• detect_bypass_vectors previously wrote a silent empty report whenever the model returned malformed JSON, indistinguishable from "no bypasses found." It now retries the model call up to MAX_BYPASS_PARSE_ATTEMPTS = 3 if the returned result is not JSON (empty json is acceptable). Only after 3 failed attempts it stops, reports the failure to the user, and returns an empty report.
• test_generation.md: rewritten to branch up front: ask the user which cases they want, they can choose general test cases (legitimate allow, disallow, ares, promptfoo), or/and bypass test cases.

Confidence score of ARES in test validation

ARES cases now inherit their parent test case's confidence score, verdict, and predicted label in the evaluation report instead of showing a blank confidence. An ARES case is a jailbreak-transformed variant of a parent disallow case, so it carries the parent's ValidationResult (matched by the shared user_input) rather than being independently re-validated.

Guard attack-file reads against missing files

translate_case in cli.py already passes None for any attack file whose tool wasn't specified in ATTACK_TOOLS, and merge_with_ares / merge_with_promptfoo skip merging when the path is None. This PR adds a second layer: merge_with_ares and merge_with_promptfoo now also verify the file actually exists on disk before reading it, so a missing or unsaved attack file is skipped gracefully instead of crashing with FileNotFoundError.

The same applies in classify_guidance.py: even though the call site already checks whether promptfoo is in the attack settings, classify_promptfoo_cases now double-checks the file's existence before opening it.

This protects against cases where ATTACK_TOOLS differs between the generation and evaluation runs, or where an attack file was expected but never written.

Other refinements

• cross_validate.py: For bypass and promptfoo cases, any audit verdict other than keep is collapsed to remove (rather than relabeled), with a marker appended to the reason. A failed adversarial probe is dropped instead of polluting the ordinary case set since their intentions are malicious.
• extract_tool_args.py: skip cases that already carry an arguments block (already translated). This makes test_case_translation re-runnable and avoids re-translating the entire corpus when only newly-added (bypass) cases need translation (the previous behavior re-invoked translation on every case, which is inefficient and slow).
• Reset policy to empty.
• Example test cases replaced with their translated form.

Checks

• make ci passes (lint, Rego lint, license headers, build smoke)
• make test passes (policy scorecard — needed if policy behavior changed)
• CHANGELOG.md updated under ## [Unreleased] (if user-facing)
• Commits are signed off for the DCO (git commit -s)

[araujof]

LGTM