[FEATURE/BUGS]: Add Ares case confidence score & guard attack-file reads against missing files & Conduct OPA policy bypasses to generate adversarial test cases

.env_template

@@ -73,6 +73,11 @@ CASE_FILE=references/test_cases.json
 FINAL_TEST_CASES=references/test_cases/
 GREY_GUIDANCE_FILE=references/grey_guidance.json
 
+# Abstract bypass cases synthesized from the policy analysis.
+BYPASS_CASE_FILE=references/bypass_cases.json
+# Directory for the structured bypass vector report (JSON + Markdown).
+BYPASS_REPORT_DIR=references/bypass/
+
 BATCH_PROCESSING=true
 BATCH_SIZE=10
 CASE_GENERATION_BATCH_SIZE=5

CHANGELOG.md

@@ -18,27 +18,36 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 ### Fixed
 
 - Tier-3 label validation no longer aborts the entire loop on a single LLM error. Transient failures now fall back for that case and continue; the loop only aborts after N consecutive failures (default 5, configurable via `run_validation`) indicating the LLM is genuinely unavailable. On abort, the remaining un-evaluated cases are still recorded as uncertain so validation metrics no longer silently shrink.
-- OPA scorecard no longer silently scores request failures as "deny". Added a curl timeout (configurable via `SMITH_OPA_TIMEOUT`, default 5s) and exit-code checking; failed requests are logged to `errors.txt` and excluded from TP/FP/TN/FN counts.
-- Invalid `ATTACK_TOOLS` values now fail fast with an actionable error instead of silently disabling red-teaming, and the CLI prints which attack tools are enabled vs skipped.
-- Updated SKILL.md documentation to reflect "test all deny" instead of "all test failed".
+- OPA scorecard no longer silently scores request failures as "deny". Added a curl timeout and exit-code checking; failed requests are logged to `errors.txt` and excluded from TP/FP/TN/FN counts.
+- Invalid `ATTACK_TOOLS` values now fail with an actionable error instead of silently disabling red-teaming, and the CLI prints which attack tools are enabled vs skipped.
+- Attack-case readers no longer crash with `FileNotFoundError` when an attack file was never created (e.g. `ATTACK_TOOLS` differed between the generation and evaluation runs). `classify_promptfoo_cases`, `merge_with_ares`, and `merge_with_promptfoo` now guard on the file existing before reading it and skip gracefully with a log message when it is absent.
+- ARES cases now inherit their parent test case's confidence score, verdict, and predicted label in the evaluation report instead of showing a blank confidence. An ARES case is a jailbreak-transformed variant of a parent `disallow` case, so it carries the parent's `ValidationResult` (matched by the parent's `user_input`) rather than being independently re-validated.
+- Updated SKILL.md documentation to reflect "all test deny" instead of "all test failed".
 - Updated car-price and call-for-papers examples' Promptfoo configuration with missing system variables.
 
 ### Added
 
 - Documentation site (Hugo) with guides for configuration, quickstart, policy creation, testing, refinement, cross-validation, Promptfoo integration, and contributing.
+- **Policy-bypass test-case generation** (`smith --flag bypass_case_generation`): a new pipeline that analyzes the current policy against the guidance to find divergences, then synthesizes adversarial cases targeting each gap.
+  - New package `src/smith/policy_agent/policy_analysis/bypass/`: `analyze_bypass.py` (`detect_bypass_vectors`), `synthesize_cases.py` (`synthesize_bypass_cases`), `schema.py` (`BypassVector`/`BypassReport`).
+  - `cli.py`: new `generate_bypass_cases()` function and the `bypass_case_generation` flag (detect → synthesize → convert), guarded against a missing/empty policy.
+  - `convert_test_case.py`: new `convert_bypass_case()` routes bypass cases into `disallow/` or `allow/` with a `bypass_test_case` prefix.
+  - `.env_template`: new vars `BYPASS_CASE_FILE` and `BYPASS_REPORT_DIR`.
+  - `test_generation.md` rewritten to ask up front which cases to generate. User can choose general test cases (legitimate allow, disallow, ares, promptfoo), or/and bypass test cases.
 - Integrated Promptfoo policy plugin for generating malicious test cases from guidances, with translation support for string-typed variables.
 - `ATTACK_TOOLS` environment variable to select which red-teaming tools to run (`ares`, `promptfoo`, `ares,promptfoo`, or `none`).
-- "Remove" decision category in cross-validation for ambiguous/invalid test cases.
 - Clean-up bash script (`scripts/clean_generated.sh`) to reset generated intermediates when switching examples.
 
 ### Changed
 
 - Made ARES and Promptfoo optional dependencies — either tool can be used independently or skipped entirely.
-- Cross-validation now focuses on arguments and subject fields only, improving accuracy.
+- Cross-validation now focuses on arguments and subject fields only, improving accuracy. "Remove" decision category in cross-validation for ambiguous/invalid test cases. Cross-validation now also discards failed adversarial probes instead of relabeling them: for bypass and promptfoo cases, any audit verdict other than `keep` is collapsed to `remove` (with a marker appended to the reason), since their intent is malicious and a failed probe should not pollute the ordinary case set.
 - Cluster indexing uses sequential numbers; noise group appears as the last numbered cluster instead of `-1`.
 - Renamed the target-agent LLM environment variables (`RITS_*`) to `INFERENCE_MODEL`/`INFERENCE_BASE_URL`/`INFERENCE_API_KEY` across `.env_template`, examples, and documentation. `OLLAMA_BASE_URL` (no `/v1` suffix) is now reserved solely for promptfoo's native ollama provider, resolving the previous duplicate-variable collision.
 - Updated example configurations (call-for-papers, car-price, RagChatbot) with revised system variables and regenerated smith outputs.
-- Verified Promptfoo test cases now live in `references/test_cases/disallow/` (removed separate `promptfoo_malicious/` folder).
+- Converted Promptfoo test cases now live in `references/test_cases/disallow/` (removed separate `promptfoo_malicious/` folder).
+- `test_case_translation` skips cases that already carry an `arguments` block (already translated), making translation re-runnable and avoiding a full-corpus re-translation when only newly-added bypass cases need it.
+- Reset `assets/policy.rego` to empty as a fresh starting point for policy creation.
 
 ## [0.1.1] - 2026-06-29
 - Repackaged `scripts/` into an installable `smith` Python package using a `src/`

README.md

@@ -14,7 +14,8 @@ An open skill for AI code agents that automates OPA policy creation, test genera
 Smith is a skill (plugin) for AI code agents that manages the full lifecycle of [Open Policy Agent (OPA)](https://www.openpolicyagent.org/) policies (more types of policies will be supported). It enables agents to:
 
 - **Create** OPA policies from natural language guidance and an agent description.
-- **Generate** both synthetic legitimate and adversarial test cases using LLM-based fuzzing and existing red-teaming tools.
+- **Generate** both synthetic legitimate and adversarial test cases using LLM-based fuzzing and existing red-teaming tools, plus
+policy-bypass cases that target divergences between the guidance and the current policy.
 - **Test** policies against generated and custom test suites.
 - **Refine** policies automatically through iterative feedback loops (patches for failed test cases, linting, etc.).
 
@@ -35,13 +36,7 @@ Guidance Description (NLP) + Agent Description → Enforceable Policy Creation 
 
 Place the entire `smith` folder under the `skills/` or `plugin/` directory of your code agent (Claude Code, Bob, Aider, etc.). The coding agent automatically recognizes Smith as an open skill.
 
-For more details of how to use skills in different coding agents, you can refer to the following links:
-
-If you are using bob: https://bob.ibm.com/docs/ide/features/skills
-
-If you are using Claude: https://code.claude.com/docs/en/skills
-
-If you are using Aider Desk: https://aiderdesk.hotovo.com/docs/features/skills
+For more details of how to use skills in different coding agents, see [Bob](https://bob.ibm.com/docs/ide/features/skills), [Claude](https://code.claude.com/docs/en/skills), and [Aider](https://aiderdesk.hotovo.com/docs/features/skills).
 
 ## Install
 
@@ -172,25 +167,27 @@ For an SSE-based MCP server instead, set `MCP_TRANSPORT=sse` and `MCP_URL=http:/
 Smith operates as an agent skill with a CLI backend. The AI agent reads instructions from `SKILL.md` and orchestrates the appropriate workflows by invoking the `smith` CLI or following embedded markdown guides.
 
 ```
-┌──────────────────────────────────────────────────────────────────────────┐
-│                                  Smith                                   │
-│                                                                          │
-│  SKILL.md ──→ Orchestration ──→ smith CLI                                │
-│                                    │                                     │
-│              ┌─────────────────────┼────────────────┬─────────┐          │
-│              ▼                     ▼                ▼         ▼          │
-│         Policy              Test Case Gen        Policy     Policy       │
-│         Creation                 │               Testing   Refinement    │
-│              │          ┌────────┼────────┐        │         │           │
-│              ▼          ▼        ▼        ▼        └────⇄────┘           │
-│         OPA Policy  Legitimate  ARES  Promptfoo                          │
-│         (.rego)         │        │        │                              │
-│                         └────────┼────────┘                              │
-│                                  ▼                                       │
-│                          Test Case Evaluation                            │
-└──────────────────────────────────────────────────────────────────────────┘
+┌────────────────────────────────────────────────────────────────────────────────┐
+│                                     Smith                                      │
+│                                                                                │
+│  SKILL.md ──→ Orchestration ──→ smith CLI                                      │
+│                                    │                                           │
+│              ┌─────────────────────┼──────────────────────┬─────────┐          │
+│              ▼                     ▼                      ▼         ▼          │
+│         Policy              Test Case Gen              Policy     Policy       │
+│         Creation                 │                     Testing   Refinement    │
+│              │        ┌──────────┼──────────┬────────┐    │         │          │
+│              ▼        ▼          ▼          ▼        ▼    └────⇄────┘          │
+│         OPA Policy Legitimate  ARES    Promptfoo  Bypass                       │
+│         (.rego)        │        │          │        │                          │
+│                        └────────┴────┬─────┴────────┘                          │
+│                                      ▼                                         │
+│                              Test Case Evaluation                              │
+└────────────────────────────────────────────────────────────────────────────────┘
 ```
 
+Bypass cases are generated from the current policy (a `Policy Creation` output) rather than the guidance alone, so they close the loop back into `Test Case Gen`.
+
 
 ## Core Concepts
 
@@ -210,7 +207,13 @@ The policy only references data available from tool arguments and system variabl
 
 ### Test Case Generation
 
-The agent follows `test_generation/test_generation.md` to generate test cases through a multi-stage pipeline:
+The agent follows `test_generation/test_generation.md`, which first asks which kind of test cases you want, then runs the matching command(s):
+
+- **Guidance-targeted cases** — legitimate + adversarial cases derived from the guidance (broad coverage).
+- **Policy-bypass cases** — adversarial cases that target divergences between the guidance and the **current policy** (requires an existing, non-empty policy).
+- **Both.**
+
+#### Guidance-targeted generation
 
 ```bash
 # CLI commands used for test case generation
@@ -226,6 +229,21 @@ This runs the following stages:
 
 All results are stored in `./references/test_cases/`.
 
+#### Policy-bypass generation
+
+```bash
+# Requires an existing, non-empty policy at assets/policy.rego
+smith --flag bypass_case_generation
+```
+
+Instead of generating from the guidance alone, this analyzes the **current policy against the guidance** to find where they diverge (e.g. a rule that omits an existence check, a numeric comparison with no type assertion, a substring match a malformed value slips past), then synthesizes adversarial cases that exercise each divergence:
+
+1. **Detect** — Compare the full Rego policy to the guidance and classifies each divergence by mechanism (`omitted_field`, `type_confusion`, `malformed_value`, `keyword_evasion`). If the model returns malformed JSON, the call is retried (up to `MAX_BYPASS_PARSE_ATTEMPTS`, default 3) before giving up with an empty report.
+2. **Synthesize** — Turn each divergence into concrete abstract cases.
+3. **Convert** — Write them into `./references/test_cases/{allow,disallow}/` with a `bypass_test_case` prefix.
+
+The divergence report is saved to `./references/bypass/` (JSON + Markdown). If the policy is missing or empty, generation is skipped with a message.
+
 ### Test Case Evaluation
 
 ```bash
@@ -262,6 +280,7 @@ Calls the agent's `/extract_tool_call` endpoint to extract tool names and argume
 
 - Cases where the returned tool name doesn't match the expected one are flagged as mismatches and removed
 - Cases labeled as "other" (general questions that don't invoke any tool) are moved to `./references/test_cases/malicious/` for future guardrail features
+- Cases that already carry an `arguments` block are skipped, so translation is re-runnable. After adding bypass cases you can re-run it to translate only the new cases instead of re-translating the whole corpus
 
 ### Policy Testing
 
@@ -278,7 +297,7 @@ Two cross-validation workflows handle different failure scenarios after policy t
 
 **Policy Cross-Validation** — When policy testing returns 0 test cases evaluated or 100% failure, the policy likely has structural issues (input path mismatches or OPA syntax bugs). The agent follows `opa_policy/policy_cross_validation/policy_cross_validation.md` to diagnose and fix these issues before proceeding to refinement.
 
-**Test Case Cross-Validation** — When policy testing produces mixed pass/fail results, some failures may be caused by mislabeled test cases rather than policy bugs. This workflow uses an LLM to check each failed case against the guidance and suggests corrections (move to correct folder or remove).
+**Test Case Cross-Validation** — When policy testing produces mixed pass/fail results, some failures may be caused by mislabeled test cases rather than policy bugs. This workflow uses an LLM to check each failed case against the guidance and suggests corrections (move to correct folder or remove). Adversarial cases (filenames prefixed `bypass_test_case` or `promptfoo_test_case`) are an exception: any non-`keep` verdict is collapsed to `remove` rather than relabeled, so a failed malicious probe is discarded instead of being moved into `allow/`.
 
 ```bash
 smith --flag cross_validate          # generate report of mislabeled cases

assets/policy.rego

@@ -1,10 +1,2 @@
 # Copyright 2026 Smith authors
 # SPDX-License-Identifier: Apache-2.0
-
-package mcp.policies
-
-default allow := false
-
-# === Input Accessors ===
-subject := input.extensions.subject
-args := object.get(input, "arguments", {})