An open skill for AI code agents that automates OPA policy creation, test generation, testing, and iterative refinement.

What's Smith?

Smith is a skill (plugin) for AI code agents that manages the full lifecycle of Open Policy Agent (OPA) policies (more types of policies will be supported). It enables agents to:

• Create OPA policies from natural language guidance and an agent description.
• Generate both synthetic legitimate and adversarial test cases using LLM-based fuzzing and existing red-teaming tools, plus policy-bypass cases that target divergences between the guidance and the current policy.
• Test policies against generated and custom test suites.
• Refine policies automatically through iterative feedback loops (patches for failed test cases, linting, etc.).

Guidance Description (NLP) + Agent Description → Enforceable Policy Creation → Test Generation → Policy Testing ⇄ Policy Refinement

What Smith Needs from You

1. Guidance file — A natural language description of your access control policies (e.g., "managers can only view compensation for their own team")
2. Agent server with endpoints — Your agent must expose:
   • /chat — Used by Promptfoo for red-teaming test generation
   • /extract_tool_call — Used to auto-detect MCP tool parameters and definitions from user prompts
3. System variable file — A JSON file listing the system variables available in your agent (e.g., roles, teams, claims)
4. Keep both your agent server and MCP server running during Smith's operation

Deployment

Place the entire smith folder under the skills/ or plugin/ directory of your code agent (Claude Code, Bob, Aider, etc.). The coding agent automatically recognizes Smith as an open skill.

For more details of how to use skills in different coding agents, see Bob, Claude, and Aider.

Install

Prerequisites

• Python 3.11+
• uv (package management)
• OPA + Regal (policy testing / linting; OPA runs in Docker for the scorecard)
• ARES (red-teaming framework) — optional
• Promptfoo (red-teaming framework) — optional

Which red-teaming tools to use is controlled by the ATTACK_TOOLS environment variable (see Configuration). Set to none to skip red-teaming entirely.

1. Python environment

python -m venv .venv
source .venv/bin/activate

2. ARES (red-teaming framework). Installs into src/smith/test_generation/ares/ with its own .venv, which is the layout the test-generation pipeline expects (src/smith/test_generation/attack.py invokes ares/.venv/bin/ares):

cd src/smith/test_generation/ares
python -m venv .venv
source .venv/bin/activate
curl https://raw.githubusercontent.com/IBM/ares/refs/heads/main/install.sh | bash
ares install-plugin ares-autodan
ares install-plugin ares-human-jailbreak
ares install-plugin ares-garak
deactivate
# Setup ares configuration
cp ../ares_config/qwen-owasp-llm-01.yaml ./example_configs
cp ../ares_config/human_jailbreaks.json ./assets
export ARES_HOME=/absolute/path/to/smith/src/smith/test_generation/ares
# Switch back to the original Python environment
cd ../../../../
source .venv/bin/activate

3. Promptfoo (red-teaming framework).

npm install -g promptfoo
# To disable promptfoo remote connection:
export PROMPTFOO_DISABLE_TELEMETRY=1
export PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=true
export PROMPTFOO_DISABLE_SHARING=true

Install Smith

Smith uses uv for package management. From the repo root:

make install        # creates a uv venv and installs Smith (editable) + dev tools

Or install directly (dependencies are declared in pyproject.toml):

uv pip install -e .   # or: pip install -e .

This installs the smith CLI command.

Configuration

cd ..
cp .env_template .env

Fill in every placeholder value in .env before running Smith. The most important variables:

Variable	Description
BASE_URL	Absolute path to your skill folder, with a trailing slash, e.g. /path/.bob/skills/smith/
OPENAI_API_KEY	API key for your LLM provider
OPENAI_BASE_URL	Base URL for LLM API endpoint
MODEL_SONNET	Model used across the pipelines (e.g., GCP/claude-4-sonnet by default)
AGENT_URL	URL of the target agent server (must expose /chat and /extract_tool_call); default http://localhost:9000
INFERENCE_MODEL	Model name for the target agent's LLM (e.g., qwen3.5:latest for Ollama, or a RITS model name)
INFERENCE_BASE_URL	Base URL for the agent's LLM API (e.g., http://localhost:11434/v1 for Ollama)
INFERENCE_API_KEY	API key for the agent's LLM (use ollama for local Ollama)
OLLAMA_BASE_URL	Base URL for promptfoo's native ollama provider during red-team generation (no /v1 suffix); default http://localhost:11434
MCP_TRANSPORT	MCP transport type: sse or stdio
MCP_URL	MCP server URL (SSE transport only); default http://localhost:8000/sse
MCP_COMMAND / MCP_ARGS / MCP_CWD	MCP launch command, args, and working dir (stdio transport only — see the commented examples in .env_template)
TARGET_AGENT_PATH	Relative path to the target MCP server directory, e.g., examples/RagChatbot_MCPServer/ for the HR agent
GUIDANCE_FILE	Path to the policy guidance file, e.g., examples/RagChatbot_MCPServer/smith/guidance.txt
SYSTEM_VAR_FILE	Path to the system-variables JSON (e.g., examples/<agent>/smith/system_vars.json). Required — test generation fails without it
PROMPTFOO_CONFIG_FILE / PROMPTFOO_OUTPUT_FILE	Promptfoo red-team config and generated output paths
ATTACK_TOOLS	Comma-separated list of red-teaming tools to run during test generation. Valid values: ares, promptfoo, ares,promptfoo, none. Default: ares,promptfoo
ARES_HOME	Absolute path to the ARES installation directory (e.g., /path/to/smith/src/smith/test_generation/ares). Only required when ATTACK_TOOLS includes ares

See .env_template for the full list, including model sampling (TEMP, TOP_P), test-case evaluation thresholds, and refinement/clustering parameters.

Start the target agent and MCP server

Detailed instructions for each agent example can be found in the examples/<agent>/README.md.

Smith talks to a running target agent (for /chat and /extract_tool_call) and to its MCP server (to extract tool definitions). Start both before running any smith flag.

Each example under examples/<agent>/ ships its own agent.py (a FastAPI app exposing /chat and /extract_tool_call), server.py (the MCP server), and a requirements.txt. Using call-for-papers-mcp as a concrete example:

cd examples/call-for-papers-mcp
pip install -r requirements.txt

# Start the agent server on the port AGENT_URL points to (default 9000).
uvicorn agent:app --port 9000

This example's agent spawns its MCP server itself over stdio (agent.py launches python server.py), so you do not start the MCP server separately. Match .env accordingly:

MCP_TRANSPORT=stdio
MCP_COMMAND=python
MCP_ARGS=server.py
MCP_CWD=examples/call-for-papers-mcp

For an SSE-based MCP server instead, set MCP_TRANSPORT=sse and MCP_URL=http://localhost:8000/sse, and start that server on its own. Check each example's own README.md for specifics.

How It Works

Smith operates as an agent skill with a CLI backend. The AI agent reads instructions from SKILL.md and orchestrates the appropriate workflows by invoking the smith CLI or following embedded markdown guides.

┌────────────────────────────────────────────────────────────────────────────────┐
│                                     Smith                                      │
│                                                                                │
│  SKILL.md ──→ Orchestration ──→ smith CLI                                      │
│                                    │                                           │
│              ┌─────────────────────┼──────────────────────┬─────────┐          │
│              ▼                     ▼                      ▼         ▼          │
│         Policy              Test Case Gen              Policy     Policy       │
│         Creation                 │                     Testing   Refinement    │
│              │        ┌──────────┼──────────┬────────┐    │         │          │
│              ▼        ▼          ▼          ▼        ▼    └────⇄────┘          │
│         OPA Policy Legitimate  ARES    Promptfoo  Bypass                       │
│         (.rego)        │        │          │        │                          │
│                        └────────┴────┬─────┴────────┘                          │
│                                      ▼                                         │
│                              Test Case Evaluation                              │
└────────────────────────────────────────────────────────────────────────────────┘

Bypass cases are generated from the current policy (a Policy Creation output) rather than the guidance alone, so they close the loop back into Test Case Gen.

Core Concepts

Policy Creation

Create OPA policies from natural language specifications. The agent follows opa_policy/policy_creation/opa_policy_creation.md to generate a .rego policy file.

Inputs (located in <TARGET_AGENT_PATH>/smith/):

• guidance.txt — natural language policy rules
• tool_definitions.json — MCP tool list with parameters (auto-generated by smith --flag get_mcp_parameter), maps to input.arguments.*
• system_vars.json — system/session variables (e.g., roles, teams), maps to input.extensions.subject.*
• test_case_template.json (in ./references/) — defines the OPA input envelope structure

Output: OPA policy saved to <TARGET_AGENT_PATH>/smith/policy_generated.rego

The policy only references data available from tool arguments and system variables. If a guidance rule requires context not available in either, it is logged as a suggestion rather than added to the policy.

Test Case Generation

The agent follows test_generation/test_generation.md, which first asks which kind of test cases you want, then runs the matching command(s):

• Guidance-targeted cases — legitimate + adversarial cases derived from the guidance (broad coverage).
• Policy-bypass cases — adversarial cases that target divergences between the guidance and the current policy (requires an existing, non-empty policy).
• Both.

Guidance-targeted generation

# CLI commands used for test case generation
smith --flag test_generation

This runs the following stages:

1. Decomposition — Break guidance into testable atomic conditions
2. Variable Extraction — Identify system/mutable variables and their domains
3. Grey Condition Extraction — Identify ambiguous boundary conditions, user needs to approve guidances from grey condition extraction and then merge them to clean space
4. Legitimate and Adversarial Case Generation — Create benign (allow and disallow) inputs that should pass the policy. Create adversarial inputs using ARES and Promptfoo. Finally, combine into structured test cases

All results are stored in ./references/test_cases/.

Policy-bypass generation

# Requires an existing, non-empty policy at assets/policy.rego
smith --flag bypass_case_generation

Instead of generating from the guidance alone, this analyzes the current policy against the guidance to find where they diverge (e.g. a rule that omits an existence check, a numeric comparison with no type assertion, a substring match a malformed value slips past), then synthesizes adversarial cases that exercise each divergence:

1. Detect — Compare the full Rego policy to the guidance and classifies each divergence by mechanism (omitted_field, type_confusion, malformed_value, keyword_evasion). If the model returns malformed JSON, the call is retried (up to MAX_BYPASS_PARSE_ATTEMPTS, default 3) before giving up with an empty report.
2. Synthesize — Turn each divergence into concrete abstract cases.
3. Convert — Write them into ./references/test_cases/{allow,disallow}/ with a bypass_test_case prefix.

The divergence report is saved to ./references/bypass/ (JSON + Markdown). If the policy is missing or empty, generation is skipped with a message.

Test Case Evaluation

# CLI commands used for test case evaluation
smith --flag test_case_evaluation

This runs three steps:

1. Classify promptfoo cases — Match each promptfoo red-team case to a specific guidance rule. Uses local embedding similarity (sentence-transformers) to retrieve top-N candidate guidances, then an LLM selects the most relevant one from the candidates.
2. Validate labels — Verify that each test case's assigned label (allow/disallow) is correct using a three-tier approach, each tier assigns a confidence score:
   • Tier 1 (Rule-based) — Fast pattern matching for clear-cut cases (e.g., bypass keywords)
   • Tier 2 (Embedding + NLI) — Semantic similarity check; cases with high confidence and label agreement are resolved, others are escalated
   • Tier 3 (LLM Judge) — Cases with low confidence or label disagreement from Tier 2 are judged by an LLM
3. Generate HTML report — Produces an interactive report at references/test_case_report.html

The report groups all test cases by guidance item, with condition sub-tabs. Cases are labeled by source (Generated, ARES, Promptfoo) and type (allow/disallow) with color coding.

To regenerate the HTML report standalone:

cd src/smith/test_case_evaluation/visualization
python build_report.py

Test Case Translation

# CLI commands used for test case translation
smith --flag test_case_translation

Calls the agent's /extract_tool_call endpoint to extract tool names and argument values for each test case. The agent receives the user prompt and user profile, and returns the resolved tool name and arguments, which are written back into the test case files.

• Cases where the returned tool name doesn't match the expected one are flagged as mismatches and removed
• Cases labeled as "other" (general questions that don't invoke any tool) are moved to ./references/test_cases/malicious/ for future guardrail features
• Cases that already carry an arguments block are skipped, so translation is re-runnable. After adding bypass cases you can re-run it to translate only the new cases instead of re-translating the whole corpus

Policy Testing

Evaluate the current policy against all test cases and report pass/fail with coverage metrics.

# CLI commands used in policy testing
smith --flag policy_testing

Cross-Validation

Two cross-validation workflows handle different failure scenarios after policy testing:

Policy Cross-Validation — When policy testing returns 0 test cases evaluated or 100% failure, the policy likely has structural issues (input path mismatches or OPA syntax bugs). The agent follows opa_policy/policy_cross_validation/policy_cross_validation.md to diagnose and fix these issues before proceeding to refinement.

Test Case Cross-Validation — When policy testing produces mixed pass/fail results, some failures may be caused by mislabeled test cases rather than policy bugs. This workflow uses an LLM to check each failed case against the guidance and suggests corrections (move to correct folder or remove). Adversarial cases (filenames prefixed bypass_test_case or promptfoo_test_case) are an exception: any non-keep verdict is collapsed to remove rather than relabeled, so a failed malicious probe is discarded instead of being moved into allow/.

smith --flag cross_validate          # generate report of mislabeled cases
smith --flag apply_cross_validate    # apply approved corrections

Policy Refinement

Iterative improvement workflow:

1. Red Feedback — Cluster failed malicious inputs and patch policy rules, following opa_policy/policy_patch/policy_patch.md
2. Regal Formatting — Lint and format policy with Regal. It follows opa_policy/policy_regal/policy_regal.md
3. Duplication Removal — Detect and remove redundant rules via graph analysis, LLM review, and voting. It follows opa_policy/policy_duplication/policy_duplication.md

# CLI commands used in the refinement loop
smith --flag red_suggestion # clustering failed test cases
smith --flag regal_suggestion # get regal suggestions
smith --flag duplication_suggestion # get both LLM and graph generated duplication removal suggestions

Project Structure

smith/
├── .claude/                 # Claude Code agent configuration
├── assets/                  # Policy files and OPA data
│   ├── policy.rego          # Target policy under management
│   └── opa/                 # OPA intermediate results (AST, graphs, backups)
├── examples/                # Agent examples
├── opa_policy/              # Skills related to OPA policy
│   ├── policy_creation/     # OPA policy creation workflow
│   ├── policy_cross_validation/ # Fix structural/syntax issues (0 cases or 100% fail)
│   ├── policy_defect/       # Introduce intentional defects for testing (only for testing purpose, it is not part of Smith main skill)
│   ├── policy_patch/        # OPA policy patching workflow
│   ├── policy_regal/        # Regal formatting workflow
│   └── policy_duplication/  # Deduplication workflow
├── references/              # All intermediate results (incl. scorecard/ outputs)
├── scripts/                 # Utility bash scripts (e.g. clean_generated.sh)
├── pyproject.toml           # Packaging, dependencies, ruff/black config
├── src/smith/               # The `smith` Python package
│   ├── cli.py               # Main CLI entry point (smith.cli:main)
│   ├── policy_agent/        # OPA policy analysis and refinement
│   ├── policy_generation/   # MCP tool extraction and policy generation
│   ├── test_generation/     # Test case generation and translation pipeline
│   ├── test_case_evaluation/ # Label validation and report generation
│   ├── policy_testing/      # OPA scorecard harness (score_card.sh, coverage)
│   └── tools/               # Repo tooling (e.g. license headers)
├── tests/                   # Placeholder for the test suite (TODO)
├── test_generation/         # Test generation skill markdown file
├── .env_template            # Environment template
├── SKILL.md                 # Main agent skill instructions
└── README.md

Contributing

Contributions are welcome! Please read CONTRIBUTING.md for the development workflow, coding standards, source-file license headers, and the Developer Certificate of Origin (DCO) sign-off requirement. A green make ci locally means a green pipeline.

Security

Please report security vulnerabilities privately — see SECURITY.md. Do not open public issues for security reports.

Code of Conduct

This project follows the Contributor Covenant. By participating, you are expected to uphold it.

License

Smith is licensed under the Apache License 2.0.