Summary
Add a CLI command to automatically generate a promptfoo configuration file from Smith inputs (guidance, tool definitions, and system variables), so users can get a suggestion version of the config files, no longer need to create it manually before running adversarial tests.
User Story 1
- As a: security engineer onboarding a new target agent with Smith
- I want: a promptfoo configuration file to be automatically generated from my Smith inputs
- So that: I can run adversarial promptfoo tests immediately without manually authoring the config
Acceptance Criteria
Scenario: Generate promptfoo config from Smith inputs
Given a target agent with a populated smith/ folder (guidance.txt, tool_definitions.json, system_vars.json)
When the user runs smith --flag generate_promptfoo_config
Then a valid promptfoo configuration file is written to the expected output path
Design Sketch
It should follow:
- Generate a config file
- Validate the config file, by running promptfoo with this config file.
Summary
Add a CLI command to automatically generate a promptfoo configuration file from Smith inputs (guidance, tool definitions, and system variables), so users can get a suggestion version of the config files, no longer need to create it manually before running adversarial tests.
User Story 1
Acceptance Criteria
Scenario: Generate promptfoo config from Smith inputs
Given a target agent with a populated smith/ folder (guidance.txt, tool_definitions.json, system_vars.json)
When the user runs smith --flag generate_promptfoo_config
Then a valid promptfoo configuration file is written to the expected output path
Design Sketch
It should follow: