Skip to content

[FEATURE]: support auto generation/updation of promptfoo config files & UI connecting IR and smith - #39

Draft
dhl123 wants to merge 2 commits into
mainfrom
hl/promptfoo_config_auto
Draft

[FEATURE]: support auto generation/updation of promptfoo config files & UI connecting IR and smith#39
dhl123 wants to merge 2 commits into
mainfrom
hl/promptfoo_config_auto

Conversation

@dhl123

@dhl123 dhl123 commented Jul 29, 2026

Copy link
Copy Markdown
Member

Summary

Closes: #36

Changes

  • Auto-generate Promptfoo config: new smith --flag generate_promptfoo_config stage that creates/updates a Promptfoo redteam configuration from guidance and system variables, using a customizable template (PROMPTFOO_CONFIG_TEMPLATE).
  • Policy Explorer UI bridge: added explorer_server.py to serve an interactive HTML view of the policy alongside IR data for visual inspection. It also supports reseting smith configuration based on selected guidance.
  • LLM-based tool classification for promptfoo cases: during test generation, promptfoo cases are now classified to a target tool name via a single LLM call against the MCP tool definitions, removing the hardcoded "Promptfoo" placeholder. This steps aims to make test translation apply the same tool-name mismatch check to all cases uniformly. Also, when LLM calls several tools in one turn to make the decision, it avoids the promptfoo cases are assigned with the first tool call name blindly, causing false positives. For example, employee agent usually calls list_employee before every add_employee prompt. So a malicious promptfoo case will be assigned a wrong label list_employee
  • Refactored get_tool_definitions() helper to deduplicate MCP tool extraction across CLI flags.

Checks

  • make ci passes (lint, Rego lint, license headers, build smoke)
  • make test passes (policy scorecard — needed if policy behavior changed)
  • CHANGELOG.md updated under ## [Unreleased] (if user-facing)
  • Commits are signed off for the DCO (git commit -s)

Notes (optional)

Design sketch, screenshots, or extra context.

Signed-off-by: Hailun Ding <hailun.ding@ibm.com>
@dhl123 dhl123 added the enhancement New feature or request label Jul 29, 2026
@dhl123 dhl123 changed the title [FEATURE]: support auto generation/updation of promptfoo config files [FEATURE]: support auto generation/updation of promptfoo config files & UI connecting IR and smith Jul 29, 2026
Signed-off-by: Hailun Ding <hailun.ding@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE]: Auto-generate promptfoo configuration from Smith inputs

1 participant