Skip to content

feat: add employee hub agent example for benchmarking - #34

Merged
dhl123 merged 3 commits into
IBM:mainfrom
naamaz:examples/employee-agent
Jul 28, 2026
Merged

feat: add employee hub agent example for benchmarking#34
dhl123 merged 3 commits into
IBM:mainfrom
naamaz:examples/employee-agent

Conversation

@naamaz

@naamaz naamaz commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a new agent example under examples/employee/ — the Enterprise Employee Hub — to expand the set of benchmarking targets requested in #33.

The example is a self-contained MCP agent:

  • SQLite-backed employee hub with a pure-Python API layer (api/)
  • A FastMCP tool server (server.py)
  • A LangGraph ReAct agent (agent.py) plus a small web UI (web.py, ui/)
  • DB bootstrap/seed scripts (init_db.py, seed.py, schema.sql)

It also ships the Smith per-target-agent inputs so it can be pointed at directly:

  • smith/guidance.txt — natural-language policy rules
  • smith/system_vars.json — session/system variables
  • smith/mcp_tool_summary.md — tool summary
  • smith/smith_outputs/specs/*.json — generated per-tool specs

Notes

  • Only files under examples/employee/ are added (55 files); no other files are touched.
  • .DS_Store and __pycache__ are excluded via .gitignore.
  • .env.example contains placeholders only (no secrets).

Relates to #33

🤖 Generated with Claude Code

@naamaz
naamaz requested review from araujof and dhl123 as code owners July 23, 2026 09:17
@dhl123

dhl123 commented Jul 24, 2026

Copy link
Copy Markdown
Member

Hi @naamaz

I made some modifications to this PR for a complete Smith run and adds the generated artifacts, so the example can be used to exercise Smith end-to-end (policy creation → test generation → testing → refinement).

Changes

  • Add promptfooconfig.yaml: Promptfoo configuration for red-team test generation against the agent.
  • Align the model with the global .env: use the same model for agent execution, as the rest of the examples for consistency.
  • Document tool parameters in the config.
  • Expose the agent's /chat and /extract_tool_call endpoints.
  • Add tool_definitions.json: auto-generated MCP tool definitions (maps to input.arguments.*).
  • Run Smith end-to-end, generated guidance, OPA policy, and test cases. After refinement, 10 test cases still fail because they cannot be enforced by a stateless OPA policy (they need runtime DB context such as manager/direct-report relationships, target org, current country, computed leave balance; or a dynamic clock for the six-month expiry rule). All are documented in extension_suggestions.json as future-extension suggestions, not part of the current policy.
  • Add README.md documenting the example, Smith files, the CLI workflow, and testing results.

Notes

  • The goal here is to confirm Smith runs end-to-end and produces meaningful output on a messy, not a fully correct policy.

Summary

Overall I think this is good to go (i.e. ready to be merged).

@naamaz

naamaz commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

It looks good to me. Please go ahead and merge it. @dhl123

@dhl123

dhl123 commented Jul 27, 2026

Copy link
Copy Markdown
Member

Hi @naamaz, please also rebase the previous commits to include your signature. You can use "git commit -s -m". The merging is blocked because some commits do not have verified signatures. Thanks!

naamaz and others added 3 commits July 28, 2026 09:49
Add the Enterprise Employee Hub example agent under examples/employee:
a SQLite-backed employee hub with a pure-Python API layer, a FastMCP
tool server, and a LangGraph ReAct agent. Includes the Smith inputs
(smith/guidance.txt, system_vars.json, mcp_tool_summary.md) and
generated tool specs so it can be used as a benchmarking target.

Relates to IBM#33

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: naamaz <naamaz@il.ibm.com>
Signed-off-by: Hailun Ding <hailun.ding@ibm.com>
Signed-off-by: Hailun Ding <hailun.ding@ibm.com>
@naamaz
naamaz force-pushed the examples/employee-agent branch from c3b60df to 11084f9 Compare July 28, 2026 07:05
@naamaz

naamaz commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

@dhl123 Done. Rebased so all commits are signed and now show as Verified. Ready to merge whenever you're ready :) thanks!

@araujof araujof changed the title examples: add employee hub agent example for benchmarking feat: add employee hub agent example for benchmarking Jul 28, 2026

@araujof araujof left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dhl123
dhl123 merged commit 4b313cb into IBM:main Jul 28, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants