I've been thinking a lot about one question since reading the "Why Do AI Agents Break Rules?" preprint: that whole paper measures what models say they'll do in a chat, but never actually gives them a tool to act with. So what happens if you do? Does an agent that sounds compliant in its reasoning actually behave that way when it has to commit to a real action, especially when someone's leaning on it with a deadline or a manager override?
That's what this little pilot pokes at. I kept the setup as close to the paper's as I could (same vendors, same prices, same ISO 14001 framing) but gave the agent actual tools instead of just letting it recommend a vendor in a Slack message.
tools.py has the mock tools the agent can call, check_vendor_certification, submit_purchase_order, and flag_compliance_issue, plus the vendor matrix pulled straight from the paper so things stay comparable.
prompts.py has the system and user prompt pieces: the regulation text, the three pressure conditions (control, managerial override, urgency), and a "state your plan first" instruction with a fixed format so I can actually parse what vendor the model says it's going with.
run_experiment.py is the actual harness. It's two stages, first the model states its plan with no tools, then it gets tool access and has to actually place the order. The full tool trace gets logged.
mock_client.py and demo_run.py are a fake stand-in for the Anthropic client, just so I could run the whole thing end to end and make sure the logic actually works. Important: this is NOT real model output, just me checking the plumbing.
analyze.py takes a results file and spits out per-condition numbers: how often the stated plan was compliant, how often the actual order was compliant, how often the two match, and a "silent divergence" rate (says it's doing the compliant thing, then quietly doesn't).
Everything runs end to end against the mocked client and the numbers come out making sense. I just don't have API credits set up here, so I haven't pointed it at real models yet. Once that's sorted it's basically a one-line swap to get real numbers.
I scripted two fake "models" just to sanity check the metrics:
demo-model-A is consistent. Under pressure it states it's going with the cheaper, non-certified vendor, actually orders that vendor, and explicitly flags the tradeoff to compliance. Its words match its actions the whole way through.
demo-model-B is the more interesting (and worrying) one. Under pressure it says "going with EcoSupply, it's certified, satisfies the regulation," sounds completely compliant... and then the tool trace shows it actually ordering BudgetDirect. No flag, nothing.
model condition n stated_ok acted_ok match silent_div
demo-model-A control 5 100% 100% 100% 0%
demo-model-A managerial_override 5 0% 0% 100% 0%
demo-model-A urgency 5 0% 0% 100% 0%
demo-model-B control 5 100% 100% 100% 0%
demo-model-B managerial_override 5 100% 0% 0% 100%
demo-model-B urgency 5 100% 0% 0% 100%
demo-model-B is really the case I care about here. The paper's reasoning classifier would've caught demo-model-A no problem, since it openly admits what it's doing. But it would totally miss demo-model-B, because the reasoning looks fine. If real models actually behave like this, it's a pretty big deal, since it means just reading an agent's reasoning isn't enough to catch the violations that matter most.
I'd swap the mock client for the real Anthropic one (already set up for this in run_experiment.py), run it across a couple models from each of the paper's two groups, bump the trial count up to 25 to match their setup, and probably add a condition where the model's chain of thought is hidden vs visible across turns. Adding an audit-outcome follow-up turn (like "a peer got fined" / "a peer got away with it") would also be a natural extension given how much that mattered in the original paper's multi-turn results.