Skip to content

Add model-output chaos end-to-end example - #3827

Open
venkatkrish543re wants to merge 1 commit into
strands-agents:mainfrom
venkatkrish543re:feat/model-chaos-example
Open

Add model-output chaos end-to-end example#3827
venkatkrish543re wants to merge 1 commit into
strands-agents:mainfrom
venkatkrish543re:feat/model-chaos-example

Conversation

@venkatkrish543re

@venkatkrish543re venkatkrish543re commented Aug 14, 2026

Copy link
Copy Markdown

Description

Adds an end-to-end example for the model output chaos plugin (strands-agents/evals#284), mirroring chaos_testing.py: pre hook effects (FullRefusal, EmptyResponse) cancel the model call via BeforeModelCallEvent; post hook effects (MalformedJson, Confabulation, SuccessFraming) corrupt output via MessageAddedEvent. Includes an EventProbe plugin asserting both hook events fire in the real agent loop

Also extends the chaos testing docs page with a Model Output Effects section (kept on the existing page since the ChaosPlugin/ChaosCase machinery sections are shared )

Verification

Real Agent(plugins=[chaos_plugin]), Bedrock default model, 5 cases (baseline + 4 effect maps):

Related Issues

strands-agents/evals#284

Documentation PR

Type of Change

Documentation update

Testing

How have you tested the change? Verify that the changes do not break functionality or introduce new warnings.

  • I ran hatch run prepare

Checklist

  • I have read the CONTRIBUTING document
  • I have reviewed and understand every line of code in this PR, including any generated by AI tools, and I can explain why it works
  • My change is focused and reasonably small; I have split unrelated work into separate PRs
  • I have added any necessary tests that prove my fix is effective or my feature works
  • I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@venkatkrish543re
venkatkrish543re requested a review from a team as a code owner August 14, 2026 20:49
@github-actions github-actions Bot added python Pull requests that update python code documentation Documentation changes, improvements, additions, content updates, site improvements, examples, guides area-hooks Features or requests that might be implementable via hooks labels Aug 14, 2026
@venkatkrish543re
venkatkrish543re marked this pull request as draft August 14, 2026 20:54
Comment thread site/docs/examples/evals-sdk/model_chaos_testing.py Outdated
Comment thread site/docs/examples/evals-sdk/model_chaos_testing.py Outdated
Comment thread site/docs/examples/evals-sdk/model_chaos_testing.py Outdated
@ybdarrenwang

Copy link
Copy Markdown
Contributor

Can we rename the script chaos_model_testing.py, and the existing chaos_testing.py -> chaos_tool_testing.py?

Comment thread site/src/content/docs/user-guide/evals-sdk/chaos_testing.mdx Outdated
Comment thread site/src/content/docs/user-guide/evals-sdk/chaos_testing.mdx Outdated
Comment thread site/src/content/docs/user-guide/evals-sdk/chaos_testing.mdx Outdated
Comment thread site/src/content/docs/user-guide/evals-sdk/chaos_testing.mdx
Comment thread site/src/content/docs/user-guide/evals-sdk/chaos_testing.mdx Outdated
Adds chaos_model_testing.py, an end-to-end example for the model-output chaos plugin (strands-agents/evals#284): pre-model-call effects (FullRefusal, EmptyResponse) cancel the model call via BeforeModelCallEvent; post-model-call effects (MalformedJson, Confabulation, SuccessFraming) corrupt output via MessageAddedEvent. Uses assertion-mode judging for stable evaluation. Renames chaos_testing.py to chaos_tool_testing.py and extends the chaos-testing docs page with a Model Output Effects section.
@venkatkrish543re
venkatkrish543re force-pushed the feat/model-chaos-example branch from 89b1983 to 83675d4 Compare August 17, 2026 22:56
@venkatkrish543re
venkatkrish543re marked this pull request as ready for review August 17, 2026 23:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-hooks Features or requests that might be implementable via hooks documentation Documentation changes, improvements, additions, content updates, site improvements, examples, guides python Pull requests that update python code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants