fix(ai): arg resilience — coerce stringified args, reject non-object args, + LLM-response fuzz harness - #1275
Merged
Conversation
…pts/query/…)
Some models serialize nested object/array tool params as JSON strings even though
the schema says type:object/array. A stringified `opts` then reached _get_action_label
as a str and crashed the batch label with `AttributeError: 'str' object has no
attribute 'get'` (opts.get); it also silently vanished in _run_runner
(_sanitize_child_opts returns {} for a non-dict), so the sub-runner ran without the
model's intended options.
Fix the whole class once at the tool-call boundary: coerce_stringified_args() parses
any arg the tool schema declares object/array but the model sent as a string, run in
_process_tool_calls BEFORE decrypt/convert (so _decrypt_dict doesn't mangle it). This
covers opts, targets, query, choices, add_finding fields — present and future. A
malformed value is left as-is so the handler returns a clean error. Also guard
_get_action_label defensively (display must never crash on a residual str opts).
Generalizes the query-specific coercion from #1273 to every object/array arg.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NNjPggRSVZ2xnLb7ZxWP5H
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…l args Adds tests/unit/test_ai_resilience.py: fakes the LLM (patches call_llm) and drives the REAL _run_loop with a battery of weird responses, asserting the invariant that a malformed response is handled turn-locally and the loop SURVIVES — never aborts the session via the top-level `except -> Error.from_exception; return` catch-all (monitored precisely) and never raises. Two layers: a curated table (stringified opts/query/targets/choices, wrong-type args, broken JSON, missing fields, unknown tools, empty/huge/no-usage responses) and a seeded fuzzer (200 random malformed argument strings across every tool). The harness immediately surfaced a real gap it now guards: a model emitting non-object arguments (a bare JSON int/array/string, e.g. `12345` or `["nmap"]`) made tool_call_to_action call `.items()` on a non-dict -> AttributeError -> top-level catch-all -> whole conversation aborted. Guard it: reject non-dict arguments to None so the caller feeds a clean error back and the loop continues. This is the tool that would have caught the stringified-opts/query crashes before they were hit manually. Complements the boundary coercion in this PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NNjPggRSVZ2xnLb7ZxWP5H
ocervell
changed the title
ocervell
added a commit
that referenced
this pull request
Jul 5, 2026
…ackend) (#1280) ## Bug ``` 🟢Query({'_type': 'ip', 'host': 'cachyos.local'}) -> failed results (limit: 10) [ERR] ai TypeError: '>=' not supported between instances of 'int' and 'str' File ".../query/json.py", line 182, in _execute_search if limit and len(matched) >= limit: ``` The model sent `query_workspace`'s `limit` as a **string** (`"10"`); it reached the backend and broke the `>=` comparison. Same "model stringifies args" class as #1275, but `coerce_stringified_args` only handles object/array params — not scalar ints like `limit`. ## Fix Coerce `limit` to `int` in `_handle_query`; bad/None values fall back to the default (100). ## Tests Stringified `"10"` → `10` reaches the backend; a non-numeric limit falls back to 100. `test_ai_actions.py` 99 passed; the original crash reproduces as resolved. Targets `ai-resiliency` (#1241). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the agent loop resilient to weird LLM tool-call responses — and adds the harness that proves it.
The fixes
opts/query/targets/choices/…). Some models serialize nested object params as JSON strings even though the schema saysobject/array. A stringifiedoptscrashed_get_action_label('str' has no attribute 'get') and silently vanished in_run_runner.coerce_stringified_args()runs at the tool-call boundary (before decrypt/convert) andjson.loads-es any such arg once. Malformed values are left as-is for a clean handler error.12345,["nmap"]) madetool_call_to_actioncall.items()on a non-dict →AttributeError→ the top-level catch-all → whole conversation aborted. Now rejected cleanly toNoneso the caller feeds an error back and the loop continues. (Surfaced by the harness below.)The harness —
tests/unit/test_ai_resilience.pyFakes the LLM (patches
call_llm) and drives the real_run_loopwith a battery of weird responses, asserting the invariant:Two layers:
argumentsstrings across every tool; deterministic so any failure reproduces.This is the tool that would have caught the stringified-
opts/querycrashes before they were hit by hand — and it found the non-object-args gap on its first run.Tests
TestWeirdToolCalls+TestWeirdContentResponses+TestMalformedArgFuzzer(28) all green;TestCoerceStringifiedArgs+tool_call_to_actionnon-object guard. Full AI suite: no new failures vs branch baseline.Targets
ai-resiliency(#1241).🤖 Generated with Claude Code