Skip to content

fix(chat): finalize streamed tool calls only when the stream completes - #629

Open
mattapperson wants to merge 6 commits into
mainfrom
claude/defer-streamed-tool-call-finalize
Open

mattapperson wants to merge 6 commits into
mainfrom
claude/defer-streamed-tool-call-finalize

Conversation

@mattapperson

@mattapperson mattapperson commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Related to #574

Root cause of #574 is server-side. OpenRouter's backend does not honor eager_input_streaming: false on tools; this package already forwards the field correctly (tools[i].eager_input_streaming), so the invalid streamed Claude tool arguments can't be fixed here. This PR fixes a provider-side bug that hid that failure.

Bug: In doStream, a tool-call part was emitted as soon as the accumulated arguments happened to parse as JSON. With an "early root close" from the upstream (a valid JSON prefix followed by more fragments), e.g. fragments '{"a":[1]}', '</a>\n<b>', '[2]}', the provider emitted a truncated-but-valid tool call with input {"a":[1]} and then kept emitting tool-input-delta parts after it. The model's real (malformed) payload never reached the AI SDK's tool-call error handling / experimental_repairToolCall (#575).

Fix: Tool calls are no longer finalized mid-stream. They are finalized in flush() using the full accumulated arguments:

  • finish reason tool-calls (including the existing missing/other -> tool-calls inference from Kimi K2.5 returns undefined finishReason after tool calls, breaking agentic loops #420) or stop: every tool call is forwarded, invalid JSON as-is ('{}' only for empty arguments, as before).
  • other finish reasons (length, error, content-filter): only tool calls with parsable arguments are forwarded, matching previous behavior for truncated output.
  • every tool call is still emitted exactly once, with the full tool-input-start -> delta... -> end -> tool-call lifecycle, including single-chunk arguments and streams that end without a finish_reason.

The early emission dates back to the original AI SDK v5 port (#77), inherited from the openai-compatible provider; nothing relies on mid-stream emission beyond event ordering. Because arguments are no longer parsed per delta, the JsonCompletenessScanner optimization from #586 is no longer needed and is removed (it was internal, not exported).

Behavior change: tool-input-start/tool-input-delta still stream as arguments arrive (including single-chunk arguments), but tool-input-end and tool-call now arrive at the end of the stream instead of as soon as the JSON closes, so a tool's onInputAvailable and the UI tool-input-available state fire at end of stream. With ai@7, tool execution timing is unchanged (tools already run after the model call ends). For whitespace trailing a complete object, the input now includes it (still valid JSON).

Delta merging hardening (also in this PR):

  • Tool-call deltas without an index (optional in the schema since chore: make index optional #99) are now handled. Before, the first one landed at toolCalls[-1] and was never emitted. Now a delta with a new id (or, without an id, a function name) starts a new tool call and other deltas continue the most recent one. A repeated name with no id only splits off a separate call when the previous arguments are a complete JSON object/array and the delta starts a new, different one. The first delta at a new index binds to an indexless call it continues.
  • A delta that reuses an existing index with a different non-empty id starts a new tool call instead of being concatenated onto the previous one under the wrong name. Provider indexes map to dense internal slots.
  • An identical resend of already complete JSON object/array arguments ({"a":1} sent twice) is ignored instead of producing {"a":1}{"a":1}. This only triggers when the accumulated arguments are a complete JSON container and the delta is byte-identical, so appending it could never produce valid JSON.
  • Tool calls dropped at flush because their arguments are incomplete (length, error, ...) now get a closing tool-input-end (no tool-call), so UI parts don't stay in input-streaming.

Checklist

  • I have run pnpm stylecheck and pnpm typecheck
  • I have run pnpm test and all tests pass
  • I have added tests for my changes (if applicable)
  • I have updated documentation (if applicable)

Changeset

  • I have run pnpm changeset to create a changeset file

Tests added (mocked SSE streams), in src/chat/index.test.ts plus e2e/issues/issue-574-streamed-tool-call-finalization.test.ts (which also checks experimental_repairToolCall receives the full arguments): early root close, multi-chunk args with braces in strings, single-chunk args, interleaved parallel tool calls, stream without finish_reason, invalid args with stop, truncated args with length (plus tool-input-end for the dropped call), no per-delta parsing, deltas without index (single call, two calls, multi-chunk continuations), a different id reusing an index, repeated id at the same index, identical full-args resend, and a fragment equal to incomplete accumulated args (not dropped).


Devin Review

devin-ai-integration[bot]

This comment was marked as resolved.

mattapperson and others added 3 commits November 4, 2026 09:37
A tool-call part was emitted as soon as the accumulated arguments parsed
as JSON. An upstream early root close (valid JSON prefix followed by more
fragments) therefore produced a truncated tool call followed by stray
tool-input-delta parts, hiding the malformed input from the AI SDK's
tool-call repair. Tool calls are now finalized in flush() with the full
accumulated arguments; each is still emitted exactly once, including when
the stream ends without a finish_reason.

Related to #574

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… test

Emit tool-input-start and the initial tool-input-delta as soon as a tool
call's first chunk carries arguments, keeping only tool-input-end and
tool-call deferred to flush. Add e2e/issues regression test for #574.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Handle tool-call deltas without an index: a new id (or function name)
  starts a new tool call, other deltas continue the most recent one.
  Previously these were stored at index -1 and never emitted.
- Start a new tool call when a delta reuses an index with a different id.
- Ignore an identical resend of already complete JSON arguments.
- Emit tool-input-end for started tool calls dropped at flush (e.g.
  'length'/'error' finish) so UI parts leave input-streaming.
- Clarify the changeset on the tool-call / tool-input-end timing change.
@mattapperson
mattapperson force-pushed the claude/defer-streamed-tool-call-finalize branch from c0284b9 to 958c45a Compare November 4, 2026 14:39
devin-ai-integration[bot]

This comment was marked as resolved.

Treat an id-less indexless delta that repeats the current tool call's
name as a continuation, and bind the first delta at an unseen index to
the most recent tool call when that call started without an index and
the delta does not identify a different call.
devin-ai-integration[bot]

This comment was marked as resolved.

…oundaries

A repeated function name without an id no longer continues the previous
tool call once its arguments are already a complete JSON object/array.
devin-ai-integration[bot]

This comment was marked as resolved.

A repeated function name without an id now starts a separate tool call
only when the previous arguments are a complete JSON object/array and
the delta begins a new, different one. Whitespace and identical resends
continue the call. Cheap string checks gate the full parse so long
same-name streams are not reparsed per fragment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants