Skip to content

fix(scorers): detect AI SDK v5 tool parts in toolCallAccuracy #1364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
VihaanAgarwal wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(scorers): detect AI SDK v5 tool parts in toolCallAccuracy #1364

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/tool-call-accuracy-v5-parts.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"@voltagent/scorers": patch
---

Fix the code-based `toolCallAccuracy` scorer missing tool calls in AI SDK v5 message parts

The scorer read tool calls from message `parts` only when a part's `type` was exactly `"tool_call"`. AI SDK v5 encodes a tool call in message parts as `type: "tool-<toolName>"`, so those calls were skipped and the scorer reported the expected tool as not called even when it was. It now recognizes `tool-<name>` parts too, matching how it already reads them from tool-call lists.
30 changes: 30 additions & 0 deletions packages/scorers/src/tool-call-accuracy.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,36 @@ describe("createToolCallAccuracyScorerCode", () => {
expect(result.score).toBe(1);
});

it("extracts tool names from AI SDK v5 message parts (type: tool-<name>)", async () => {
const scorer = createToolCallAccuracyScorerCode({
expectedTool: "searchProducts",
});

const result = await scorer.scorer({
payload: {
messages: [
{
id: "msg-1",
role: "assistant",
parts: [
{
type: "tool-searchProducts",
toolCallId: "call-1",
state: "output-available",
input: {},
output: {},
},
],
},
],
} satisfies ToolCallAccuracyPayload,
params: {} as ToolCallAccuracyParams,
});

expect(result.status).toBe("success");
expect(result.score).toBe(1);
});

it("supports custom buildPayload mappings", async () => {
const scorer = createToolCallAccuracyScorerCode<{ events: Array<{ name: string }> }>({
expectedTool: "searchProducts",
Expand Down
6 changes: 5 additions & 1 deletion packages/scorers/src/tool-call-accuracy.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -276,7 +276,11 @@ function extractToolNamesFromParts(parts: unknown[]): string[] {
}

const partType = normalizeMessageType(part.type);
if (partType !== "tool_call") {
// AI SDK v5 encodes a tool call in message parts as `type: "tool-<name>"`,
// not `"tool_call"`. Accept those too (extractToolName reads the name from
// the type); tool results and streaming events are still excluded because
// shouldExtractToolNameFromType returns false for them.
if (partType !== "tool_call" && !shouldExtractToolNameFromType(partType)) {

@cubic-dev-ai cubic-dev-ai Bot Jun 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: The stream-event guard shouldExtractToolNameFromType rejects valid AI SDK v5 tool-<toolName> parts when the tool name collides with excluded prefixes or suffixes (e.g., input_validator → tool_input_validator matches the tool_input_ prefix, generate_delta matches the _delta suffix). This causes false negatives where the scorer silently skips actually-called tools, undermining the fix for AI SDK v5 support.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At packages/scorers/src/tool-call-accuracy.ts, line 283:

<comment>The stream-event guard `shouldExtractToolNameFromType` rejects valid AI SDK v5 `tool-<toolName>` parts when the tool name collides with excluded prefixes or suffixes (e.g., `input_validator` → `tool_input_validator` matches the `tool_input_` prefix, `generate_delta` matches the `_delta` suffix). This causes false negatives where the scorer silently skips actually-called tools, undermining the fix for AI SDK v5 support.</comment>

<file context>
@@ -276,7 +276,11 @@ function extractToolNamesFromParts(parts: unknown[]): string[] {
+    // not `"tool_call"`. Accept those too (extractToolName reads the name from
+    // the type); tool results and streaming events are still excluded because
+    // shouldExtractToolNameFromType returns false for them.
+    if (partType !== "tool_call" && !shouldExtractToolNameFromType(partType)) {
       continue;
     }
</file context>

continue;
}

Comment on lines +283 to 286

@cubic-dev-ai cubic-dev-ai Bot Jun 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: AI SDK v5 dynamic tool parts with type: "dynamic-tool" and a toolName field are still skipped because normalizeMessageType produces "dynamic_tool", which fails both "tool_call" and shouldExtractToolNameFromType (startsWith("tool") is false). extractToolName would otherwise read the toolName from these parts, so dynamic tool calls will still score as not called.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At packages/scorers/src/tool-call-accuracy.ts, line 283:

<comment>AI SDK v5 dynamic tool parts with `type: "dynamic-tool"` and a `toolName` field are still skipped because `normalizeMessageType` produces `"dynamic_tool"`, which fails both `"tool_call"` and `shouldExtractToolNameFromType` (`startsWith("tool")` is false). `extractToolName` would otherwise read the `toolName` from these parts, so dynamic tool calls will still score as not called.</comment>

<file context>
@@ -276,7 +276,11 @@ function extractToolNamesFromParts(parts: unknown[]): string[] {
+    // not `"tool_call"`. Accept those too (extractToolName reads the name from
+    // the type); tool results and streaming events are still excluded because
+    // shouldExtractToolNameFromType returns false for them.
+    if (partType !== "tool_call" && !shouldExtractToolNameFromType(partType)) {
       continue;
     }
</file context>
Suggested change
if (partType !== "tool_call" && !shouldExtractToolNameFromType(partType)) {
continue;
}
// AI SDK v5 encodes a tool call in message parts as `type: "tool-<name>"`,
// not `"tool_call"`. Dynamic tools use `type: "dynamic-tool"` with a
// `toolName` field. Accept those too (extractToolName reads the name from
// the type or toolName); tool results and streaming events are still
// excluded because shouldExtractToolNameFromType returns false for them.
if (partType !== "tool_call" && partType !== "dynamic_tool" && !shouldExtractToolNameFromType(partType)) {
continue;
}

Expand Down
Loading