FEAT: add MessageToolCallScorer for tool calls recorded in stored messages - #2854
Open
WatchTree-19 wants to merge 1 commit into
Open
WatchTree-19 wants to merge 1 commit into
WatchTree-19 wants to merge 1 commit into
Conversation
…sages Scores the ToolsCalled condition from function_call and function_call_output pieces, with no trace pipeline. A tool counts when a model-authored call is paired by call ID with a later output in the conversation through the scored response. PyRIT tolerant-mode dispatch errors do not count. Stored messages are partial evidence, so the scorer returns true or undetermined, never false. Handles both the Responses and Chat Completions call formats. No observation payload or replay yet; that needs a message-scoped tool-event payload.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
MessageToolCallScorer, the message-side counterpart toOtelToolCallScorer. It scores the sameToolsCalledcondition from the function-call pieces PyRIT already stores, so it works without a trace pipeline.A tool counts when a model-authored
function_callpiece is paired by call ID with a laterfunction_call_outputpiece in the scored message's conversation, up to and including that message, which is the same scoperesolve_message_trace_scopeuses. Both the Responses (call_id,name) and Chat Completions (id,function.name) formats are read. Requested calls with no output do not count, calls insimulated_assistanthistory are ignored, and PyRIT's tolerant-mode dispatch outputs (function_not_found,missing_function_name,malformed_arguments) do not count, since the function never ran. An error returned by the tool itself still counts, in line withToolsCalledcounting an attempt regardless of success.Stored messages are partial evidence: hosted tools and several response section types are not persisted, and targets that run tools themselves store no outputs. The scorer therefore returns true when every required tool ran and undetermined otherwise, never false, and it composes with
OtelToolCallScorerunderTrueFalseScoreAggregator.OR. It does not create observations, soscore_observation_asyncis not supported for it.Tests and Documentation
tests/unit/score/test_message_tool_call_scorer.py(30 tests): both call formats, unpaired requests, dispatch errors versus tool-reported errors, call ID pairing, output ordering, later turns and other conversations, role and data type filtering, multiple required tools, an OR composite withOtelToolCallScorer, and an end-to-endPromptSendingAttackthrough a mockedOpenAIResponseTargettool loop.tests/unit/score,tests/unit/docsandtests/unit/registrypass; ruff and ty are clean.doc/code/scoring/5_tool_call_scorer.pyand synced the.ipynbwith JupyText. No code cells changed, so no re-execution was needed.