fix(reflect): keep tools schema stable across forced turns and synthesis - #4066
fix(reflect): keep tools schema stable across forced turns and synthesis#4066leilei3167 wants to merge 5 commits into
Conversation
Ollama/LM Studio used to narrow tools[] per named tool_choice and the final synthesis swapped the system prompt / dropped tools, which forces hybrid models to full-reprefill. Encode forced tools as a message suffix and make in-budget synthesis a prefix extension instead. Fixes vectorize-io#3865
|
@nicoloboschi quick check-in — forced Reflect turns on ollama/lmstudio keep the full tools schema (force via a user suffix) and in-budget final synthesis stays a prefix extension instead of swapping system/dropping tools. Happy to tweak the nudge wording if you want it stronger/weaker. |
Drop unused build_final_prompt import (ruff) and give forced-synthesis mocks a third call_with_tools response for tool_choice=none synthesis.
Resolve test_reflect_agent conflict by keeping prefix-stable call_with_tools synthesis mocks (drop legacy mock_llm.call path).
|
The current free-threaded 3.14 failure is deterministic rather than the earlier flake, so another empty retrigger will not clear it. Both failures come from This file is outside this PR's diff and the same assertion is present on current |
|
Thanks for the clear diagnosis — agreed this is the baseline |
Problem
On Ollama / LM Studio (providers that silently drop
tool_choice="required"), Reflect's forced iterations narrowtools[]to the named function, and the forced final synthesis swaps the system prompt and drops tools entirely. Templates that render the tools block before system text (Qwen-style) then diverge early in the prompt. Hybrid / Gated-DeltaNet models cannot resume from a partial prefix, so each iteration full-prefills the accumulated history.Root cause
OpenAICompatibleLLM.call_with_toolsfilterstoolsdown to the forced function for namedtool_choice, then omitsrequiredfor ollama/lmstudio — correct for forcing a call, but the tools schema changes every forced turn._forced_final_synthesisstarts a fresh tool-less conversation withbuild_final_system_prompt, so synthesis is not a prefix extension of the agent loop.Fix
toolsarray byte-identical. Encode the forced function as a user-message suffix (You must call \name` now) instead of filteringtools[]. Providers that honorrequired` still use the previous narrow+required path.toolsarray; append a "produce the final answer, no further tool calls" nudge and call withtool_choice=none. Over-budget split synthesis is unchanged (conversation no longer fits as a prefix).How verified
pytest tests/test_tool_choice_required_downgrade.py tests/test_lmstudio_tool_choice.py tests/test_reflect_agent.py tests/test_reflect_split_synthesis.py -m 'not hs_llm_core'→ 83 passed (env-only pg0/real-LLM setup errors unrelated)Fixes #3865