Skip to content

fix(reflect): keep tools schema stable across forced turns and synthesis - #4066

Open
leilei3167 wants to merge 5 commits into
vectorize-io:mainfrom
leilei3167:fix/reflect-stable-tools-3865
Open

fix(reflect): keep tools schema stable across forced turns and synthesis#4066
leilei3167 wants to merge 5 commits into
vectorize-io:mainfrom
leilei3167:fix/reflect-stable-tools-3865

Conversation

@leilei3167

Copy link
Copy Markdown
Contributor

Problem

On Ollama / LM Studio (providers that silently drop tool_choice="required"), Reflect's forced iterations narrow tools[] to the named function, and the forced final synthesis swaps the system prompt and drops tools entirely. Templates that render the tools block before system text (Qwen-style) then diverge early in the prompt. Hybrid / Gated-DeltaNet models cannot resume from a partial prefix, so each iteration full-prefills the accumulated history.

Root cause

  1. OpenAICompatibleLLM.call_with_tools filters tools down to the forced function for named tool_choice, then omits required for ollama/lmstudio — correct for forcing a call, but the tools schema changes every forced turn.
  2. _forced_final_synthesis starts a fresh tool-less conversation with build_final_system_prompt, so synthesis is not a prefix extension of the agent loop.

Fix

  • Forced iterations (ollama/lmstudio): keep the full tools array byte-identical. Encode the forced function as a user-message suffix (You must call \name` now) instead of filtering tools[]. Providers that honor required` still use the previous narrow+required path.
  • In-budget final synthesis: keep the same system prompt and tools array; append a "produce the final answer, no further tool calls" nudge and call with tool_choice=none. Over-budget split synthesis is unchanged (conversation no longer fits as a prefix).

How verified

  • pytest tests/test_tool_choice_required_downgrade.py tests/test_lmstudio_tool_choice.py tests/test_reflect_agent.py tests/test_reflect_split_synthesis.py -m 'not hs_llm_core' → 83 passed (env-only pg0/real-LLM setup errors unrelated)

Fixes #3865

Ollama/LM Studio used to narrow tools[] per named tool_choice and the final
synthesis swapped the system prompt / dropped tools, which forces hybrid
models to full-reprefill. Encode forced tools as a message suffix and make
in-budget synthesis a prefix extension instead.

Fixes vectorize-io#3865
@leilei3167

Copy link
Copy Markdown
Contributor Author

@nicoloboschi quick check-in — forced Reflect turns on ollama/lmstudio keep the full tools schema (force via a user suffix) and in-budget final synthesis stays a prefix extension instead of swapping system/dropping tools. Happy to tweak the nudge wording if you want it stronger/weaker.

Drop unused build_final_prompt import (ruff) and give forced-synthesis
mocks a third call_with_tools response for tool_choice=none synthesis.
Resolve test_reflect_agent conflict by keeping prefix-stable
call_with_tools synthesis mocks (drop legacy mock_llm.call path).
@koriyoshi2041

Copy link
Copy Markdown
Contributor

The current free-threaded 3.14 failure is deterministic rather than the earlier flake, so another empty retrigger will not clear it. Both failures come from tests/test_opencode_go_session_header.py:110: the test reads result.ok, while LLMCallResult exposes the parsed payload through result.content; test_named_result_stubs.py flags the same line for that reason.

This file is outside this PR's diff and the same assertion is present on current main (eb32332e), so it does not look owned by the stable-tools change. Once main fixes that test, a rebase should separate this PR's signal from the baseline failure.

@leilei3167

Copy link
Copy Markdown
Contributor Author

Thanks for the clear diagnosis — agreed this is the baseline result.ok vs result.content assertion on main, not from the stable-tools change. I'll rebase once that test lands on main so CI can speak for this PR again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reflect's per-iteration tools narrowing (and final system-prompt swap) forces a full prefill on hybrid/Ollama models

2 participants