Bedrock prompt cache misses on the next turn: reasoning is not replayed and dynamic instructions sit after the system cache point #617

Summary

With Bedrock Claude models, promptCache: true and extended thinking, the first call of a turn sent shortly after the previous one (well inside the 5 min TTL) usually reads only the system prompt from the cache and writes the whole conversation again. Two independent causes:

1. C. Reasoning is not replayed, so the rebuilt history differs from what was cached. During a run, assistant steps carry signed reasoningContent. On the next turn the history is rebuilt from the stored message: formatAgentMessages moves think parts to additional_kwargs.reasoning_content, and convertToConverseMessages does not emit them (with or without a signature). From the first assistant step of the previous turn onward the bytes differ, so the previous turn's tail checkpoint can never match.
2. A. Dynamic instructions follow the system cache point on Bedrock. AgentContext.buildSystemMessage places the dynamic block (for example the code-execution file note, the web-search date context, memories) after the system cachePoint. For Anthropic and OpenRouter it is moved next to the last human message (shouldMoveDynamicInstructions), but not for Bedrock. Any change in that block, such as a newly attached file or the hour rolling over, invalidates every message after the system prompt even when the history is identical.

A third factor makes (1) worse: Bedrock gets a single message checkpoint at the tail (addBedrockTailCacheControl), and Bedrock only looks back about 20 content blocks from each cachePoint. The one prefix that does still match (up to the human message that opened the previous turn, written as the tail of that turn's first call) is out of reach once the previous turn had a few tool steps.

Reproduction

1. Bedrock Claude model with promptCache: true and reasoning enabled, an agent with at least one tool.
2. Turn 1: a request that makes the model call a tool 5 or more times.
3. Within a minute, turn 2: any short follow-up.
4. Usage of the first call of turn 2: cacheReadInputTokens equals the system prompt size, and cacheWriteInputTokens covers the entire history.
5. Variant for (A): same, but attach a file on turn 2 so the code-execution file note changes. Even with a one-call turn 1, turn 2 reads only the system prompt.

A unit-level repro without network: build turn 1 in memory (AIMessages with signed reasoning_content plus tool calls), run each call through addBedrockTailCacheControl + systemRunnable + convertToConverseMessages, then rebuild turn 2 from the persisted payload via formatAgentMessages, and compare the flattened Converse blocks against a prefix cache with a 20-block lookback.

Suggested fixes

• A: on Bedrock with prompt caching, move the dynamic block out of the system message into a message right after the latest human message (as is already done for Anthropic), so it sits after the message checkpoints. It is worth wrapping it in an explicit marker, because Converse merges consecutive user messages into one turn and the block otherwise reads as part of the user's text.
• Lookback: place the second message cachePoint on the latest human message before the tail. On a turn's first call that is the human message that opened the previous turn, whose prefix is already cached. This stays within the limit of 4 cache points.
• C: replay reasoning to Bedrock as reasoningContent blocks when a signature is available, and keep the signature when the run's content is persisted (today only Vertex signatures survive). Until signatures are persisted, the two changes above recover most of the loss, because the matching prefix ends right before the first assistant step.

Impact

Measured over one week on a production deployment: at least 44% of the turns sent less than 5 minutes after the previous one (352 of 805) read only the system prompt on their first call. Making the second checkpoint reachable plus moving the dynamic block brought the first call of each follow-up turn back to reading everything up to the previous turn's opening message.

Related in LibreChat

LibreChat persists think content parts without the provider signature, so even a converter that replays reasoning has nothing signed to send on the next turn. If preferred, this part can be tracked as a separate LibreChat issue.