fix(bedrock): inline mid-conversation system messages to preserve prompt cache#4534
fix(bedrock): inline mid-conversation system messages to preserve prompt cache#4534mickgvirtu wants to merge 1 commit into
Conversation
|
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughSummary by CodeRabbit
Walkthrough
ChangesBedrock inline system reminders and tool CachePoint
Sequence DiagramssequenceDiagram
participant Client
participant BedrockConverter
participant SystemMessage
participant ToolCall
participant ToolResult
Client->>BedrockConverter: ConvertBifrostMessagesToBedrockMessages(inlineSystemReminders=true)
BedrockConverter->>SystemMessage: Leading system messages → hoist to system block
BedrockConverter->>SystemMessage: Mid-conversation system messages → wrap as <system-reminder> user turn
BedrockConverter->>ToolCall: Emit tool call, append CachePoint if CacheControl present
BedrockConverter->>ToolResult: Emit tool result, append CachePoint if CacheControl present
BedrockConverter-->>Client: messages[], systemBlocks[], error
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 golangci-lint (2.12.2)level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies" Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
core/providers/bedrock/bedrock_test.go (1)
6471-6477: ⚡ Quick winCover the lone
developerbranch too.The test comment and converter predicate both cover
systemanddeveloper, but this test only exercisessystemReminderTextMsg; add a developer case so the single-message early return cannot regress forrole=developer.Suggested test expansion
func TestLoneSystemMessageReturnsUserMessage(t *testing.T) { - for _, inline := range []bool{true, false} { - input := []schemas.ResponsesMessage{systemReminderTextMsg("You are Claude Code.")} - messages, systemMessages, err := bedrock.ConvertBifrostMessagesToBedrockMessages(context.Background(), input, inline) - require.NoError(t, err) - assert.Empty(t, systemMessages, "lone system message must not populate the system block (inline=%v)", inline) - require.Len(t, messages, 1, "lone system message must yield exactly one message (inline=%v)", inline) - assert.Equal(t, bedrock.BedrockMessageRoleUser, messages[0].Role) - } + cases := []struct { + name string + msg schemas.ResponsesMessage + }{ + {name: "system", msg: systemReminderTextMsg("You are Claude Code.")}, + {name: "developer", msg: developerReminderTextMsg("Developer instructions.")}, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + for _, inline := range []bool{true, false} { + input := []schemas.ResponsesMessage{tc.msg} + messages, systemMessages, err := bedrock.ConvertBifrostMessagesToBedrockMessages(context.Background(), input, inline) + require.NoError(t, err) + assert.Empty(t, systemMessages, "lone %s message must not populate the system block (inline=%v)", tc.name, inline) + require.Len(t, messages, 1, "lone %s message must yield exactly one message (inline=%v)", tc.name, inline) + assert.Equal(t, bedrock.BedrockMessageRoleUser, messages[0].Role) + } + }) + } }As per coding guidelines, Go changes should include deterministic tests and table-driven coverage for behavior changes.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core/providers/bedrock/bedrock_test.go` around lines 6471 - 6477, The test TestLoneSystemMessageReturnsUserMessage only covers the system message role by using systemReminderTextMsg, but the converter predicate and test comment indicate both system and developer roles should be handled. Expand the test to also cover the developer message role by adding a developer message case alongside the existing system message case. Use a table-driven approach or add a separate developer message input to ensure the single-message early return path in ConvertBifrostMessagesToBedrockMessages is exercised for both role types without regression.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@core/providers/bedrock/bedrock_test.go`:
- Around line 6471-6477: The test TestLoneSystemMessageReturnsUserMessage only
covers the system message role by using systemReminderTextMsg, but the converter
predicate and test comment indicate both system and developer roles should be
handled. Expand the test to also cover the developer message role by adding a
developer message case alongside the existing system message case. Use a
table-driven approach or add a separate developer message input to ensure the
single-message early return path in ConvertBifrostMessagesToBedrockMessages is
exercised for both role types without regression.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 16db7350-4432-40ec-b7cf-7f4942b4ecba
📒 Files selected for processing (2)
core/providers/bedrock/bedrock_test.gocore/providers/bedrock/responses.go
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
|
Thanks — both addressed:
|
mickgvirtu
force-pushed
the
pr-bedrock-midconv-system-cache
branch
from
d0aa598 to
c8cc345
Compare
…cache Bedrock's prompt cache is prefix-based: a mid-conversation role=system message (e.g. the reminders Claude Code injects) hoisted into the top-level system block grows that prefix every turn and collapses the cached conversation to the tools/system floor. This is the Bedrock counterpart of the native Anthropic provider's mid-conversation system support (SupportsMidConversationSystem) — Bedrock has no message-level system role, so the inlined message is rendered as a user turn. Gated by an inlineSystemReminders bool the caller computes via IsAnthropicModelFamily(ctx, model) (alias-aware), so non-Anthropic families keep the historical hoist-everything behavior. Tool-call/result cache_control breakpoints are preserved as CachePoint blocks carrying the requested TTL. Adds regression tests including a positive cache_control->CachePoint+TTL assertion, the lone-system early return, and the no-leading-system gate. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
mickgvirtu
force-pushed
the
pr-bedrock-midconv-system-cache
branch
from
c8cc345 to
d09b2f8
Compare
|
Rebased onto |
…pic-base providers to preserve prompt cache Any CUSTOM provider (a provider key that is not one of Bifrost's built-in providers) reaching the Anthropic converter now keeps mid-conversation role:system messages inline instead of hoisting them into the leading system block. Such a provider exists only because the operator set base_provider_type to an Anthropic-compatible base and pointed it at a self-hosted engine (sglang, vLLM, TGI, llama.cpp). These engines are prefix/radix KV-cache based and render role:system inline via their chat template, so hoisting Claude Code's per-turn reminders forks the prefix cache every turn and strands the cacheable tail (tools + history). This covers GLM, Kimi (Kimi-K2 on vLLM), and any other self-hosted Anthropic-compatible model in one rule. Built-in non-Anthropic providers (Bedrock, Vertex, Azure, standard SGL/OpenAI) keep their historical behavior: their keys are standard, so they fall through to the existing model-based opus-4.8/Fable gate. This is the custom-provider sibling of the Bedrock fix in maximhq#4534. Adds schemas.IsStandardProvider (built from StandardProviders) so the gate can distinguish a custom provider key from a built-in one without an import cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…pic-base providers to preserve prompt cache Any CUSTOM provider (a provider key that is not one of Bifrost's built-in providers) reaching the Anthropic converter now keeps mid-conversation role:system messages inline instead of hoisting them into the leading system block. Such a provider exists only because the operator set base_provider_type to an Anthropic-compatible base and pointed it at a self-hosted engine (sglang, vLLM, TGI, llama.cpp). These engines are prefix/radix KV-cache based and render role:system inline via their chat template, so hoisting Claude Code's per-turn reminders forks the prefix cache every turn and strands the cacheable tail (tools + history). This covers GLM, Kimi (Kimi-K2 on vLLM), and any other self-hosted Anthropic-compatible model in one rule. Built-in non-Anthropic providers (Bedrock, Vertex, Azure, standard SGL/OpenAI) keep their historical behavior: their keys are standard, so they fall through to the existing model-based opus-4.8/Fable gate. This is the custom-provider sibling of the Bedrock fix in maximhq#4534. Adds schemas.IsStandardProvider (built from StandardProviders) so the gate can distinguish a custom provider key from a built-in one without an import cycle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Known limitation: non-Anthropic Bedrock families (Nova, imported models) still hoistFlagging the boundary of this fix. The inline behavior is gated on That leaves a real gap: Amazon Nova also prefix-caches -- This narrow gating is deliberate, not an oversight: the Bedrock inline transform is lossy -- because Converse has no message-level system role, the reminder is rewritten into a (For contrast, the Anthropic-compatible custom-provider counterpart -- #4592 / #4593 -- is a no-op transform: the wire format has a native inline Filing this as a known limitation / follow-up rather than expanding this PR's scope. |
2 participants
Summary
Fixes #4068. On the Bedrock provider,
ConvertBifrostMessagesToBedrockMessageshoists everyrole:system/role:developermessage into Bedrock's top-levelsystemblock, regardless of position. Because Bedrock's Converse prompt cache is prefix-based, arole:systemmessage injected mid-conversation (e.g. the reminders Claude Code emits) grows thesystemprefix in front of the cached conversation and collapses cache reads to the system+tools floor — recurring on every such turn.Changes
system; messages appearing after the conversation starts are inlined in place. Non-Anthropic models keep the historical hoist-everything behavior.SupportsMidConversationSystemhandling. Bedrock has no message-levelsystemrole, so an inlined message is rendered as auserturn (wrapped in<system-reminder>…</system-reminder>, matching the convention clients already use for pre-wrapped reminders).inlineSystemReminders boolcomputed by the caller viaIsAnthropicModelFamily(ctx, model)(alias-aware, consistent with the other Anthropic gates in the file).cache_controlon tool calls/results is preserved as aCachePointcarrying the requested TTL.Type of change
Affected areas
How to test
go test ./core/providers/bedrock/Adds
TestMidConversationSystemReminderStaysInline,…HoistedForNonAnthropic,TestToolCacheControlBecomesCachePointWithTTL(positive TTL assertion), a lone-system early-return test, and a no-leading-system-block gate test.Issue #4068 has the full root-cause plus a real cache-read trace (cached tokens dropping to the system/tools floor and recovering after the prefix re-warms). Related native-side work: #4276, #3879.
Notes for reviewers: this re-adds a parameter the converter previously dropped — happy to thread it differently (e.g. derive from a typed context) if you prefer. The
<system-reminder>wrapping follows the client convention; open to gating it if you'd rather it not be implicit.