fix: Keep the Prompt Prefix Stable Across Turns and Conversations

[Shabonasar]

Summary

Provider prompt caches reuse a request only up to its first changed byte. Since #15419 the system prompt carries a per-turn timestamp, and with Anthropic or OpenRouter prompt caching the dynamic system tail is moved before the last user message and shifts every turn, so long conversations and many conversations of one agent lose most of the cache (#16877). This PR adds endpoints.agents.promptCache.stableContext (default false, which keeps today's behaviour). With it on, prompt time is anchored to the conversation start, a later date is announced by a <context_update> block at the start of the first user message sent after it, static instructions precede per-chat context, and the per-turn parts of the tail stay out of the stable instructions.

Related to #17102, #16877, #12907, #15419, #16563. Base: dev. First PR of the stack in #17102.

How it works

dev:    [tools][S + time(turn)] u1 a1 [tail] u2           time changes every turn; tail moves every turn (Anthropic/OpenRouter caching)
lever:  [tools][S + time(conversation start)] u1 a1 [<context_update> date changed] u2 a2 u3
        S = static instructions first, then per-chat context; memory, RAG and queued files stay in the per-turn tail

• {{current_datetime}} and friends, and the web search runtime context, resolve against the conversation start (conversationCreatedAt, minute precision, user timezone). The client no longer resolves promptPrefix with the browser clock for Agents routes.
• The date block is derived from stored data (the message createdAt and the conversation anchor), so it renders identically on every later turn. After a compaction the comparison restarts from the conversation start date.
• The user message is stored with createdAt = turnStartedAt. endpoints.agents.promptMessageTimestamps (default false) adds the send time to each user message.
• The history notes, the instruction split and the lever live in packages/api (contextUpdates.ts, instructionPlan.ts, promptCache.ts); the controller only calls them.

Measurements

G1: plain chat, 10 turns, timestamp changing	dev	lever off	lever on	uncached prompt tokens, dev -> lever on
OpenAI chat format (LiteLLM endpoint)	0.0%	0.0%	56.5%	84240 -> 31966
Anthropic Messages format (LiteLLM endpoint, cache_control ignored by the backend)	0.0%	0.0%	54.9%	63275 -> 34217

Run: group G1 measured on Qwen3.8-27B (vLLM behind NVIDIA Dynamo, through LiteLLM), OpenAI chat and Anthropic Messages, dev / lever off / lever on; the same self-hosted model answers both formats, the real OpenAI and Anthropic APIs were not measured; full method in #17102. This PR is the part of the stack that moves G1.

Note: the backend caches in 1600-token blocks, so a request reuses cache only in whole blocks and the first turns of a short conversation (prompt under two blocks) cannot hit; shares are therefore lower on short chats than on long ones.

Verification

npm run lighthouse (3 cold navigations per run, Mongo delay 250 ms per query), dev against the head of the stack with the lever off and on; ranges over the valid runs.

	dev	lever off	lever on
LCP, ms (budget 4500)	3963-4139	3382-4197	4153-4188
CLS (budget 0.1)	0.017-0.018	0.016-0.018	0.017-0.018
TBT, ms (budget 500)	252-316	322-333	289-328
Mongo queries per run	492	493	494

All results are within the CI budgets and the differences are within run-to-run noise; the lever only adds +contextAnchor to an existing select, so message loading does the same queries in all three configurations.

Type of change

• Bug fix

Testing

Enable the lever in librechat.yaml, run a multi-turn Agents conversation across a pause, and compare the cached tokens of consecutive requests:

endpoints:
  agents:
    promptCache:
      stableContext: true

Automated tests: the test files this PR adds or changes, run per workspace with npx jest <files> (mongod-backed suites against a real mongod): packages/api 594, packages/data-provider 191, api 658 tests passing. Files: BaseClient.test.js, request.resumeMetadata.spec.js, client.test.js, initialize.test.ts, promptPrefixStability.test.ts, run-summarization.test.ts, contextUpdates.boundary.spec.ts, contextUpdates.spec.ts, instructionPlan.spec.ts, run.spec.ts, web.spec.ts, config-schemas.spec.ts.

Static checks: tsc --noEmit in packages/api, packages/data-provider; eslint and prettier on the touched files.

Screenshots / recordings

No user-facing change.

Risk / compatibility

Default off reproduces today's requests. A user who switches timezones mid-conversation re-renders earlier blocks once (one cache miss).

Checklist

• I reviewed my own changes
• Relevant tests have been added or updated
• Existing relevant tests pass
• The change does not introduce new warnings or errors
• User-facing or complex behavior is documented where necessary
• Required dependency changes have been merged/published (none)
• Required documentation PR: to LibreChat-AI/librechat.ai for endpoints.agents.promptCache.stableContext and promptMessageTimestamps, after the review settles the setting names

[lia-by-librechat]

Updated head 0c0ecd2344d571200d3af4608e8a7200f944cda6: fixes the failing run-boundary assertion, keeps artifact and always-apply subagent skill instructions ahead of per-turn context, and moves templated date/time agent instructions into the dynamic runtime tail. Added regressions for prefix reuse across timestamps, initialization ordering and isolated subagents. Local checks passed on this head: 400/400 tests across 3 affected suites, packages/api tsc --noEmit, npm run sort-imports (touched files), npm run static-checks -- --against origin/dev, and packages/api tsdown. GitHub Actions has scheduled workflows but marks all nine action_required for this fork update; no CI checks have run on this head yet.

[Shabonasar]

Updated head e05c694: merged current dev (one trivial conflict with #16686 in initialize.ts) and extended the fix after #16877.

Reordering alone was not enough. Prefix caches still broke because the system message carried a per-turn timestamp (#15419), the client resolved promptPrefix with the browser clock, and the stored user-message time did not match the time used in the prompt. This update anchors the system prompt to the conversation start (as #12907 did). Later date changes are announced lazily as a <context_update> block in the user message that crosses midnight. That keeps the guarantee from #15419 without editing the system prompt. Details, the provider payload stability tests and a live measurement are in the description.

Live measurement on Qwen3.8-27B (vLLM behind NVIDIA Dynamo, through a LiteLLM proxy), 5-turn conversation growing to 29k tokens: cached prompt tokens on turn 5 went from 0 on dev to 19,200 of 29,125, the maximum this hybrid model's 1600-token block cache allows. The turn after the date change was included.

Thanks @npdgm for the precise report and measurements in #16877, @dev-fbm for the Bedrock analysis, and @lia-by-librechat for the ordering and subagent commit kept here.

Next, as a continuation: deterministic tool order (MCP servers and tools sorted by name), then delivering mid-conversation toggles and dynamically loaded MCP tools as history events, so dynamic loading never resets the cached prefix.

[Shabonasar]

Follow-up for the tools parameter: #17097 sorts MCP servers and their tools by name, so selecting the same servers in a different order, or a catalog returning tools in a different order, no longer changes tools (serialized ahead of the system prompt, so it reset the cache from position 0). Next in the series: mid-conversation toggles as <context_update> history events, then deferred loading for picker-selected MCP servers with a fixed tools array.

[dlbogdan]

Adding an independent Strata data point and a Docker Compose preload workaround for LibreChat v0.8.8 installations affected by #16877. I am running the date-only hook below. Keeping it in a host-mounted file avoids reapplying an edit inside the container after each image replacement.

I checked the current PR head, e05c694, and #17102. This PR already addresses conversation-start anchoring, static-before-dynamic ordering, and date-change events in history. The results below are for stock v0.8.8 versus the hook, not a benchmark of this PR.

Measured results

LibreChat v0.8.8, local prefix-caching engine Strata, Qwen3.8-Flash-Next, agents on /v1/responses, one conversation; each row is the first model call after a new user message:

Build	Prompt tokens	Reused	Read in	Prefill
Stock	93,780	8,192	85,588	37.5 s
Stock	112,067	8,192	103,875	44.3 s
Hook	97,715	81,920	15,795	7.8 s
Hook	98,031	90,112	7,919	4.4 s
Hook	99,137	90,112	9,025	5.7 s

Requests captured on the wire between LibreChat and the engine showed a 33,662-character system message that was byte-identical, with the same SHA-256, across consecutive user turns with the hook. The hook changes only the timestamp text; other request content is untouched.

These are operational measurements at different prompt lengths, not a controlled identical-request benchmark. In these runs, reuse increased from 8,192 to 81,920–90,112 tokens and prefill dropped from 37.5–44.3 seconds to 4.4–7.8 seconds.

The working hook

In v0.8.8, ToolService.js and handleTools.js destructure buildWebSearchDynamicContext from @librechat/api during module initialization. Preloading this CommonJS file wraps the export before those consumers capture it. The original function still produces the context; the wrapper trims its ISO timestamp to the date.

Save as ./patches/websearch-date-stable.cjs:

// websearch-date-stable.cjs — interim workaround for #16877
try {
  const api = require('@librechat/api');
  const orig = api.buildWebSearchDynamicContext;
  const RE = /(Date(?: & Time)?:\s*)(\d{4}-\d{2}-\d{2})T\d{2}:\d{2}:\d{2}[^\s]*/;
  if (typeof orig !== 'function') {
    console.warn('[patch] buildWebSearchDynamicContext not found, skipping');
  } else if (orig.__dateStable !== true) {
    let warned = false;
    const wrapped = function (now) {
      const out = orig.call(this, now);
      if (typeof out !== 'string') return out;
      const fixed = out.replace(RE, (m, label, date) => label + date);
      if (fixed === out && /\d{2}:\d{2}:\d{2}/.test(out) && !warned) {
        warned = true;
        console.warn('[patch] volatile time-of-day text present but unmatched: upstream format changed');
      }
      return fixed;
    };
    wrapped.__dateStable = true;
    api.buildWebSearchDynamicContext = wrapped;
    console.warn('[patch] web_search runtime context trimmed to date-only');
  }
} catch (err) {
  console.warn('[patch] did not apply:', err && err.message);
}

Docker Compose deployment

Merge these entries under the LibreChat service in docker-compose.override.yaml:

    environment:
      - NODE_OPTIONS=--require /app/patches/websearch-date-stable.cjs
    volumes:
      - type: bind
        source: ./patches
        target: /app/patches
        read_only: true

The hook's log line can appear once per Node process (npm, cross-env, node), which is expected.

No file in the image is modified. The host file and Compose override persist across image replacements and are picked up when the service is recreated with docker compose up -d, provided the relevant exports and initialization behavior remain compatible. This also makes the workaround a convenient Compose-level toggle for measuring the timestamp's cache cost on an existing deployment.

Scope and caveats

• The model receives the date without the time of day from this context; anything requiring the clock needs a tool. The date still changes daily, invalidating the cached prefix from that point onward.
• Date-only context can make the complete system message identical across conversations of the same agent on a given day if all other system content is unchanged. The measurements above verify consecutive turns within one conversation. The PR instead preserves a shared static prefix plus conversation-specific temporal context, and handles later date changes in history. These are different reuse strategies, and the PR already makes that distinction.
• The regex depends on the emitted format. A space-separated timestamp or a different label such as Timestamp: will stop matching. The warning catches unmatched output containing HH:MM:SS, but it is not a complete compatibility check and some format changes can go unnoticed.
• When the regex does not match, the returned text passes through unchanged; the wrapper remains installed. The outer try/catch handles installation failures, not errors from later calls to the wrapped function.
• Persistence across image updates does not guarantee compatibility with future code. In particular, the current PR's helper changes the label and accepts a second timezone argument, while this exact v0.8.8 wrapper forwards only now. Remove the preload when adopting the upstream fix rather than assuming a nonmatching regex makes the installed wrapper harmless.

This is a temporary mitigation for the volatile web-search timestamp, complementing the date-only local patch reported in #16877. It does not fix other dynamic instructions, tool-schema changes, or the broader causes of prefix instability tracked in #17102.

[Shabonasar]

Updated the branch:

• Merged the current dev (no conflicts); head 4c59edb.
• One commit on top of the previous head: the per-turn parts of the dynamic tail stay structured (no string matching), additional_instructions keep the dev order, and the lever, history notes and instruction split moved out of client.js into packages/api.
• Description rewritten to the short form: idea, request layout, the G1 measurement. The other groups and the method are in Tracking: stable prompt prefix cache across turns, conversations and providers #17102.

@dlbogdan thank you for the independent numbers in #16369 (comment): 8,192 -> 81,920-90,112 reused tokens and 37-44 s -> 4-8 s prefill with only the web search time trimmed confirms that the timestamp alone costs most of the prefix on long conversations.

One note for anyone using that preload: drop it when moving to a build with this PR. Here buildWebSearchDynamicContext takes an options object ({ turnStartedAt, conversationStartedAt, timezone, stableContext }) instead of now. With endpoints.agents.promptCache.stableContext: true the context reads Conversation started: <conversation start, minute precision, user timezone>, which is already stable across turns, so the hook's pattern no longer matches and is not needed; with the lever off the text is the same as on dev.