Repository navigation
Description
🔴 Required Information
Describe the Bug:
GeminiContextCacheManager checks the size of the cacheable prefix before it calls caches.create, so that it never sends a cache below the model's minimum. That check was added in #6137 to fix #5847. It works by scaling the real token count of the previous prompt (cacheable_contents_token_count) by the prefix's share of an estimate. The estimate comes from _estimate_request_tokens, which only counts part.text:
for content in contents:
for part in content.parts or []:
if part.text:
total_chars += len(part.text)function_call, function_response and inline_data parts count as zero. In an agent that uses tools, these are often most of the prompt, so the share comes out wrong in both directions:
- Overestimate, so the 400 is back. In a tool loop, the newest content is usually the tool's
function_response(roleuser). It falls outside the cached prefix but is invisible to the estimate. If it's large, the prefix's share comes out close to 1, the gate passes a prefix far below the minimum, andcaches.createfails with400 INVALID_ARGUMENT. This is the failure fix(models): gate Gemini cache creation on cacheable prefix tokens #6137 was meant to prevent. - Underestimate, so caching never turns on. When large tool results sit inside the prefix and the uncached tail is text, the prefix's share is underestimated. A prefix that is above the minimum gets rejected, and agents that use tools heavily get no caching.
Steps to Reproduce:
pip install google-adk- Run this script. It calls the manager directly with a mocked client, so it needs no API key:
import asyncio
from unittest import mock
from google.genai import types
from google.adk.agents.context_cache_config import ContextCacheConfig
from google.adk.models.cache_metadata import CacheMetadata
from google.adk.models.gemini_context_cache_manager import GeminiContextCacheManager
from google.adk.models.llm_request import LlmRequest
big = "lorem ipsum " * 10000 # ~120k chars of tool output
contents = [
types.Content(role="user", parts=[types.Part(text="Summarize the doc at x")]),
types.Content(role="model", parts=[types.Part(function_call=types.FunctionCall(name="fetch_doc", args={"url": "x"}))]),
types.Content(role="user", parts=[types.Part(function_response=types.FunctionResponse(name="fetch_doc", response={"result": big}))]),
]
req = LlmRequest(
model="gemini-3-flash",
contents=contents,
config=types.GenerateContentConfig(system_instruction="You are a helpful agent."),
cache_config=ContextCacheConfig(min_tokens=0),
)
req.cacheable_contents_token_count = 30_050 # real count reported for the previous prompt
client = mock.MagicMock()
client.vertexai = False
client.aio.caches.create = mock.AsyncMock(side_effect=Exception("400 INVALID_ARGUMENT"))
mgr = GeminiContextCacheManager(client)
n = mgr._find_count_of_contents_to_cache(contents) # 2: user text + function_call
print("estimated prefix tokens:", mgr._estimate_cacheable_prefix_tokens(req, n))
req.cache_metadata = CacheMetadata(fingerprint=mgr._generate_cache_fingerprint(req, n), contents_count=n)
asyncio.run(mgr.handle_context_caching(req))
print("caches.create calls:", client.aio.caches.create.await_count)Expected Behavior:
The cacheable prefix (a short user message and one function call, about 10 real tokens) is estimated well below the 4,096-token minimum for Gemini 3, so caches.create is never called.
Observed Behavior:
Failed to create cache: 400 INVALID_ARGUMENT
estimated prefix tokens: 30050
caches.create calls: 1
The whole 120k-character tool result is credited to the prefix, so the gate passes and the API rejects the cache.
Environment Details:
- ADK Library Version: 2.14.1 (main @ c74df83)
- Desktop OS: Windows 11 (the bug isn't OS-specific)
- Python Version: 3.11
Model Information:
- Are you using LiteLLM: No
- Which model is being used: gemini-3-flash (any model that
_minimum_cache_tokenscovers: gemini-2.5-, gemini-3)
🟡 Optional Information
Regression:
No. The gate has used a text-only estimate since it was introduced in #6137.
Proposed fix:
Count the payloads of non-text parts in _estimate_request_tokens: the JSON size of function_call.args and function_response.response (plus the function name), and a rough size for inline_data and file_data. Then the prefix's share reflects what is actually sent. Add regression tests for both directions: a large function_response in the uncached tail must not pass the gate, and a large one in the prefix must not block it. I'm happy to send a PR.
Activity
surajksharma07 commentedon Nov 19, 2026
@karthik-0306. Reproduced both directions on 2.14.1 and 1.27.2: a ~10-token prefix passes the gate as 30050, and a prefix holding the big tool result gets estimated at ~2k and is skipped.
As a stopgap until #8234 lands, patch GeminiContextCacheManager._estimate_request_tokens to also count function_call args and function_response payloads (json.dumps with default=str), plus a rough size for inline_data and file_data. That fixed both cases on my side.
Since the repro mocks the client, could you try it once in a real tool loop with caching on? Please confirm the 400s stop and a cache actually gets created when the prefix is large enough.
If it holds up we can move #8234 forward. It would also help to have regression tests for both directions there, and maybe cover executable_code / code_execution_result parts too.
fix(models): give media a rough size when sizing a context cache
karthik-0306 commentedon Nov 20, 2026
Thanks @surajksharma07 for reproducing both directions on 2.14.1 and 1.27.2! I've updated #8234 with your suggestions. Point by point:
1. Media and tool payloads in the estimate.
The estimate counts function call and function response names plus their JSON payloads (
json.dumps(..., default=str), as you suggested). A follow-up commit now also givesinline_data/file_dataa rough size: one small image, 258 tokens, per item, including media a tool returns inFunctionResponse.parts. Media token cost depends on resolution or duration rather than byte length, so this is deliberately a floor rather than a byte-based guess. It keeps a large trailing image or document from making a small prefix look like the whole prompt.2. Regression tests for both directions.
test_large_trailing_tool_result_does_not_pass_cache_size_gate: a tiny prefix followed by a large uncached payload. No cache is created.test_large_cached_tool_result_passes_cache_size_gate: a large tool result inside the prefix. The cache is created.3.
executable_code/code_execution_result.Both are counted. The trailing-payload test is parametrized over a function response,
executable_code,code_execution_result, media, and tool-returned media. All 6 regression cases fail onmainand pass with the PR. The whole test file passes (67 tests) when merged onto the currentmain.4. Real tool loop with caching on.
gemini-3-flash-preview,ContextCacheConfig(min_tokens=0), a tool returning a ~25.8k-token document, 4 turns, with a 7k-character pasted message on turn 2:maincaches.createis called400 INVALID_ARGUMENTSo the missed-caching direction reproduces against the live API, and the PR fixes it.
Two parts I couldn't confirm myself:
429 TotalCachedContentStorageTokensPerModelFreeTier ... limit=0, on bothmainand the PR. That step needs a paid key or Vertex AI. I'd be grateful if you could check it on your side.max(previous, current)contents, so by the time the previous prompt is large, the prefix already holds the tool result, and I couldn't trigger it with a real model. That direction is covered by the unit test above, which builds the request state from your repro.surajksharma07 commentedon Nov 20, 2026
Thanks @karthik-0306, the live run is useful. Merged 137c9fa onto today’s main (c81876a). It’s 27 commits behind but auto-merges, including 43e959e, which now skips cache creation for models that refuse explicit caching in the same function. Main still has the bug, so 2.15.0 doesn’t fix it either: the tiny prefix is estimated at 30050 and caches.create is called, and the prefix holding the tool result comes out at ~1977 and is skipped. With the PR they’re 16 (no create) and ~30168 (create called). The 6 new cases fail on main and pass with the PR, all cache tests pass (79), and models/agents/flows give 4399 passed. The 13 failures there fail the same way on plain main (older google-genai on my side).
Agree on the tiny-prefix 400: in a plain tool loop the previous prompt doesn’t contain the trailing tool result yet, so missed caching is the direction people actually hit, and the unit test is enough for the other one. One small thing: the PR description still says media isn’t counted, could you update it for the 258-token floor?
@xuanyang15 could you review #8234 and approve the 2 pending workflows? If anyone on your side can run it once on Vertex, that would close the last gap.
karthik-0306 commentedon Nov 20, 2026
Thanks @surajksharma07, I've updated the #8234 description: it now covers the 258-token media floor, the parametrized tests and the live-run results.