Skip to content

Context cache size gate ignores function call/response parts, so caches.create still fails with 400 INVALID_ARGUMENT in tool loops #8225

Description

@karthik-0306

🔴 Required Information

Describe the Bug:
GeminiContextCacheManager checks the size of the cacheable prefix before it calls caches.create, so that it never sends a cache below the model's minimum. That check was added in #6137 to fix #5847. It works by scaling the real token count of the previous prompt (cacheable_contents_token_count) by the prefix's share of an estimate. The estimate comes from _estimate_request_tokens, which only counts part.text:

for content in contents:
  for part in content.parts or []:
    if part.text:
      total_chars += len(part.text)

function_call, function_response and inline_data parts count as zero. In an agent that uses tools, these are often most of the prompt, so the share comes out wrong in both directions:

  1. Overestimate, so the 400 is back. In a tool loop, the newest content is usually the tool's function_response (role user). It falls outside the cached prefix but is invisible to the estimate. If it's large, the prefix's share comes out close to 1, the gate passes a prefix far below the minimum, and caches.create fails with 400 INVALID_ARGUMENT. This is the failure fix(models): gate Gemini cache creation on cacheable prefix tokens #6137 was meant to prevent.
  2. Underestimate, so caching never turns on. When large tool results sit inside the prefix and the uncached tail is text, the prefix's share is underestimated. A prefix that is above the minimum gets rejected, and agents that use tools heavily get no caching.

Steps to Reproduce:

  1. pip install google-adk
  2. Run this script. It calls the manager directly with a mocked client, so it needs no API key:
import asyncio
from unittest import mock
from google.genai import types
from google.adk.agents.context_cache_config import ContextCacheConfig
from google.adk.models.cache_metadata import CacheMetadata
from google.adk.models.gemini_context_cache_manager import GeminiContextCacheManager
from google.adk.models.llm_request import LlmRequest

big = "lorem ipsum " * 10000  # ~120k chars of tool output
contents = [
    types.Content(role="user", parts=[types.Part(text="Summarize the doc at x")]),
    types.Content(role="model", parts=[types.Part(function_call=types.FunctionCall(name="fetch_doc", args={"url": "x"}))]),
    types.Content(role="user", parts=[types.Part(function_response=types.FunctionResponse(name="fetch_doc", response={"result": big}))]),
]
req = LlmRequest(
    model="gemini-3-flash",
    contents=contents,
    config=types.GenerateContentConfig(system_instruction="You are a helpful agent."),
    cache_config=ContextCacheConfig(min_tokens=0),
)
req.cacheable_contents_token_count = 30_050  # real count reported for the previous prompt

client = mock.MagicMock()
client.vertexai = False
client.aio.caches.create = mock.AsyncMock(side_effect=Exception("400 INVALID_ARGUMENT"))
mgr = GeminiContextCacheManager(client)

n = mgr._find_count_of_contents_to_cache(contents)  # 2: user text + function_call
print("estimated prefix tokens:", mgr._estimate_cacheable_prefix_tokens(req, n))
req.cache_metadata = CacheMetadata(fingerprint=mgr._generate_cache_fingerprint(req, n), contents_count=n)
asyncio.run(mgr.handle_context_caching(req))
print("caches.create calls:", client.aio.caches.create.await_count)

Expected Behavior:
The cacheable prefix (a short user message and one function call, about 10 real tokens) is estimated well below the 4,096-token minimum for Gemini 3, so caches.create is never called.

Observed Behavior:

Failed to create cache: 400 INVALID_ARGUMENT
estimated prefix tokens: 30050
caches.create calls: 1

The whole 120k-character tool result is credited to the prefix, so the gate passes and the API rejects the cache.

Environment Details:

  • ADK Library Version: 2.14.1 (main @ c74df83)
  • Desktop OS: Windows 11 (the bug isn't OS-specific)
  • Python Version: 3.11

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: gemini-3-flash (any model that _minimum_cache_tokens covers: gemini-2.5-, gemini-3)

🟡 Optional Information

Regression:
No. The gate has used a text-only estimate since it was introduced in #6137.

Proposed fix:
Count the payloads of non-text parts in _estimate_request_tokens: the JSON size of function_call.args and function_response.response (plus the function name), and a rough size for inline_data and file_data. Then the prefix's share reflects what is actually sent. Add regression tests for both directions: a large function_response in the uncached tail must not pass the gate, and a large one in the prefix must not block it. I'm happy to send a PR.

Activity

  1. surajksharma07 commented on Nov 19, 2026

    @surajksharma07
    Collaborator

    @karthik-0306. Reproduced both directions on 2.14.1 and 1.27.2: a ~10-token prefix passes the gate as 30050, and a prefix holding the big tool result gets estimated at ~2k and is skipped.

    As a stopgap until #8234 lands, patch GeminiContextCacheManager._estimate_request_tokens to also count function_call args and function_response payloads (json.dumps with default=str), plus a rough size for inline_data and file_data. That fixed both cases on my side.

    Since the repro mocks the client, could you try it once in a real tool loop with caching on? Please confirm the 400s stop and a cache actually gets created when the prefix is large enough.

    If it holds up we can move #8234 forward. It would also help to have regression tests for both directions there, and maybe cover executable_code / code_execution_result parts too.

  2. added a commit that references this issue on Nov 20, 2026
    137c9fa
  3. karthik-0306 commented on Nov 20, 2026

    @karthik-0306
    Author

    Thanks @surajksharma07 for reproducing both directions on 2.14.1 and 1.27.2! I've updated #8234 with your suggestions. Point by point:

    1. Media and tool payloads in the estimate.
    The estimate counts function call and function response names plus their JSON payloads (json.dumps(..., default=str), as you suggested). A follow-up commit now also gives inline_data / file_data a rough size: one small image, 258 tokens, per item, including media a tool returns in FunctionResponse.parts. Media token cost depends on resolution or duration rather than byte length, so this is deliberately a floor rather than a byte-based guess. It keeps a large trailing image or document from making a small prefix look like the whole prompt.

    2. Regression tests for both directions.

    • test_large_trailing_tool_result_does_not_pass_cache_size_gate: a tiny prefix followed by a large uncached payload. No cache is created.
    • test_large_cached_tool_result_passes_cache_size_gate: a large tool result inside the prefix. The cache is created.

    3. executable_code / code_execution_result.
    Both are counted. The trailing-payload test is parametrized over a function response, executable_code, code_execution_result, media, and tool-returned media. All 6 regression cases fail on main and pass with the PR. The whole test file passes (67 tests) when merged onto the current main.

    4. Real tool loop with caching on.
    gemini-3-flash-preview, ContextCacheConfig(min_tokens=0), a tool returning a ~25.8k-token document, 4 turns, with a 7k-character pasted message on turn 2:

    main this PR
    Turn 2, prefix holding the tool result estimated at 3,519 tokens (< 4,096), cache creation skipped passes the size gate, caches.create is called
    400 INVALID_ARGUMENT 0 0

    So the missed-caching direction reproduces against the live API, and the PR fixes it.

    Two parts I couldn't confirm myself:

    • A cache actually being created. My key is free tier, and the API rejects every explicit cache with 429 TotalCachedContentStorageTokensPerModelFreeTier ... limit=0, on both main and the PR. That step needs a paid key or Vertex AI. I'd be grateful if you could check it on your side.
    • The tiny-prefix 400 in a live single-agent loop. The manager caches max(previous, current) contents, so by the time the previous prompt is large, the prefix already holds the tool result, and I couldn't trigger it with a real model. That direction is covered by the unit test above, which builds the request state from your repro.
  4. surajksharma07 commented on Nov 20, 2026

    @surajksharma07
    Collaborator

    Thanks @karthik-0306, the live run is useful. Merged 137c9fa onto today’s main (c81876a). It’s 27 commits behind but auto-merges, including 43e959e, which now skips cache creation for models that refuse explicit caching in the same function. Main still has the bug, so 2.15.0 doesn’t fix it either: the tiny prefix is estimated at 30050 and caches.create is called, and the prefix holding the tool result comes out at ~1977 and is skipped. With the PR they’re 16 (no create) and ~30168 (create called). The 6 new cases fail on main and pass with the PR, all cache tests pass (79), and models/agents/flows give 4399 passed. The 13 failures there fail the same way on plain main (older google-genai on my side).

    Agree on the tiny-prefix 400: in a plain tool loop the previous prompt doesn’t contain the trailing tool result yet, so missed caching is the direction people actually hit, and the unit test is enough for the other one. One small thing: the PR description still says media isn’t counted, could you update it for the 258-token floor?

    @xuanyang15 could you review #8234 and approve the 2 pending workflows? If anyone on your side can run it once on Vertex, that would close the last gap.

  5. karthik-0306 commented on Nov 20, 2026

    @karthik-0306
    Author

    Thanks @surajksharma07, I've updated the #8234 description: it now covers the 258-token media floor, the parametrized tests and the live-run results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions