Repository navigation
Description
Problem Description
In google-adk, ContextCacheConfig is intended to provide explicit context caching across agents in an application. However, inspecting the internal implementation of GeminiContextCacheManager reveals two structural limitations that prevent context caching from working on the majority of enterprise production traffic:
1. The Turn 1 "Chicken-and-Egg" Miss
In GeminiContextCacheManager._create_new_cache_with_contents:
if llm_request.cacheable_contents_token_count is None:
logger.info(
"No previous token count available, skipping cache creation for initial request"
)
return NoneBecause ADK waits for the response of the first turn to populate cacheable_contents_token_count to validate the minimum 4096-token threshold, the initial turn of every session is systematically executed without an explicit cache.
2. Session-Scoped vs App-Scoped Caching
The generated cache name is ephemeral (adk-cache-<timestamp>-<N>contents) and bound to the session state. For thousands of distinct users interacting with the exact same agent (having identical system instructions and tools), ADK attempts to create thousands of separate Vertex AI caches rather than reusing a shared, application-level cache.
Real-World Production Impact (High-Volume Enterprise Agents)
In large-scale production customer-facing agents on Vertex AI with a large prompt footprint (~30,000 tokens for comprehensive business guidelines + OpenAPI/Function tool schemas):
- User Session Distribution: A substantial majority of user sessions (>60%) are single-turn (the user asks a targeted question, performs an action, and exits).
- Gemini 2.5 vs Gemini 3.5 / 3.x behavior:
- On
gemini-2.5-flash, Google's implicit caching mitigated this by caching the long prefix behind the scenes. - On
gemini-3.5-flash-lite, implicit caching requires strict prefix alignment. Consequently, in staging and production tests, Turn 1 experiences a 0% cache hit, while Turn 2+ of the same session jumps to an 85–87% cache hit once ADK creates the session cache. - As a result, more than 60% of enterprise traffic pays full input token pricing for ~30k tokens that are 100% static across all users.
- On
Proposed Solution / Architectural Suggestion
Introduce an App-Scoped / Static Context Cache pattern in ADK:
- Pre-caching / Warmup Hook:
Allow anApporAgentto declare a static invariant prefix (e.g.,static_instruction+tools). At application startup (e.g. FastAPI lifespan or worker initialization), compute tokens viaclient.aio.models.count_tokensor allow pre-creating a shared Vertex AI cache (client.aio.caches.create) with an application TTL. - Immediate Turn-1 Cache Injection:
If an application-level cache reference is available,GeminiContextCacheManagershould attachcached_contentimmediately on Turn 1 instead of returningNonewhencacheable_contents_token_count is None. - Session Cache on Top of App Cache (Optional Tiering):
The application cache covers the invariant instructions + tools, while conversation history continues to be handled either without caching or via session delta.
Willingness to Contribute
We have already prototyped this pattern in production sidecars by creating an explicit Vertex AI cache during the application lifespan and passing the cache reference to Gemini requests.
If the maintainers are aligned with this RFC/enhancement, we would be very happy to prepare and submit a Pull Request!
Activity
feat(caching): support static application-scoped context cache on ini…
llalitkumarrr commentedon Oct 16, 2026
Hello @hugueslys,
Thank you for raising this request and for your contribution. We will review it and reach out to you if we require further information. In the meantime we appreciate your patience.
feat: attach agent cached_content seed on initial turn