Skip to content

Support App-scoped / Static Context Caching for Enterprise Agents (Solving the Turn-1 Cache Miss) #7536

Closed

Description

@hugueslys

Problem Description

In google-adk, ContextCacheConfig is intended to provide explicit context caching across agents in an application. However, inspecting the internal implementation of GeminiContextCacheManager reveals two structural limitations that prevent context caching from working on the majority of enterprise production traffic:

1. The Turn 1 "Chicken-and-Egg" Miss

In GeminiContextCacheManager._create_new_cache_with_contents:

if llm_request.cacheable_contents_token_count is None:
    logger.info(
        "No previous token count available, skipping cache creation for initial request"
    )
    return None

Because ADK waits for the response of the first turn to populate cacheable_contents_token_count to validate the minimum 4096-token threshold, the initial turn of every session is systematically executed without an explicit cache.

2. Session-Scoped vs App-Scoped Caching

The generated cache name is ephemeral (adk-cache-<timestamp>-<N>contents) and bound to the session state. For thousands of distinct users interacting with the exact same agent (having identical system instructions and tools), ADK attempts to create thousands of separate Vertex AI caches rather than reusing a shared, application-level cache.


Real-World Production Impact (High-Volume Enterprise Agents)

In large-scale production customer-facing agents on Vertex AI with a large prompt footprint (~30,000 tokens for comprehensive business guidelines + OpenAPI/Function tool schemas):

  • User Session Distribution: A substantial majority of user sessions (>60%) are single-turn (the user asks a targeted question, performs an action, and exits).
  • Gemini 2.5 vs Gemini 3.5 / 3.x behavior:
    • On gemini-2.5-flash, Google's implicit caching mitigated this by caching the long prefix behind the scenes.
    • On gemini-3.5-flash-lite, implicit caching requires strict prefix alignment. Consequently, in staging and production tests, Turn 1 experiences a 0% cache hit, while Turn 2+ of the same session jumps to an 85–87% cache hit once ADK creates the session cache.
    • As a result, more than 60% of enterprise traffic pays full input token pricing for ~30k tokens that are 100% static across all users.

Proposed Solution / Architectural Suggestion

Introduce an App-Scoped / Static Context Cache pattern in ADK:

  1. Pre-caching / Warmup Hook:
    Allow an App or Agent to declare a static invariant prefix (e.g., static_instruction + tools). At application startup (e.g. FastAPI lifespan or worker initialization), compute tokens via client.aio.models.count_tokens or allow pre-creating a shared Vertex AI cache (client.aio.caches.create) with an application TTL.
  2. Immediate Turn-1 Cache Injection:
    If an application-level cache reference is available, GeminiContextCacheManager should attach cached_content immediately on Turn 1 instead of returning None when cacheable_contents_token_count is None.
  3. Session Cache on Top of App Cache (Optional Tiering):
    The application cache covers the invariant instructions + tools, while conversation history continues to be handled either without caching or via session delta.

Willingness to Contribute

We have already prototyped this pattern in production sidecars by creating an explicit Vertex AI cache during the application lifespan and passing the cache reference to Gemini requests.

If the maintainers are aligned with this RFC/enhancement, we would be very happy to prepare and submit a Pull Request!

Activity

  1. added
    core[Component] This issue is related to the core interface and implementation
    on Oct 15, 2026
  2. changed the issue type fromtoon Oct 16, 2026
  3. llalitkumarrr commented on Oct 16, 2026

    @llalitkumarrr
    Collaborator

    Hello @hugueslys,

    Thank you for raising this request and for your contribution. We will review it and reach out to you if we require further information. In the meantime we appreciate your patience.

  4. added
    needs review[Status] The PR/issue is awaiting review from the maintainer
    on Oct 16, 2026
  5. added a commit that references this issue on Nov 17, 2026
    3256db0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

core[Component] This issue is related to the core interface and implementationneeds review[Status] The PR/issue is awaiting review from the maintainer

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions