You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Telemetry drops the inference span of a sub-agent that inherits a non-Gemini model #7892
Describe the Bug:
A sub-agent with no model of its own, under a non-Gemini root (for example LiteLlm("openai/gpt-4o")), loses its generate_content span and its inference metrics when the google-genai instrumentor is active.
tracing._agent_model_name resolves an agent's model as agent.model if agent.model != "" else agent._default_model. It ignores the model the agent inherits from its nearest LlmAgent ancestor, which is what LlmAgent.canonical_model (and so llm_request.model) actually runs. The child is therefore judged by the process default (gemini-3.5-flash). _should_emit_native_telemetry then defers the call to the google-genai instrumentor, which never sees a LiteLLM call. The inference token metrics are gated on the same name.
The opposite case fails too. A sub-agent that inherits a Gemini model while set_default_model names a non-Gemini model is recorded by ADK instead of the instrumentor, so its span comes from a different recorder than its Gemini root's. (An earlier version of this issue said "recorded twice". ADK's maybe_without_genai_instrumentation peels the instrumentor off a call that ADK records, so the span isn't duplicated.)
Steps to Reproduce:
Turn telemetry on with GoogleGenAiSdkInstrumentor().instrument() and OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental.
Run root = LlmAgent(name="root", model=LiteLlm("openai/gpt-4o"), sub_agents=[LlmAgent(name="child", instruction="answer")]), and have root transfer to child.
Expected Behavior:
Two model calls produce two generate_content openai/gpt-4o spans (root and child), with duration and token metrics for both.
@adityasingh2400 Reproduced both directions on 2.13.0 and current main: the child under an openai/gpt-4o root loses its generate_content span, and a child inheriting Gemini while the default model is non-Gemini gets recorded by ADK as well as the instrumentor.
Walking up parent_agent for the model name in _agent_model_name, like #7893 does, fixed both cases for me with the default model still as the fallback. The telemetry tests I could run locally showed no new failures.
Could you check that on your side and add a test for the duplicate case too? It might also be worth trying llm_request.model first at those call sites since that’s the model actually sent. Please test it well with the instrumentor on before #7893 goes for review.
Thanks, @surajksharma07. I added test_generate_content_span_with_inherited_gemini_model to #7893 (40a79f0). It fails on main and passes with the fix.
I also ran the Gemini case end to end with the real instrumentor (0.7b2), answering Gemini's HTTP call with canned responses. That showed I'd overstated it here: the child isn't recorded twice. When ADK records a call itself, maybe_without_genai_instrumentation takes the instrumentor off it, so on main the child's span came from ADK while its Gemini root's came from the instrumentor. With the fix, both come from the instrumentor. I've corrected the issue text and the PR description.
On llm_request.model: basic.py sets it from agent.canonical_model.model, so with the ancestor walk it matches _agent_model_name for every request the flow builds. The two differ only if a before_model_callback renames the model. Switching would also bypass _is_gemini_agent, which test_functional.py, test_node_functional.py and test_instrumentation.py patch, so I've left it out of this PR. I can follow up if a maintainer wants it.
In tests/unittests/telemetry, the same 13 schema-recording cases fail locally with and without this change; everything else passes.
Thanks @adityasingh2400, and for correcting the duplicate part. Looking at 40a79f0, the parent_agent loop picks the nearest LlmAgent ancestor and skips non-LlmAgent parents the same way canonical_model does, and the new test covers the Gemini-root case. Since _inference_system_name uses the same helper, the child’s provider name should come out right now too.
Keeping llm_request.model for a separate PR is fine with me, since it only differs when a before_model_callback changes the model. This still reproduces on 2.13.0 and on current main, and #7893 merges cleanly on top of main.
@jawoszek could you approve the pending workflows and take a look at #7893 when you get a chance?
Activity
surajksharma07 commentedon Nov 2, 2026
@adityasingh2400 Reproduced both directions on 2.13.0 and current main: the child under an openai/gpt-4o root loses its generate_content span, and a child inheriting Gemini while the default model is non-Gemini gets recorded by ADK as well as the instrumentor.
Walking up parent_agent for the model name in _agent_model_name, like #7893 does, fixed both cases for me with the default model still as the fallback. The telemetry tests I could run locally showed no new failures.
Could you check that on your side and add a test for the duplicate case too? It might also be worth trying llm_request.model first at those call sites since that’s the model actually sent. Please test it well with the instrumentor on before #7893 goes for review.
adityasingh2400 commentedon Nov 3, 2026
Thanks, @surajksharma07. I added
test_generate_content_span_with_inherited_gemini_modelto #7893 (40a79f0). It fails on main and passes with the fix.I also ran the Gemini case end to end with the real instrumentor (0.7b2), answering Gemini's HTTP call with canned responses. That showed I'd overstated it here: the child isn't recorded twice. When ADK records a call itself,
maybe_without_genai_instrumentationtakes the instrumentor off it, so on main the child's span came from ADK while its Gemini root's came from the instrumentor. With the fix, both come from the instrumentor. I've corrected the issue text and the PR description.On
llm_request.model:basic.pysets it fromagent.canonical_model.model, so with the ancestor walk it matches_agent_model_namefor every request the flow builds. The two differ only if abefore_model_callbackrenames the model. Switching would also bypass_is_gemini_agent, whichtest_functional.py,test_node_functional.pyandtest_instrumentation.pypatch, so I've left it out of this PR. I can follow up if a maintainer wants it.In
tests/unittests/telemetry, the same 13 schema-recording cases fail locally with and without this change; everything else passes.surajksharma07 commentedon Nov 4, 2026
Thanks @adityasingh2400, and for correcting the duplicate part. Looking at 40a79f0, the parent_agent loop picks the nearest LlmAgent ancestor and skips non-LlmAgent parents the same way canonical_model does, and the new test covers the Gemini-root case. Since _inference_system_name uses the same helper, the child’s provider name should come out right now too.
Keeping llm_request.model for a separate PR is fine with me, since it only differs when a before_model_callback changes the model. This still reproduces on 2.13.0 and on current main, and #7893 merges cleanly on top of main.
@jawoszek could you approve the pending workflows and take a look at #7893 when you get a chance?
fix: trace sub-agents by the model they inherit