Skip to content

[live] ChatBot as a tool #3954

Closed
Closed

Description

@simone-viozzi

** Please make sure you read the contribution guide and file the issues in the right place. **
Contribution guide.

Is your feature request related to a problem? Please describe.
When building conversational Live agents (e.g., voice assistants) that integrate existing text-based chatbot agents, the current ADK AgentTool has two significant limitations:

  1. Synchronous blocking: Currently, when a Live agent invokes another agent using AgentTool, the main conversation freezes until the sub-agent completes processing. This leads to poor conversational UX, especially in voice interactions, where silence can confuse or frustrate users.

  2. Lack of persistent state: The current AgentTool implementation creates a fresh session for every invocation, making it impossible to reuse or persist conversational state between interactions. This severely limits the reusability of existing conversational chatbots, as the chat history and context are lost across invocations.

These issues make it challenging to reuse existing chatbots effectively in streaming or voice-based scenarios.

Describe the solution you'd like
I'd like the ADK to officially support an improved AgentTool with two key enhancements:

  1. Asynchronous execution: Allow AgentTool to be invoked asynchronously in a streaming manner, so the calling Live agent can remain responsive and conversational while the child agent processes the request in the background.

  2. Persistent conversation sessions: Enable AgentTool to maintain persistent conversational state and session context between multiple invocations, allowing existing chatbot agents to seamlessly preserve history and context.

The ideal solution would look similar to:

class StatefulAgentTool(AgentTool):
    """AgentTool that maintains conversation history across invocations.

    For each parent session, a dedicated child session is created and reused.
    This allows the wrapped agent to see prior conversation turns.

    Attributes:
        _sessions: Map of parent_session_id → (Runner, child_session_id).
    """

    def __init__(self, agent: LlmAgent, **kwargs: Any) -> None:
        """Initialize the stateful agent tool.

        Args:
            agent: The LlmAgent to wrap.
            **kwargs: Passed to AgentTool.__init__.
        """
        super().__init__(agent, **kwargs)
        self._sessions: dict[str, tuple[Runner, Session]] = {}

    async def _get_or_create_session(
        self,
        tool_context: ToolContext,
    ) -> tuple[Runner, Session]:
        """Get existing or create new (Runner, child_session_id) for parent.

        Args:
            parent_session_id: The parent voicebot session ID.

        Returns:
            Tuple of (Runner, child_session_id) for this parent session.
        """
        parent_session_id = tool_context.session.id
        if parent_session_id in self._sessions:
            log.debug(
                "Reusing existing child session for parent=%s",
                parent_session_id,
            )
            return self._sessions[parent_session_id]

        invocation_context = tool_context._invocation_context
        parent_app_name = invocation_context.app_name if invocation_context else None
        child_app_name = parent_app_name or self.agent.name
        runner = Runner(
            app_name=child_app_name,
            agent=self.agent,
            artifact_service=ForwardingArtifactService(tool_context),
            session_service=InMemorySessionService(),
            memory_service=InMemoryMemoryService(),
            credential_service=tool_context._invocation_context.credential_service,
            plugins=list(tool_context._invocation_context.plugin_manager.plugins),
        )

        state_dict = {
            k: v
            for k, v in tool_context.state.to_dict().items()
            if not k.startswith("_adk")  # Filter out adk internal states
        }
        child_session = await runner.session_service.create_session(
            app_name=child_app_name,
            user_id=tool_context._invocation_context.user_id,
            state=state_dict,
        )

        log.debug(
            "Created new child session=%s for parent=%s",
            child_session.id,
            parent_session_id,
        )

        self._sessions[parent_session_id] = (runner, child_session)
        return runner, child_session

    async def run_async(
        self,
        *,
        args: dict[str, Any],
        tool_context: ToolContext,
    ) -> Any:
        """Run the wrapped agent, reusing session for this parent.

        Args:
            args: Tool arguments (passed to agent as "request").
            tool_context: ADK tool context with parent session info.

        Returns:
            The agent's response string.
        """
        runner, child_session = await self._get_or_create_session(tool_context)

        if self.skip_summarization:
            tool_context.actions.skip_summarization = True

        if isinstance(self.agent, LlmAgent) and self.agent.input_schema:
            input_value = self.agent.input_schema.model_validate(args)
            content = types.Content(
                role="user",
                parts=[
                    types.Part.from_text(
                        text=input_value.model_dump_json(exclude_none=True)
                    )
                ],
            )
        else:
            content = types.Content(
                role="user",
                parts=[types.Part.from_text(text=args["request"])],
            )

        # Accumulate all text parts from all events (not just the last one)
        # to capture multi-step responses like: "I'll send SMS" → tool call → "SMS sent"
        all_text_parts: list[str] = []
        async with Aclosing(
            runner.run_async(
                user_id=child_session.user_id,
                session_id=child_session.id,
                new_message=content,
            )
        ) as agen:
            async for event in agen:
                # Forward state delta to parent session.
                if event.actions.state_delta:
                    tool_context.state.update(event.actions.state_delta)
                # Collect text parts from events with content
                if event.content and event.content.parts:
                    for part in event.content.parts:
                        if part.text:
                            all_text_parts.append(part.text)

        if not all_text_parts:
            return ""
        merged_text = "\n\n".join(all_text_parts)
        if isinstance(self.agent, LlmAgent) and self.agent.output_schema:
            tool_result = self.agent.output_schema.model_validate_json(
                merged_text
            ).model_dump(exclude_none=True)
        else:
            tool_result = merged_text
        return tool_result

    def cleanup_session(self, parent_session_id: str) -> None:
        """Remove stored session for a parent (call on voicebot session end).

        TODO: Wire this up to voicebot session lifecycle (e.g., on disconnect/timeout).

        Args:
            parent_session_id: The parent session ID to clean up.
        """
        if parent_session_id in self._sessions:
            log.debug("Cleaning up child session for parent=%s", parent_session_id)
            del self._sessions[parent_session_id]


def make_agent_tool_async(agent_tool: StatefulAgentTool):
    """Create a streaming tool that wraps an LlmAgent via AgentTool.

    The returned function:
    - yields exactly once with {"status": "started"}
    - then waits for the agent to finish
    - then yields {"status": "done", "reply": <agent reply>}
    """

    async def execute_async_agent_tool(
        request: str,
        user_language: str,
        tool_context: ToolContext,
    ) -> AsyncGenerator[dict[str, str], None]:
        pending = tool_context.state.get("pending_jobs", {})

        if agent_tool.agent.name in pending:
            yield {
                "status": "already running - wait for completion - do not call again until **done**",
            }
            log.debug(f"Agent tool {agent_tool.agent.name} already running")
            return

        pending[agent_tool.agent.name] = "started"
        tool_context.state["pending_jobs"] = pending

        # First (and only) "pending" event
        yield {
            "status": (
                "started - the chatbot is processing your request, please wait... "
                "you may continue the conversation to the user while waiting."
            ),
        }

        log.debug(
            f"Started long-running agent tool for agent_name={agent_tool.agent.name}"
        )

        if "chatbot_chat_history" not in tool_context.state:
            tool_context.state["chatbot_chat_history"] = []
        user_entry = {
            "role": "user",
            "text": request,
            "language": user_language,
        }
        tool_context.state["chatbot_chat_history"].append(user_entry)
        log.debug(
            f"Added user chat history entry: session_id={tool_context.session.id}, "
            f"entry={user_entry}, total_history_len={len(tool_context.state['chatbot_chat_history'])}"
        )

        # Run the wrapped AgentTool - conversation history is now managed by
        # StatefulAgentTool which reuses sessions per parent session ID
        reply = await agent_tool.run_async(
            args={"request": request, "user_language": user_language},
            tool_context=tool_context,
        )

        # TODO need to handle when the LLM returns None

        yield {
            "status": "done",
            "reply": reply,
        }

        tool_context.state["pending_jobs"].pop(agent_tool.agent.name, None)
        log.debug(
            f"Completed long-running agent tool for agent_name={agent_tool.agent.name}"
        )

        ai_entry = {
            "role": "ai",
            "text": reply,
            "language": user_language,
        }
        tool_context.state["chatbot_chat_history"].append(ai_entry)
        log.debug(
            f"Added AI chat history entry: session_id={tool_context.session.id}, "
            f"entry={ai_entry}, total_history_len={len(tool_context.state['chatbot_chat_history'])}"
        )

    # --- This is the important part for name & description ---
    execute_async_agent_tool.__name__ = agent_tool.name

    # TODO use _get_declaration to copy over function signature?
    # Use the agent's description *plus* a note that it's long-running.
    base_desc = agent_tool.agent.description or ""
    execute_async_agent_tool.__doc__ = base_desc + (
        "\n\nNOTE: This is a long-running streaming version of this tool. "
        "It first returns status='started', then later returns status='done' with the reply. "
        "If called while already running, returns status='already running'. "
        "Wait for status='done' before calling this tool again."
    )

    return execute_async_agent_tool


voicebot_single = Agent(
    name="voicebot",
    model=voice_model,
    instruction=...,
    tools=[
        make_agent_tool_async(
            StatefulAgentTool(
                chatbot,
                skip_summarization=True,
            )
        ),
    ],
    description=(
        ...
    ),
)

This pattern allows the Live agent to stay conversational and interactive, improving the UX dramatically by providing continuous responsiveness while preserving the chat context.

Describe alternatives you've considered
I've developed a proof-of-concept that partially solves these issues by:

  • Creating a subclass StatefulAgentTool to manage persistent child agent sessions.
  • Developing a wrapper function make_agent_tool_async to turn the existing AgentTool into a streaming, asynchronous version.

However, this PoC approach has limitations:

  • substantial manual prompting is required to teach the Live model how to handle asynchronous tool states (started, done, etc.), making it difficult to scale or generalize.
  • The Stateful Agent Tool does not sync the session state with the VoiceBot

Additional context
This feature aligns with ongoing efforts to improve streaming tool capabilities, referenced in issue #2014.

Activity

  1. added
    live[Component] This issue is related to live, voice and video chat
    on Dec 17, 2025
  2. added theissue type on Dec 17, 2025
  3. klateefa commented on Feb 11, 2026

    @klateefa

    @simone-viozzi Thanks for the update. The request is currently being reviewed, and I will let you know if anything further is needed. Thank you for your understanding.

  4. added
    needs review[Status] The PR/issue is awaiting review from the maintainer
    on Feb 11, 2026
  5. allen-stephen commented on Jul 14, 2026

    @allen-stephen
    Collaborator

    Thanks for the detailed proposal, @simone-viozzi!

    Non-blocking execution landed in 5620d8f. Set response_scheduling (WHEN_IDLE / SILENT / INTERRUPT) on a tool and it runs in a background task while the live agent keeps conversing. See the new contributing/samples/live/live_non_blocking_tool_agent sample for an example.

    For the persistent conversation aspect, we'll continue tracking that in #6019. Let's consolidate that discussion there.

    Closing this out — please follow #6019 for the persistence work, and feel free to reopen if any issues remain on the live side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

live[Component] This issue is related to live, voice and video chatneeds review[Status] The PR/issue is awaiting review from the maintainer

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions