Skip to content

feat(simulate): replay a session against the local agent with its own LLM responses - #1082

Open
u9g wants to merge 1 commit into
mainfrom
jason/simulate-replay
Open

u9g wants to merge 1 commit into
mainfrom
jason/simulate-replay

Conversation

@u9g

@u9g u9g commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Adds lk agent simulate replay SESSION_ID [entrypoint] --user-auth. It reruns a recorded session against the agent on disk, answering every LLM request with the response the same request got in the session. It fails at the first request that differs from the session's, or that arrives out of order.

Needs the agent SDK's replay mode: livekit/agents#8179 (Python), livekit/agents-js#2878 (Node). An agent on an SDK without it calls its provider instead, and the replay reports that the expected request never came.

How it works:

  • Reads the session's llm_request spans from the Public API and rebuilds each LLM request the session made, in order, with the response it got. A span recorded as an input delta is rebuilt from its base span.
  • Runs the agent locally in console mode, with LIVEKIT_LLM_REPLAY_URL pointing at a local server that answers those requests in the session's order.
  • Sends the session's user turns to the agent as text, each once the agent has made every request that came before it and gone idle.
  • Fails on a request that differs in model, provider, instructions, tools or any input message, and shows the recorded and replayed values side by side. Also fails on a request that arrives out of order, one past the session's last, and one that never comes. Passes once the agent has made every request the session made, and no more.
  • Tools run for real, so a tool that now returns something else fails the replay at the request its result feeds.
  • Refuses a session that ended less than three minutes ago: spans land out of order for a while after a session ends, and a replay missing the last requests would pass on a prefix.

Known limits: tool parameter schemas are not recorded on spans (only names and descriptions), so a schema-only change is not detected. A voice session with interruptions diverges, because the recording holds the cut-off assistant text and a text replay produces the full reply.

Tested: unit tests for rebuilding deltas, the diff, user turns, and the server's ordering (cmd/lk/simulate_replay_calls_test.go), the full cmd/lk suite, vet, golangci-lint. End to end against real sessions on LiveKit Cloud, for a Python agent and a Node agent: the demo in the comments shows a pass, a tool-result change, and a prompt change.

… LLM responses

`lk agent simulate replay SESSION_ID [entrypoint] --user-auth` reruns a
recorded session against the agent on disk. It reads the session's
llm_request spans from the Public API, rebuilds each LLM request the
session made (following input deltas back to their base span) and the
response it got, and runs the agent in console mode with
LIVEKIT_LLM_REPLAY_URL pointing at a local server that answers those
requests in the session's order. The session's user turns are sent to the
agent as text, each once the agent has made every request that preceded it
and gone idle.

The replay fails at the first LLM request that differs from the session's
(model, provider, instructions, tools, or any input message, so a changed
prompt or tool result shows exactly where), at a request that arrives out
of order or after the session's last, and when a request never comes. It
passes once the agent has made every request the session made, and no more.

A session's spans land out of order for a while after it ends, and a replay
of a session missing its last requests would pass on a prefix, so a session
is replayed only once it ended three minutes ago.
@u9g

u9g commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

Replaying session RM_Y7FMScSYoSBT: against the unchanged agent all 6 LLM requests match; with a tool's eligibility threshold changed it fails at request 6 on the tool result; with the prompt edited it fails at request 1 on the instructions.

simulate-replay.mp4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant