Repository navigation
Conversation
… LLM responses `lk agent simulate replay SESSION_ID [entrypoint] --user-auth` reruns a recorded session against the agent on disk. It reads the session's llm_request spans from the Public API, rebuilds each LLM request the session made (following input deltas back to their base span) and the response it got, and runs the agent in console mode with LIVEKIT_LLM_REPLAY_URL pointing at a local server that answers those requests in the session's order. The session's user turns are sent to the agent as text, each once the agent has made every request that preceded it and gone idle. The replay fails at the first LLM request that differs from the session's (model, provider, instructions, tools, or any input message, so a changed prompt or tool result shows exactly where), at a request that arrives out of order or after the session's last, and when a request never comes. It passes once the agent has made every request the session made, and no more. A session's spans land out of order for a while after it ends, and a replay of a session missing its last requests would pass on a prefix, so a session is replayed only once it ended three minutes ago.
Contributor
Author
|
Replaying session RM_Y7FMScSYoSBT: against the unchanged agent all 6 LLM requests match; with a tool's eligibility threshold changed it fails at request 6 on the tool result; with the prompt edited it fails at request 1 on the instructions. simulate-replay.mp4 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
lk agent simulate replay SESSION_ID [entrypoint] --user-auth. It reruns a recorded session against the agent on disk, answering every LLM request with the response the same request got in the session. It fails at the first request that differs from the session's, or that arrives out of order.Needs the agent SDK's replay mode: livekit/agents#8179 (Python), livekit/agents-js#2878 (Node). An agent on an SDK without it calls its provider instead, and the replay reports that the expected request never came.
How it works:
llm_requestspans from the Public API and rebuilds each LLM request the session made, in order, with the response it got. A span recorded as an input delta is rebuilt from its base span.LIVEKIT_LLM_REPLAY_URLpointing at a local server that answers those requests in the session's order.Known limits: tool parameter schemas are not recorded on spans (only names and descriptions), so a schema-only change is not detected. A voice session with interruptions diverges, because the recording holds the cut-off assistant text and a text replay produces the full reply.
Tested: unit tests for rebuilding deltas, the diff, user turns, and the server's ordering (
cmd/lk/simulate_replay_calls_test.go), the fullcmd/lksuite, vet, golangci-lint. End to end against real sessions on LiveKit Cloud, for a Python agent and a Node agent: the demo in the comments shows a pass, a tool-result change, and a prompt change.