Skip to content

feat(lk): lk agent simulate compare prints the compare page for runs, or their data with --json - #1107

Draft
u9g wants to merge 5 commits into
mainfrom
jason/sim-compare
Draft

u9g wants to merge 5 commits into
mainfrom
jason/sim-compare

Conversation

@u9g

@u9g u9g commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Adds lk agent simulate compare RUN_ID RUN_ID [RUN_ID...]. It compares finished or running simulation runs by ID, with the first run as the baseline. It prints one of two things:

  • Default: the URL of the dashboard's compare page (livekit/web#7065), …/projects/<project>/simulations/compare?runs=a,b.
  • --json: {"version": 1, "runs": [...]}, where each run is exactly what lk agent simulate export prints for it (the run, plus its decoded summary), in the order given. Like export, it refuses a run that hasn't finished, and prints nothing unless every run is ready.

It takes 2 to 4 distinct run IDs. Without a project ID (credentials from LIVEKIT_* or flags), it asks for --project or --json instead of printing a broken link. 4 is the compare page's limit: the page drops any runs past it, so the CLI rejects them instead.

No protocol or backend change. The --compare KNOB=… flag this PR used to add is gone.

Commits: a refactor that splits building a run's export JSON from encoding it, then the subcommand.

@u9g

u9g commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

Demo against two real staging runs: compare prints the dashboard compare page, --json prints the runs as export does, one run is rejected, and the old --compare knob flag no longer exists.

sim-compare.mp4

@u9g
u9g force-pushed the jason/sim-compare branch from f6828da to 79378ec Compare November 6, 2026 23:08
@u9g u9g changed the title feat(lk): lk agent simulate --compare runs every scenario once per option of a knob feat(lk): lk agent simulate compare prints the compare page for runs, or their data with --json Nov 6, 2026
u9g added 3 commits November 6, 2026 16:06
With credentials from LIVEKIT_* or inline flags the project ID is empty, and
compare printed /projects//simulations/compare?... and exited 0. Return no
URL in that case, as simulationDashboardURL does, and say how to get one.
export only prints finished runs, so stdout is a complete export or nothing,
but compare --json printed whatever state each run was in. Hold every run to
the same check before printing anything.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant