Skip to content

perf(agent): reload Python dev agents in ~120 ms from a preloaded interpreter - #1104

Draft
u9g wants to merge 3 commits into
jason/python-import-failure-diagnosisfrom
jason/python-dev-warm-reload
Draft

u9g wants to merge 3 commits into
jason/python-import-failure-diagnosisfrom
jason/python-dev-warm-reload

Conversation

@u9g

@u9g u9g commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Stacked on #1102, which is stacked on #1094.

Why a reload was slow

I timed a lk agent dev reload (save → registered worker) on #1094 with the starter template. It takes ~1.2 s, more than a cold start:

Step Time
Debounce 500 ms
Old process's graceful shutdown; the new one waits for it ~200 ms
Fresh interpreter imports livekit.agents and every dependency again ~470 ms

The last step is repeated work: only the user's own files changed.

What this changes

  • Spare interpreter. For Python dev sessions, lk keeps one spare interpreter that has already imported every installed module the last worker used. The list is recorded per project in the user cache dir. A reload activates the spare through stdin, so only the user's modules are imported.
    • The spare starts once the active worker connects to the dev channel, so it doesn't compete with that worker's imports.
    • It's discarded, falling back to a fresh interpreter, if the dependencies changed since it started: pyproject.toml, the lockfiles and site-packages are fingerprinted by mtime.
    • Modules the project itself provides are never preloaded, even if a stale copy is installed.
    • Output the spare prints before activation is held, then flushed on activation.
  • Overlapped shutdown. When the old process has no running jobs, it shuts down while the new one starts. With running jobs the order is unchanged (capture, stop, start, restore), so a restored job never overlaps the original.
  • Shorter debounce, 500 → 100 ms (separate commit, easy to drop). With the spare, the debounce made up 97% of a reload. One save's events arrive within a few ms; I checked that a chunked write and a temp-file-and-rename save (vim, JetBrains) each trigger exactly one reload.

Results

Median of 5–10 runs, against a local livekit-server --dev on macOS, livekit-agents 1.8.10. The PTY harness answers OSC 11/DA1 like a real terminal.

Agent main #1094 this PR
agent-starter-python, reload 1222 ms 1173 ms 122 ms
openai/silero/deepgram/cartesia, reload 1346 ms 1311 ms 117 ms
agent-starter-python, cold start 579 ms 487 ms 496 ms

Checked by hand

  • A job dispatched with lk dispatch create runs in the worker started from the spare.
  • A reload during that job captures it and restores it into the next worker, the same as fix(agent): avoid duplicate imports and expensive watcher scans #1094.
  • No interpreters are left running after exit.
  • TestWarmPythonSparePreloadsInstalledModulesAndWaitsForActivation covers: preloading, never preloading the project's own module, holding output until activation, and recording the module list. It fails with the project-module guard removed.

Still slow, outside lk

  • Local inference process (agents SDK). Agents that use the local turn-detector model wait ~1.3 s before registering, on every start and reload, because AgentServer initializes the inference executor before it registers (worker.py, await self._inference_executor.initialize()). Prototyping that init alongside registration, with job launch waiting for it, took such an agent from 1463 to 528 ms cold, and 1091 to 123 ms per reload with this PR.
  • Eager SDK imports. livekit.agents imports openai (via inference.llm), av (codecs) and requests (telemetry) at import time, though registration needs none of them. Deferring them would take the starter template from 446 to 284 ms to registration.

u9g added 3 commits November 4, 2026 15:38
A reload paid for a fresh interpreter to import livekit.agents and the
project's dependencies (~650 ms with common plugins), although only the user's
code changed, and then waited for the old process's graceful shutdown
(~200 ms) before starting.

Keep one spare interpreter per dev session. It imports every installed module
the last worker used (recorded per project in the user cache dir), then waits
on stdin; a reload activates it, so only the user's modules are imported. The
spare starts once the active worker connects to the dev channel, so it never
competes with its imports. A spare is discarded if dependencies changed since
it started, and modules the project itself provides are never preloaded.

When the old process has no running jobs, it now shuts down while the new one
starts. With jobs, the order is unchanged so a restored job never overlaps the
original.

Reload, save to registered worker (livekit-agents 1.8.10, macOS): 1315 ms
to 512 ms, of which 500 ms is the debounce.
With a preloaded interpreter, the debounce was 97% of a reload. One save's
events arrive within a few milliseconds (write, or write to a temporary file
and rename), so 100 ms still coalesces them.
The spare must preload recorded installed modules but never one the project
provides, hold its output and run only once activated; a run must record its
installed modules for the next spare.
@CLAassistant

CLAassistant commented Nov 4, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants