Skip to content

fix(sarvam): reconnect on SpeechStream.update_options instead of ending the stream - #8382

Open
neil-sarvam wants to merge 17 commits into
livekit:mainfrom
neil-sarvam:sarvam/stt-update-options-reconnect
Open

neil-sarvam wants to merge 17 commits into
livekit:mainfrom
neil-sarvam:sarvam/stt-update-options-reconnect

Conversation

@neil-sarvam

@neil-sarvam neil-sarvam commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Split out of #8329 so that each change gets its own PR.

Problem

sarvam.SpeechStream.update_options() closes the connection so it can reopen with the new options. But _run ran one connection and returned, so the stream simply ended and transcription stopped.

Simply looping isn't enough. Swapping the connection mid-call also has to avoid losing speech or mistiming transcripts.

Fix

  • Reconnect loop: _run_connection returns True when it closed for a requested reconnect, _run loops while that's the case, and the input channel carries on into the new connection.
  • Open utterance: if speech is open on the old connection (START_SPEECH seen, or END_SPEECH seen with the final still pending), the buffered audio and a {"type": "flush"} go to the old connection.
    • The stream waits up to RECONNECT_SETTLE_TIMEOUT (2 s) for its final. An empty final also counts.
    • The turn then ends with END_OF_SPEECH either way.
    • Sarvam honors the flush whether or not the connection enabled flush_signal (checked live on saaras:v3 and v4).
  • Idle reconnect: with no utterance open, nothing is flushed.
    • Unsent samples carry into the new connection.
    • The audio sent since the last utterance ended (up to RECONNECT_REPLAY_S, 0.5 s) is replayed there too, so speech Sarvam's VAD hadn't reported yet isn't lost.
    • That replay buffer is trimmed at each final's speech_end, or at END_SPEECH when a final has none.
  • Timestamps: Sarvam times each connection from zero.
    • A connection that carries audio over is anchored where that audio sits on the previous connection's timeline, from a per-connection sample count.
    • Otherwise restart_provider_clock() re-anchors on the first new frame.

Testing

  • Unit tests in tests/test_plugin_sarvam_stt.py, all on virtual time:
    • the reconnect itself;
    • settling an open utterance, including in the window between END_SPEECH and the final;
    • an empty final, and no final at all;
    • carry-over and replay, with the 0.5 s cap and trimming at utterance boundaries;
    • transcript timing.
  • Results: each fix's test fails without it, all Sarvam tests pass, and ruff and mypy are clean.

🤖 Generated with Claude Code

…ng the stream

update_options() closes the connection to reopen it with the new options, but
SpeechStream._run ran a single connection and returned, so the stream ended.
_run now loops while a reconnect was requested; the input channel carries on
into the new connection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@neil-sarvam
neil-sarvam marked this pull request as ready for review November 5, 2026 10:52
@neil-sarvam
neil-sarvam requested a review from a team as a code owner November 5, 2026 10:52
devin-ai-integration[bot]

This comment was marked as resolved.

…s a reconnect

On a requested reconnect, send the buffered audio and a flush on the old
connection and end the turn, carry unsent samples into the new connection,
and advance start_time_offset so transcript times stay on the stream timeline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

neil-sarvam and others added 2 commits November 11, 2026 11:56
A transcript from earlier in the utterance left the settle event set, so the
reconnect closed the old connection without waiting for the flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fold the reconnect tests into the reorganized tests/test_plugin_sarvam_stt.py, following livekit#8510,
and re-anchor the provider clock with restart_provider_clock() on reconnect, following livekit#7795.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Carrying it into the new connection put it ahead of the first frame the new
connection reads, which is where the provider clock is anchored, so a pause
before that frame shifted the carried audio's transcript times. Also run the
reconnect tests on virtual time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

neil-sarvam and others added 2 commits November 12, 2026 12:40
…orrectly

Sending it on the old connection could cut off speech starting at the reconnect.
Carry it over again, and shift the new connection's timestamps back by its duration,
since the provider clock starts with it but is anchored on the first frame read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Audio already sent to the old connection, but not yet reported as speech by
Sarvam's VAD, was lost when that connection closed without a flush, so a short
reply starting at the reconnect could disappear. Keep the audio sent since the
last utterance ended (up to 0.5 s), replay it on the new connection, and shift
that connection's timestamps back by the replayed duration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@neil-sarvam

Copy link
Copy Markdown
Contributor Author

Added 44c306b after @tinalenguyen's note on #8329: an idle reconnect now replays the audio sent since the last utterance ended (up to 0.5 s), so speech Sarvam's VAD hadn't reported yet isn't lost with the old connection, and shifts the new connection's timestamps back by the replayed duration.

devin-ai-integration[bot]

This comment was marked as resolved.

neil-sarvam and others added 2 commits November 12, 2026 13:14
A retry after an error kept the previous connection's carried-audio correction.
Recompute it, and replay unclaimed audio, at the start of each connection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Track the samples sent on each connection. Audio carried into a new connection
(replay plus unsent samples) is anchored where it sits on the previous connection's
timeline, so its transcript times are right even before the next frame is read.
Replay audio is trimmed at the finished utterance's speech_end rather than cleared
at END_SPEECH, so the next utterance's onset, sent before that event arrives, is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

…or replay

Record the sample count at END_SPEECH and, when the transcript has no speech_end,
drop replay audio only up to that point, so the next turn's onset survives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

…audio

Sarvam sends 0 when it has no timestamp; _positive_time already treats it as
missing, so do the same here and fall back to the END_SPEECH position.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

… boundary

A transcript with no usable speech_end that arrives before its END_SPEECH no
longer counts everything sent so far as covered; the trim waits for END_SPEECH.
Each END_SPEECH position is used once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@tinalenguyen tinalenguyen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi, thanks for the PR! could we update the description? it still only covers the reconnect loop, but the PR now also settles the open utterance, carries and replays audio across the reconnect and re-anchors timestamps. a few findings on the reconnect path below:

1. a reconnect between END_SPEECH and the final transcript leaves the turn open — stt.py:1375

after END_SPEECH, _speaking is false and _pending_eos is true while the 1 s eos fallback waits for the final. if update_options() lands in that window, _settle_utterance_for_reconnect returns early, and the finally in _run_connection cancels both the fallback task and the message task. the final transcript is lost, END_OF_SPEECH is never emitted (so the next START_SPEECH comes without an end in between), and since no transcript released _recent_audio, up to 0.5 s of the finished utterance is replayed on the new connection and can be transcribed again. let's settle on _speaking or self._pending_eos too, waiting for the final the same way

2. the settle flush is sent even when flush_signal isn't enabled — stt.py:1401

the normal path only sends {"type": "flush"} when self._opts.flush_signal is set, and the url only passes flush_signal when it's given (default None). does sarvam honor a flush on a connection that didn't enable it? if not, every mid-speech reconnect waits the full 2 s RECONNECT_SETTLE_TIMEOUT, ends the turn without a final, and the audio is gone since _recent_audio was cleared assuming the flush covered it

3. an empty transcript doesn't end the settle wait — stt.py:1748

_handle_transcript_data returns on an empty transcript before _utterance_resolved is set, so if the flush comes back empty (a cough or noise that tripped VAD) settle blocks for 2 s with no audio going to either connection. maybe we could set _utterance_resolved for an empty final as well

Nits / fine as is

  • the base64 audio message is now built in three places (send_full_chunks, the flush tail and the settle tail), and the settle one skips _remember_sent_audio. a small _send_pcm(ws, pcm) helper would keep them in step
  • left a small comment about the carry call before the loop

neil-sarvam added a commit to neil-sarvam/agents that referenced this pull request Nov 17, 2026
…ng the stream (livekit#8382)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on an empty final

A reconnect after END_SPEECH but before the final returned without settling, so the
old connection's final and END_OF_SPEECH were lost and the finished utterance could
be replayed. Settle that window too, waiting for the final rather than the EOS
fallback timer, and let an empty final end the wait. Send PCM through one helper,
which also keeps the settle tail for replay, and drop the carry call before the
first connection since the stream never retries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@neil-sarvam

Copy link
Copy Markdown
Contributor Author

Thanks @tinalenguyen! Description updated to cover the whole PR. In 7d4caa2:

  1. Good catch. Settle now also runs when END_SPEECH has arrived but the final hasn't (_speaking or _pending_eos). It cancels the EOS fallback timer and waits for the old connection's final, so the final, END_OF_SPEECH and the replay trim all come out right. Test: test_reconnect_after_end_speech_waits_for_the_final.
  2. I checked this live. Sarvam honors {"type": "flush"} on a connection opened without flush_signal. With a clip cut mid-word, a flush brought back END_SPEECH and the final in about 80 ms, and without it no final came back. Same with flush_signal=true, on both saaras:v3 and v4. So no change here.
  3. An empty final now sets _utterance_resolved too, so noise that tripped VAD settles at once. Test: test_reconnect_settles_on_an_empty_final.
  • _send_pcm(ws, pcm) now builds the audio message in all three places, and it records the audio for replay, so the settle tail is covered too.
  • Dropped the carry call before the loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Sarvam answers a flush with END_SPEECH first and the final just after, and after an
earlier final END_SPEECH emits END_OF_SPEECH at once, which ended the wait and closed
the socket before the flushed final arrived. Only a transcript, or an empty one, ends
the wait now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@tinalenguyen tinalenguyen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the updates! the three findings and the nits are fixed, one more on the settle path and a small comment below

1. the flushed final can still come after END_OF_SPEECH — stt.py:1401

when the utterance already had a final, the END_SPEECH sarvam sends back for the flush goes through _handle_events, which sees _final_received_for_utterance and emits END_OF_SPEECH right away, so the flushed final comes after it as a standalone FINAL. test_reconnect_waits_past_end_speech_for_the_flushed_final produces exactly that order (START_OF_SPEECH, FINAL, END_OF_SPEECH, FINAL) and still passes since it only checks the texts. with no earlier final, the same END_SPEECH starts a new 1 s eos fallback, the one the settle step just cancelled, so a final slower than that also lands after END_OF_SPEECH. maybe we could have _handle_events leave END_OF_SPEECH to the settle step while it's waiting, and assert the event order in that test

…; don't reconnect after end_input

While a reconnect settles the open utterance, END_SPEECH (Sarvam's answer to the flush)
no longer emits END_OF_SPEECH or starts the fallback timer, so the flushed final comes
before END_OF_SPEECH. Once input has ended, update_options() no longer reconnects: the
old connection finishes, and a new one would only replay its audio.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@neil-sarvam

Copy link
Copy Markdown
Contributor Author

Thanks @tinalenguyen! Fixed in 75ad9ef:

  1. While a reconnect settles the open utterance, END_SPEECH (Sarvam's answer to the flush) no longer emits END_OF_SPEECH or starts the EOS fallback. The settle step closes the turn after the flushed final, so the order is now START_OF_SPEECH, FINAL, FINAL, END_OF_SPEECH. test_reconnect_waits_past_end_speech_for_the_flushed_final now asserts the full event order, and test_reconnect_without_an_earlier_final_waits_past_the_eos_fallback covers the no-earlier-final case with a final slower than 1 s.
  • update_options() after end_input(): no reconnect now, see the inline reply.

The same changes are on #8329.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants