Repository navigation
fix(sarvam): reconnect on SpeechStream.update_options instead of ending the stream - #8382
neil-sarvam wants to merge 17 commits into
Conversation
…ng the stream update_options() closes the connection to reopen it with the new options, but SpeechStream._run ran a single connection and returned, so the stream ended. _run now loops while a reconnect was requested; the input channel carries on into the new connection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s a reconnect On a requested reconnect, send the buffered audio and a flush on the old connection and end the turn, carry unsent samples into the new connection, and advance start_time_offset so transcript times stay on the stream timeline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A transcript from earlier in the utterance left the settle event set, so the reconnect closed the old connection without waiting for the flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fold the reconnect tests into the reorganized tests/test_plugin_sarvam_stt.py, following livekit#8510, and re-anchor the provider clock with restart_provider_clock() on reconnect, following livekit#7795. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Carrying it into the new connection put it ahead of the first frame the new connection reads, which is where the provider clock is anchored, so a pause before that frame shifted the carried audio's transcript times. Also run the reconnect tests on virtual time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…orrectly Sending it on the old connection could cut off speech starting at the reconnect. Carry it over again, and shift the new connection's timestamps back by its duration, since the provider clock starts with it but is anchored on the first frame read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Audio already sent to the old connection, but not yet reported as speech by Sarvam's VAD, was lost when that connection closed without a flush, so a short reply starting at the reconnect could disappear. Keep the audio sent since the last utterance ended (up to 0.5 s), replay it on the new connection, and shift that connection's timestamps back by the replayed duration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Added 44c306b after @tinalenguyen's note on #8329: an idle reconnect now replays the audio sent since the last utterance ended (up to 0.5 s), so speech Sarvam's VAD hadn't reported yet isn't lost with the old connection, and shifts the new connection's timestamps back by the replayed duration. |
A retry after an error kept the previous connection's carried-audio correction. Recompute it, and replay unclaimed audio, at the start of each connection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Track the samples sent on each connection. Audio carried into a new connection (replay plus unsent samples) is anchored where it sits on the previous connection's timeline, so its transcript times are right even before the next frame is read. Replay audio is trimmed at the finished utterance's speech_end rather than cleared at END_SPEECH, so the next utterance's onset, sent before that event arrives, is kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…or replay Record the sample count at END_SPEECH and, when the transcript has no speech_end, drop replay audio only up to that point, so the next turn's onset survives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…audio Sarvam sends 0 when it has no timestamp; _positive_time already treats it as missing, so do the same here and fall back to the END_SPEECH position. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… boundary A transcript with no usable speech_end that arrives before its END_SPEECH no longer counts everything sent so far as covered; the trim waits for END_SPEECH. Each END_SPEECH position is used once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tinalenguyen
left a comment
There was a problem hiding this comment.
hi, thanks for the PR! could we update the description? it still only covers the reconnect loop, but the PR now also settles the open utterance, carries and replays audio across the reconnect and re-anchors timestamps. a few findings on the reconnect path below:
1. a reconnect between END_SPEECH and the final transcript leaves the turn open — stt.py:1375
after END_SPEECH, _speaking is false and _pending_eos is true while the 1 s eos fallback waits for the final. if update_options() lands in that window, _settle_utterance_for_reconnect returns early, and the finally in _run_connection cancels both the fallback task and the message task. the final transcript is lost, END_OF_SPEECH is never emitted (so the next START_SPEECH comes without an end in between), and since no transcript released _recent_audio, up to 0.5 s of the finished utterance is replayed on the new connection and can be transcribed again. let's settle on _speaking or self._pending_eos too, waiting for the final the same way
2. the settle flush is sent even when flush_signal isn't enabled — stt.py:1401
the normal path only sends {"type": "flush"} when self._opts.flush_signal is set, and the url only passes flush_signal when it's given (default None). does sarvam honor a flush on a connection that didn't enable it? if not, every mid-speech reconnect waits the full 2 s RECONNECT_SETTLE_TIMEOUT, ends the turn without a final, and the audio is gone since _recent_audio was cleared assuming the flush covered it
3. an empty transcript doesn't end the settle wait — stt.py:1748
_handle_transcript_data returns on an empty transcript before _utterance_resolved is set, so if the flush comes back empty (a cough or noise that tripped VAD) settle blocks for 2 s with no audio going to either connection. maybe we could set _utterance_resolved for an empty final as well
Nits / fine as is
- the base64
audiomessage is now built in three places (send_full_chunks, the flush tail and the settle tail), and the settle one skips_remember_sent_audio. a small_send_pcm(ws, pcm)helper would keep them in step - left a small comment about the carry call before the loop
…ng the stream (livekit#8382) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… on an empty final A reconnect after END_SPEECH but before the final returned without settling, so the old connection's final and END_OF_SPEECH were lost and the finished utterance could be replayed. Settle that window too, waiting for the final rather than the EOS fallback timer, and let an empty final end the wait. Send PCM through one helper, which also keeps the settle tail for replay, and drop the carry call before the first connection since the stream never retries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks @tinalenguyen! Description updated to cover the whole PR. In 7d4caa2:
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sarvam answers a flush with END_SPEECH first and the final just after, and after an earlier final END_SPEECH emits END_OF_SPEECH at once, which ended the wait and closed the socket before the flushed final arrived. Only a transcript, or an empty one, ends the wait now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tinalenguyen
left a comment
There was a problem hiding this comment.
thanks for the updates! the three findings and the nits are fixed, one more on the settle path and a small comment below
1. the flushed final can still come after END_OF_SPEECH — stt.py:1401
when the utterance already had a final, the END_SPEECH sarvam sends back for the flush goes through _handle_events, which sees _final_received_for_utterance and emits END_OF_SPEECH right away, so the flushed final comes after it as a standalone FINAL. test_reconnect_waits_past_end_speech_for_the_flushed_final produces exactly that order (START_OF_SPEECH, FINAL, END_OF_SPEECH, FINAL) and still passes since it only checks the texts. with no earlier final, the same END_SPEECH starts a new 1 s eos fallback, the one the settle step just cancelled, so a final slower than that also lands after END_OF_SPEECH. maybe we could have _handle_events leave END_OF_SPEECH to the settle step while it's waiting, and assert the event order in that test
…; don't reconnect after end_input While a reconnect settles the open utterance, END_SPEECH (Sarvam's answer to the flush) no longer emits END_OF_SPEECH or starts the fallback timer, so the flushed final comes before END_OF_SPEECH. Once input has ended, update_options() no longer reconnects: the old connection finishes, and a new one would only replay its audio. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks @tinalenguyen! Fixed in 75ad9ef:
The same changes are on #8329. |
Split out of #8329 so that each change gets its own PR.
Problem
sarvam.SpeechStream.update_options()closes the connection so it can reopen with the new options. But_runran one connection and returned, so the stream simply ended and transcription stopped.Simply looping isn't enough. Swapping the connection mid-call also has to avoid losing speech or mistiming transcripts.
Fix
_run_connectionreturnsTruewhen it closed for a requested reconnect,_runloops while that's the case, and the input channel carries on into the new connection.START_SPEECHseen, orEND_SPEECHseen with the final still pending), the buffered audio and a{"type": "flush"}go to the old connection.RECONNECT_SETTLE_TIMEOUT(2 s) for its final. An empty final also counts.END_OF_SPEECHeither way.flush_signal(checked live on saaras:v3 and v4).RECONNECT_REPLAY_S, 0.5 s) is replayed there too, so speech Sarvam's VAD hadn't reported yet isn't lost.speech_end, or atEND_SPEECHwhen a final has none.restart_provider_clock()re-anchors on the first new frame.Testing
tests/test_plugin_sarvam_stt.py, all on virtual time:END_SPEECHand the final;🤖 Generated with Claude Code