Repository navigation
android: fix AudioRecord released under a live reader when the stop join times out - #266
Conversation
…read stopRecordingIfNeededImpl joined the reader thread while holding audioRecordStateLock and never stopped the AudioRecord first. Two consequences: - The reader acquires that lock at the top of every loop iteration, so a stop racing a loop boundary deadlocks against the join until its 2s timeout: the thread waits for the lock while the join waits for the thread. Observed in production as a ~2s stall on every capture stop cycle under churn. - A reader stuck inside a blocking AudioRecord.read() (stalled capture HAL, e.g. Bluetooth SCO route churn) has nothing to unblock it, so the join times out structurally. Stop the record inside the locked phase (a blocking read() returns promptly once the record leaves the recording state), detach it from the shared field so a concurrent init/start builds fresh state instead of touching a record the stopping thread may still be reading, and perform the join outside the lock. The stop entry points clear their request flags under the lock and run the join outside it; a concurrent stop that loses the race now returns cleanly instead of tripping the audioThread assert.
When the stop path's join timed out it released the AudioRecord anyway, while the abandoned reader thread could still be inside AudioRecord.read() on that record. Releasing a record under an in-flight read corrupts the platform client proxy accounting and aborts the process: releaseBuffer: mUnreleased out of range, !(stepCount:480 <= mUnreleased:0 <= mFrameCount:3840) On join timeout, transfer release responsibility to the reader through an atomic handoff slot (orphanedRecord + cleanupDone): the reader releases the record when it finally exits, and if it has already passed its cleanup point during the handoff the stop path reclaims and releases it. Exactly one side performs the release in every interleaving.
…state With the join no longer serializing everything under audioRecordStateLock, a reader that is being stopped could observe state belonging to the next recording session and corrupt it: - The loop-top snapshot re-reads the shared audioRecord field each iteration, so a reader that passed the while(keepAlive) check could adopt a successor record installed during the stop's join window, read it concurrently with the new reader, and stop it on exit. keepAlive is now re-checked against the snapshot (the stop path publishes keepAlive=false before detaching, under the same lock), and exit cleanup stops only the record this thread actually read (activeRecord), never the shared field. - The startup preamble read the shared field twice without the lock and asserted on its recording state, which now races the stop's detach+stop: taken under the lock with a keepAlive guard. - The mid-loop re-init leg could re-create a record after a stop had already finished, leaving a record behind that no stop path would ever release; the re-init, restart, and state verification now run in one critical section gated on a keepAlive re-check, and a record stopped by shutdown is no longer misreported as a start failure (which permanently disabled useAudioRecord). - A read unblocked by the stop path's own AudioRecord.stop() exits through the loop condition instead of logging a spurious read error.
davidliu
left a comment
There was a problem hiding this comment.
LGTM!
If you'd be willing to add the test harness in a separate PR, that'd be helpful too. The tests at sdk/android/tests/src/org/webrtc/audio are Robolectric capable (uses AndroidJUnit4 as its runner).
You'd just need to add the path to the file in the BUILD.gn here:
https://github.com/webrtc-sdk/webrtc/blob/08b9408717876048bca582530ba6f3e732a02479/sdk/android/BUILD.gn#L1914
|
Thanks! Will do - I'll port the harness to a Robolectric test under sdk/android/tests/src/org/webrtc/audio and put up a separate PR for it. Also do you want a matching PR for m146_release (it has the same stop path code), or would you rather pick it across on your side? |
We'll handle it on our side, we're not using m146_release actually, we're in the midst of an upgrade to m150. |
…oin times out (#266) ## Problem Production crash on Android (Samsung SM-G766U1, Android 16, LiveKit consumer of `io.github.webrtc-sdk:android-prefixed:144.7559.05`) during rapid capture stop/start cycling while the capture HAL was under pressure (Bluetooth SCO route churn plus a second concurrent capture stack): ``` Abort message: 'releaseBuffer: mUnreleased out of range, !(stepCount:480 <= mUnreleased:0 <= mFrameCount:3840), BufferSizeInFrames:3840' android::ClientProxy::releaseBuffer android::AudioRecord::read android.media.AudioRecord.read livekit.org.webrtc.audio.WebRtcAudioRecord$AudioRecordThread.run ``` `stopRecordingIfNeededImpl()` joins the reader thread while holding `audioRecordStateLock` and never stops the `AudioRecord` first: 1. The reader takes that lock at the top of every loop iteration, so a stop racing a loop boundary deadlocks against the join until its 2s timeout (thread waits for the lock, join waits for the thread). Device logs showed this ~2s stall on every stop cycle under churn. 2. A reader stuck inside a blocking `read()` (stalled HAL) has nothing to unblock it, so the join times out structurally. 3. On timeout the record is released anyway — **under a live reader potentially still inside `read()`** — corrupting the platform client proxy (the abort above). The leaked thread then re-snapshots the shared `audioRecord` field each iteration and its exit path stops whatever the field holds, so once a new start installs a successor record, two readers interleave on one record. The crash tombstone showed two of this class's reader threads alive, started ~400ms apart. ## Fix One commit per hazard: 1. **Stop and detach the record before joining, join outside the lock** — a blocked `read()` returns once the record leaves the recording state, so the join actually succeeds; detaching under the lock means concurrent init/start builds fresh state. 2. **Never release under a live reader** — on join timeout, release ownership transfers to the reader via an atomic handoff (`orphanedRecord`/`cleanupDone`); exactly one side releases in every interleaving. 3. **Reader-side guards** — the reader re-checks `keepAlive` after the loop-top snapshot and around re-init/start (all under the lock), tracks the record it actually read (`activeRecord`) and stops only that on exit, takes the startup assert under the lock, and no longer misreports a shutdown-stopped record as a start failure (which permanently disabled `useAudioRecord`). ## Verification Stalled-read reproduction (JVM harness emulating a blocked `read()` that only returns when the record leaves the recording state), unpatched vs patched: - Unpatched: `release()` lands while a read is in flight — the abort's exact precondition — in 5/5 stop/restart cycles; every stop burns the full 2s join timeout. - Patched: every stop completes in milliseconds, reads observe a clean `stop()`, no release-under-read, no leaked reader threads. On-device stress (SM-G766U1, 400 init/start/stop cycles with a concurrent prewarm/init thread): unpatched **346/400** stops hit the 2s join timeout with leaked reader threads; patched **0/400**, max stop 189ms, no crash. The harness is Robolectric-based (shadowed `AudioRecord` source) so it doesn't fit this tree's GN build; happy to contribute it in whatever form fits your test setup. ## Notes - Bounded trade-offs: if a join still times out (pathologically stuck read), the orphaned record is released when the reader finally exits — if it never exits the record stays held, which is strictly better than the abort; in that same path the `AUDIO_RECORD_STOP` state callback can be delivered after a successor session's `AUDIO_RECORD_START` (it reflects the actual late stop). - `m146_release` carries the same stop-path code; happy to open a matching PR there. --------- Co-authored-by: davidliu <dl@livekit.io>
Upstream v2.28.2 moved to 144.7559.14. The fork takes 144.7559.15, the last m144 build, which adds livekit/libwebrtc#288 on top: prewarmRecording() and requestStartRecording() return false instead of throwing AssertionError when AudioRecord init failed. FastMatchSession prewarms the mic on every warm-up, so that path is live here. The other three .15 backports are iOS/macOS-only. Fixes relevant to fast-match that arrive with the .09 -> .15 delta: - livekit/libwebrtc#262: RTCStatsCollector no longer dereferences a disengaged optional for a stopped transceiver (the network-thread SIGABRT class the SDK worked around with metrics-poll and teardown sequencing). - livekit/libwebrtc#266: AudioRecord is no longer released under a live reader when the stop join times out on rapid capture stop/start cycles. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Problem
Production crash on Android (Samsung SM-G766U1, Android 16, LiveKit consumer of
io.github.webrtc-sdk:android-prefixed:144.7559.05) during rapid capture stop/start cycling while the capture HAL was under pressure (Bluetooth SCO route churn plus a second concurrent capture stack):stopRecordingIfNeededImpl()joins the reader thread while holdingaudioRecordStateLockand never stops theAudioRecordfirst:read()(stalled HAL) has nothing to unblock it, so the join times out structurally.read()— corrupting the platform client proxy (the abort above). The leaked thread then re-snapshots the sharedaudioRecordfield each iteration and its exit path stops whatever the field holds, so once a new start installs a successor record, two readers interleave on one record. The crash tombstone showed two of this class's reader threads alive, started ~400ms apart.Fix
One commit per hazard:
read()returns once the record leaves the recording state, so the join actually succeeds; detaching under the lock means concurrent init/start builds fresh state.orphanedRecord/cleanupDone); exactly one side releases in every interleaving.keepAliveafter the loop-top snapshot and around re-init/start (all under the lock), tracks the record it actually read (activeRecord) and stops only that on exit, takes the startup assert under the lock, and no longer misreports a shutdown-stopped record as a start failure (which permanently disableduseAudioRecord).Verification
Stalled-read reproduction (JVM harness emulating a blocked
read()that only returns when the record leaves the recording state), unpatched vs patched:release()lands while a read is in flight — the abort's exact precondition — in 5/5 stop/restart cycles; every stop burns the full 2s join timeout.stop(), no release-under-read, no leaked reader threads.On-device stress (SM-G766U1, 400 init/start/stop cycles with a concurrent prewarm/init thread): unpatched 346/400 stops hit the 2s join timeout with leaked reader threads; patched 0/400, max stop 189ms, no crash.
The harness is Robolectric-based (shadowed
AudioRecordsource) so it doesn't fit this tree's GN build; happy to contribute it in whatever form fits your test setup.Notes
AUDIO_RECORD_STOPstate callback can be delivered after a successor session'sAUDIO_RECORD_START(it reflects the actual late stop).m146_releasecarries the same stop-path code; happy to open a matching PR there.