Skip to content

android: fix AudioRecord released under a live reader when the stop join times out - #266

Merged
davidliu merged 4 commits into
livekit:m144_releasefrom
KD6763:fix/audiorecord-stop-path-release-race
Aug 4, 2026
Merged

davidliu merged 4 commits into
livekit:m144_releasefrom
KD6763:fix/audiorecord-stop-path-release-race

Conversation

@KD6763

@KD6763 KD6763 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Problem

Production crash on Android (Samsung SM-G766U1, Android 16, LiveKit consumer of io.github.webrtc-sdk:android-prefixed:144.7559.05) during rapid capture stop/start cycling while the capture HAL was under pressure (Bluetooth SCO route churn plus a second concurrent capture stack):

Abort message: 'releaseBuffer: mUnreleased out of range,
!(stepCount:480 <= mUnreleased:0 <= mFrameCount:3840), BufferSizeInFrames:3840'
  android::ClientProxy::releaseBuffer
  android::AudioRecord::read
  android.media.AudioRecord.read
  livekit.org.webrtc.audio.WebRtcAudioRecord$AudioRecordThread.run

stopRecordingIfNeededImpl() joins the reader thread while holding audioRecordStateLock and never stops the AudioRecord first:

  1. The reader takes that lock at the top of every loop iteration, so a stop racing a loop boundary deadlocks against the join until its 2s timeout (thread waits for the lock, join waits for the thread). Device logs showed this ~2s stall on every stop cycle under churn.
  2. A reader stuck inside a blocking read() (stalled HAL) has nothing to unblock it, so the join times out structurally.
  3. On timeout the record is released anyway — under a live reader potentially still inside read() — corrupting the platform client proxy (the abort above). The leaked thread then re-snapshots the shared audioRecord field each iteration and its exit path stops whatever the field holds, so once a new start installs a successor record, two readers interleave on one record. The crash tombstone showed two of this class's reader threads alive, started ~400ms apart.

Fix

One commit per hazard:

  1. Stop and detach the record before joining, join outside the lock — a blocked read() returns once the record leaves the recording state, so the join actually succeeds; detaching under the lock means concurrent init/start builds fresh state.
  2. Never release under a live reader — on join timeout, release ownership transfers to the reader via an atomic handoff (orphanedRecord/cleanupDone); exactly one side releases in every interleaving.
  3. Reader-side guards — the reader re-checks keepAlive after the loop-top snapshot and around re-init/start (all under the lock), tracks the record it actually read (activeRecord) and stops only that on exit, takes the startup assert under the lock, and no longer misreports a shutdown-stopped record as a start failure (which permanently disabled useAudioRecord).

Verification

Stalled-read reproduction (JVM harness emulating a blocked read() that only returns when the record leaves the recording state), unpatched vs patched:

  • Unpatched: release() lands while a read is in flight — the abort's exact precondition — in 5/5 stop/restart cycles; every stop burns the full 2s join timeout.
  • Patched: every stop completes in milliseconds, reads observe a clean stop(), no release-under-read, no leaked reader threads.

On-device stress (SM-G766U1, 400 init/start/stop cycles with a concurrent prewarm/init thread): unpatched 346/400 stops hit the 2s join timeout with leaked reader threads; patched 0/400, max stop 189ms, no crash.

The harness is Robolectric-based (shadowed AudioRecord source) so it doesn't fit this tree's GN build; happy to contribute it in whatever form fits your test setup.

Notes

  • Bounded trade-offs: if a join still times out (pathologically stuck read), the orphaned record is released when the reader finally exits — if it never exits the record stays held, which is strictly better than the abort; in that same path the AUDIO_RECORD_STOP state callback can be delivered after a successor session's AUDIO_RECORD_START (it reflects the actual late stop).
  • m146_release carries the same stop-path code; happy to open a matching PR there.

KD6763 added 3 commits July 8, 2026 14:30
…read

stopRecordingIfNeededImpl joined the reader thread while holding
audioRecordStateLock and never stopped the AudioRecord first. Two
consequences:

- The reader acquires that lock at the top of every loop iteration, so a
  stop racing a loop boundary deadlocks against the join until its 2s
  timeout: the thread waits for the lock while the join waits for the
  thread. Observed in production as a ~2s stall on every capture stop
  cycle under churn.
- A reader stuck inside a blocking AudioRecord.read() (stalled capture
  HAL, e.g. Bluetooth SCO route churn) has nothing to unblock it, so the
  join times out structurally.

Stop the record inside the locked phase (a blocking read() returns
promptly once the record leaves the recording state), detach it from the
shared field so a concurrent init/start builds fresh state instead of
touching a record the stopping thread may still be reading, and perform
the join outside the lock. The stop entry points clear their request
flags under the lock and run the join outside it; a concurrent stop that
loses the race now returns cleanly instead of tripping the audioThread
assert.
When the stop path's join timed out it released the AudioRecord anyway,
while the abandoned reader thread could still be inside
AudioRecord.read() on that record. Releasing a record under an in-flight
read corrupts the platform client proxy accounting and aborts the
process:

  releaseBuffer: mUnreleased out of range,
  !(stepCount:480 <= mUnreleased:0 <= mFrameCount:3840)

On join timeout, transfer release responsibility to the reader through an
atomic handoff slot (orphanedRecord + cleanupDone): the reader releases
the record when it finally exits, and if it has already passed its
cleanup point during the handoff the stop path reclaims and releases it.
Exactly one side performs the release in every interleaving.
…state

With the join no longer serializing everything under audioRecordStateLock,
a reader that is being stopped could observe state belonging to the next
recording session and corrupt it:

- The loop-top snapshot re-reads the shared audioRecord field each
  iteration, so a reader that passed the while(keepAlive) check could
  adopt a successor record installed during the stop's join window, read
  it concurrently with the new reader, and stop it on exit. keepAlive is
  now re-checked against the snapshot (the stop path publishes
  keepAlive=false before detaching, under the same lock), and exit
  cleanup stops only the record this thread actually read (activeRecord),
  never the shared field.
- The startup preamble read the shared field twice without the lock and
  asserted on its recording state, which now races the stop's
  detach+stop: taken under the lock with a keepAlive guard.
- The mid-loop re-init leg could re-create a record after a stop had
  already finished, leaving a record behind that no stop path would ever
  release; the re-init, restart, and state verification now run in one
  critical section gated on a keepAlive re-check, and a record stopped by
  shutdown is no longer misreported as a start failure (which permanently
  disabled useAudioRecord).
- A read unblocked by the stop path's own AudioRecord.stop() exits
  through the loop condition instead of logging a spurious read error.
@davidliu
davidliu self-requested a review July 14, 2026 16:25
@davidliu

Copy link
Copy Markdown
Contributor

@KD6763 Hey, thanks for the PR. Just making sure, this is the same changes as #264?

@KD6763

KD6763 commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

@davidliu, yes same changes as #264

@davidliu davidliu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

If you'd be willing to add the test harness in a separate PR, that'd be helpful too. The tests at sdk/android/tests/src/org/webrtc/audio are Robolectric capable (uses AndroidJUnit4 as its runner).

You'd just need to add the path to the file in the BUILD.gn here:
https://github.com/webrtc-sdk/webrtc/blob/08b9408717876048bca582530ba6f3e732a02479/sdk/android/BUILD.gn#L1914

@KD6763

KD6763 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Thanks! Will do - I'll port the harness to a Robolectric test under sdk/android/tests/src/org/webrtc/audio and put up a separate PR for it.

Also do you want a matching PR for m146_release (it has the same stop path code), or would you rather pick it across on your side?

@davidliu

davidliu commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Also do you want a matching PR for m146_release (it has the same stop path code), or would you rather pick it across on your side?

We'll handle it on our side, we're not using m146_release actually, we're in the midst of an upgrade to m150.

@davidliu
davidliu merged commit 6eae56d into livekit:m144_release Aug 4, 2026
cloudwebrtc pushed a commit that referenced this pull request Aug 5, 2026
…oin times out (#266)

## Problem

Production crash on Android (Samsung SM-G766U1, Android 16, LiveKit
consumer of `io.github.webrtc-sdk:android-prefixed:144.7559.05`) during
rapid capture stop/start cycling while the capture HAL was under
pressure (Bluetooth SCO route churn plus a second concurrent capture
stack):

```
Abort message: 'releaseBuffer: mUnreleased out of range,
!(stepCount:480 <= mUnreleased:0 <= mFrameCount:3840), BufferSizeInFrames:3840'
  android::ClientProxy::releaseBuffer
  android::AudioRecord::read
  android.media.AudioRecord.read
  livekit.org.webrtc.audio.WebRtcAudioRecord$AudioRecordThread.run
```

`stopRecordingIfNeededImpl()` joins the reader thread while holding
`audioRecordStateLock` and never stops the `AudioRecord` first:

1. The reader takes that lock at the top of every loop iteration, so a
stop racing a loop boundary deadlocks against the join until its 2s
timeout (thread waits for the lock, join waits for the thread). Device
logs showed this ~2s stall on every stop cycle under churn.
2. A reader stuck inside a blocking `read()` (stalled HAL) has nothing
to unblock it, so the join times out structurally.
3. On timeout the record is released anyway — **under a live reader
potentially still inside `read()`** — corrupting the platform client
proxy (the abort above). The leaked thread then re-snapshots the shared
`audioRecord` field each iteration and its exit path stops whatever the
field holds, so once a new start installs a successor record, two
readers interleave on one record. The crash tombstone showed two of this
class's reader threads alive, started ~400ms apart.

## Fix

One commit per hazard:

1. **Stop and detach the record before joining, join outside the lock**
— a blocked `read()` returns once the record leaves the recording state,
so the join actually succeeds; detaching under the lock means concurrent
init/start builds fresh state.
2. **Never release under a live reader** — on join timeout, release
ownership transfers to the reader via an atomic handoff
(`orphanedRecord`/`cleanupDone`); exactly one side releases in every
interleaving.
3. **Reader-side guards** — the reader re-checks `keepAlive` after the
loop-top snapshot and around re-init/start (all under the lock), tracks
the record it actually read (`activeRecord`) and stops only that on
exit, takes the startup assert under the lock, and no longer misreports
a shutdown-stopped record as a start failure (which permanently disabled
`useAudioRecord`).

## Verification

Stalled-read reproduction (JVM harness emulating a blocked `read()` that
only returns when the record leaves the recording state), unpatched vs
patched:

- Unpatched: `release()` lands while a read is in flight — the abort's
exact precondition — in 5/5 stop/restart cycles; every stop burns the
full 2s join timeout.
- Patched: every stop completes in milliseconds, reads observe a clean
`stop()`, no release-under-read, no leaked reader threads.

On-device stress (SM-G766U1, 400 init/start/stop cycles with a
concurrent prewarm/init thread): unpatched **346/400** stops hit the 2s
join timeout with leaked reader threads; patched **0/400**, max stop
189ms, no crash.

The harness is Robolectric-based (shadowed `AudioRecord` source) so it
doesn't fit this tree's GN build; happy to contribute it in whatever
form fits your test setup.

## Notes

- Bounded trade-offs: if a join still times out (pathologically stuck
read), the orphaned record is released when the reader finally exits —
if it never exits the record stays held, which is strictly better than
the abort; in that same path the `AUDIO_RECORD_STOP` state callback can
be delivered after a successor session's `AUDIO_RECORD_START` (it
reflects the actual late stop).
- `m146_release` carries the same stop-path code; happy to open a
matching PR there.

---------

Co-authored-by: davidliu <dl@livekit.io>
sabahattinseven added a commit to sabahattinseven/client-sdk-android-fast-match that referenced this pull request Oct 3, 2026
Upstream v2.28.2 moved to 144.7559.14. The fork takes 144.7559.15, the last
m144 build, which adds livekit/libwebrtc#288 on top: prewarmRecording() and
requestStartRecording() return false instead of throwing AssertionError when
AudioRecord init failed. FastMatchSession prewarms the mic on every warm-up,
so that path is live here. The other three .15 backports are iOS/macOS-only.

Fixes relevant to fast-match that arrive with the .09 -> .15 delta:
- livekit/libwebrtc#262: RTCStatsCollector no longer dereferences a disengaged
  optional for a stopped transceiver (the network-thread SIGABRT class the SDK
  worked around with metrics-poll and teardown sequencing).
- livekit/libwebrtc#266: AudioRecord is no longer released under a live reader
  when the stop join times out on rapid capture stop/start cycles.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants