Skip to content

Add CustomAudioSource for pushing PCM frames from the application - #292

Draft
hiroshihorie wants to merge 4 commits into
m150_releasefrom
hiroshi/custom-audio-source
Draft

hiroshihorie wants to merge 4 commits into
m150_releasefrom
hiroshi/custom-audio-source

Conversation

@hiroshihorie

Copy link
Copy Markdown
Member

Adds a pushable audio source so applications can publish audio (app audio, screen-share audio, files) as independent tracks without mixing it into the microphone path.

Closes #210. Supersedes #258 and #282 (re-targeted to m150_release).

What this adds

  • AudioSourceInterface::SourceType { kAudioDeviceModule, kCustom } and source_type(). AudioSendStream skips AudioState registration for kCustom streams, so device capture is not also pushed into them. The type is re-evaluated on every SetSource and SetSend, so replaceTrack in either direction restores the correct path. kCustom matches RTCAudioSource::SourceType in the Windows/Linux libwebrtc API.
  • webrtc::CustomAudioSource (pc/custom_audio_source.h): a LocalAudioSource the application pushes interleaved int16 PCM into. Synchronous mode delivers on the caller's thread. Buffered mode paces 10 ms frames from a task queue, feeds silence on underrun, and offers a completion callback for back-pressure.
  • ObjC RTCCustomAudioSource and -[RTCPeerConnectionFactory customAudioSourceWithSampleRate:channels:queueSizeMs:], accepting NSData, AVAudioPCMBuffer and CMSampleBuffer.

Capture-side audio processing is bypassed for custom sources by design. Push audio that needs no such processing.

Review guide

Four commits, each builds and passes its tests on its own. The first is the only fork-wide behavior change and is inert until something overrides source_type(), which nothing in-tree does until the second commit.

Testing

  • custom_audio_source_unittest: 18/18 under a simulated clock, plus the real-thread RemoveSink fence test 10/10 under --gtest_repeat.
  • AudioSendStream* 38/38, RtpSenderReceiver* 108/108, LocalAudioSource* 2/2.
  • *WebRtcVoiceEngine* in rtc_media_unittests: failure set identical to the m150_release baseline.
  • gn check clean on the touched targets. sdk:peerconnectionfactory_base_objc builds for macOS, and the ObjC files are syntax-checked against the iOS SDK.

…source send streams

Audio sources that deliver frames through their own AddSink path,
bypassing the AudioDeviceModule, report SourceType::kCustom from
source_type(). The default, kAudioDeviceModule, is the ADM path fed
through AudioState. The type propagates from AudioSourceInterface
through webrtc::AudioSource and LocalAudioSinkAdapter to the voice
engine, which reconfigures the AudioSendStream so a kCustom stream
never registers with AudioState. This keeps device-captured audio out
of streams fed by a custom source. Config::ToString logs the type.

The enumerator names follow RTCAudioSource::SourceType in the
Windows/Linux libwebrtc API, except that the ADM value is named for
the delivery path rather than kMicrophone, since any ADM device
qualifies.

Details worth knowing:
- The type is evaluated on every SetSource call, not only on first
  attach. On replaceTrack the sink adapter object stays the same while
  the underlying source type may change, so a latched value would leave
  a device track silent (stream unregistered) or mix ADM audio into a
  custom track (stream still registered).
- The stream is cycled Stop -> reconfigure -> restart across a type
  change so AudioState registration is always added and removed under
  the same value. Without this, a change while sending could leave a
  destroyed stream registered in AudioTransportImpl (use-after-free on
  the next device capture) or unregister a never-added stream.
- StoreEncoderProperties also skips AudioState registration so a codec
  reconfigure mid-call cannot re-register a custom stream.
- The sink adapter type is assigned unconditionally in
  AudioRtpSender::SetSend, so a sourceless replacement track resets it.
  It sits after the stopped_ guard in that method.

Nothing in-tree overrides source_type() yet, so this commit is inert
on its own.
A LocalAudioSource subclass with a real sink list that applications
push interleaved int16 PCM into, bypassing the AudioDeviceModule. It
reports SourceType::kCustom so AudioState never feeds device audio
into the streams it drives.

Supports two modes: queue_size_ms == 0 delivers exact 10 ms frames
synchronously on the calling thread; queue_size_ms > 0 buffers pushed
audio and paces 10 ms deliveries on a dedicated task queue, feeding
silence on underrun and flushing a sub-frame residual padded with
silence so it never sits in the buffer adding latency. Back-pressure
is signaled through an optional per-push completion callback that
fires once the buffer drains below the notify threshold, or by
polling BufferedDurationMs().

Design notes:
- Format is validated on every push in both modes and invalid
  constructor arguments make Create() return nullptr, keeping the
  validation in one place for wrappers.
- Consumed samples are tracked with a read offset and reclaimed in one
  amortized pass, instead of memmoving the remaining buffer on every
  10 ms tick under the lock.
- The pacer delivers from a snapshot outside the state lock so pushers
  do not wait on per-sink work, and RemoveSink fences on a dedicated
  delivery mutex so a removed sink can be destroyed safely as soon as
  the call returns. Synchronous mode delivers under the state lock,
  which is what serializes concurrent pushers.
- Completion callbacks run outside the locks, so a handler may push
  the next frame or remove a sink without deadlocking.
- The destructor stops the pacer before teardown and flushes a pending
  completion instead of stranding the caller.
- BufferedDurationMs multiplies before dividing so rates like 44100
  report exactly.
Covers both modes with a simulated clock: synchronous delivery and
10 ms enforcement, argument and format validation, buffered pacing
and ordering, silence on underrun, sub-frame residual flushing,
capacity rejection, completion-callback back-pressure (inline,
deferred, single in-flight, re-entrant push from the handler),
ClearBuffer, exact duration reporting for 44.1 kHz rates, and
destruction while the pacer is running. A real-thread test verifies
RemoveSink blocks until an in-flight delivery finishes and that a
removed sink receives nothing afterwards.

Ships as its own rtc_cc_test target registered in the
peerconnection_unittests suite, following the per-file split of that
suite on M150.
RTCCustomAudioSource subclasses RTCAudioSource so it works with the
existing audioTrackWithSource:trackId:. Frames are pushed as raw
interleaved int16 NSData, AVAudioPCMBuffer (int16/float32, interleaved
or not), or CMSampleBuffer carrying linear PCM (e.g. ReplayKit app
audio). The completion handler surfaces the buffered-mode back-pressure
signal (a rejected push discards it, documented), and
bufferedDurationMs supports polling instead.

Created via RTCPeerConnectionFactory
customAudioSourceWithSampleRate:channels:queueSizeMs:, which relies on
the native Create() for argument validation and returns nil on failure.

sdk/objc/DEPS allows common_audio/include for the float to int16
conversion, matching the existing common_video/include rule.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose RTCAudioSource custom source API on iOS/macOS (parity with Windows/Linux builds)

1 participant