Repository navigation
Add CustomAudioSource for pushing PCM frames from the application - #292
Draft
hiroshihorie wants to merge 4 commits into
Draft
hiroshihorie wants to merge 4 commits into
hiroshihorie wants to merge 4 commits into
Conversation
…source send streams Audio sources that deliver frames through their own AddSink path, bypassing the AudioDeviceModule, report SourceType::kCustom from source_type(). The default, kAudioDeviceModule, is the ADM path fed through AudioState. The type propagates from AudioSourceInterface through webrtc::AudioSource and LocalAudioSinkAdapter to the voice engine, which reconfigures the AudioSendStream so a kCustom stream never registers with AudioState. This keeps device-captured audio out of streams fed by a custom source. Config::ToString logs the type. The enumerator names follow RTCAudioSource::SourceType in the Windows/Linux libwebrtc API, except that the ADM value is named for the delivery path rather than kMicrophone, since any ADM device qualifies. Details worth knowing: - The type is evaluated on every SetSource call, not only on first attach. On replaceTrack the sink adapter object stays the same while the underlying source type may change, so a latched value would leave a device track silent (stream unregistered) or mix ADM audio into a custom track (stream still registered). - The stream is cycled Stop -> reconfigure -> restart across a type change so AudioState registration is always added and removed under the same value. Without this, a change while sending could leave a destroyed stream registered in AudioTransportImpl (use-after-free on the next device capture) or unregister a never-added stream. - StoreEncoderProperties also skips AudioState registration so a codec reconfigure mid-call cannot re-register a custom stream. - The sink adapter type is assigned unconditionally in AudioRtpSender::SetSend, so a sourceless replacement track resets it. It sits after the stopped_ guard in that method. Nothing in-tree overrides source_type() yet, so this commit is inert on its own.
A LocalAudioSource subclass with a real sink list that applications push interleaved int16 PCM into, bypassing the AudioDeviceModule. It reports SourceType::kCustom so AudioState never feeds device audio into the streams it drives. Supports two modes: queue_size_ms == 0 delivers exact 10 ms frames synchronously on the calling thread; queue_size_ms > 0 buffers pushed audio and paces 10 ms deliveries on a dedicated task queue, feeding silence on underrun and flushing a sub-frame residual padded with silence so it never sits in the buffer adding latency. Back-pressure is signaled through an optional per-push completion callback that fires once the buffer drains below the notify threshold, or by polling BufferedDurationMs(). Design notes: - Format is validated on every push in both modes and invalid constructor arguments make Create() return nullptr, keeping the validation in one place for wrappers. - Consumed samples are tracked with a read offset and reclaimed in one amortized pass, instead of memmoving the remaining buffer on every 10 ms tick under the lock. - The pacer delivers from a snapshot outside the state lock so pushers do not wait on per-sink work, and RemoveSink fences on a dedicated delivery mutex so a removed sink can be destroyed safely as soon as the call returns. Synchronous mode delivers under the state lock, which is what serializes concurrent pushers. - Completion callbacks run outside the locks, so a handler may push the next frame or remove a sink without deadlocking. - The destructor stops the pacer before teardown and flushes a pending completion instead of stranding the caller. - BufferedDurationMs multiplies before dividing so rates like 44100 report exactly.
Covers both modes with a simulated clock: synchronous delivery and 10 ms enforcement, argument and format validation, buffered pacing and ordering, silence on underrun, sub-frame residual flushing, capacity rejection, completion-callback back-pressure (inline, deferred, single in-flight, re-entrant push from the handler), ClearBuffer, exact duration reporting for 44.1 kHz rates, and destruction while the pacer is running. A real-thread test verifies RemoveSink blocks until an in-flight delivery finishes and that a removed sink receives nothing afterwards. Ships as its own rtc_cc_test target registered in the peerconnection_unittests suite, following the per-file split of that suite on M150.
RTCCustomAudioSource subclasses RTCAudioSource so it works with the existing audioTrackWithSource:trackId:. Frames are pushed as raw interleaved int16 NSData, AVAudioPCMBuffer (int16/float32, interleaved or not), or CMSampleBuffer carrying linear PCM (e.g. ReplayKit app audio). The completion handler surfaces the buffered-mode back-pressure signal (a rejected push discards it, documented), and bufferedDurationMs supports polling instead. Created via RTCPeerConnectionFactory customAudioSourceWithSampleRate:channels:queueSizeMs:, which relies on the native Create() for argument validation and returns nil on failure. sdk/objc/DEPS allows common_audio/include for the float to int16 conversion, matching the existing common_video/include rule.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a pushable audio source so applications can publish audio (app audio, screen-share audio, files) as independent tracks without mixing it into the microphone path.
Closes #210. Supersedes #258 and #282 (re-targeted to
m150_release).What this adds
AudioSourceInterface::SourceType { kAudioDeviceModule, kCustom }andsource_type().AudioSendStreamskips AudioState registration forkCustomstreams, so device capture is not also pushed into them. The type is re-evaluated on everySetSourceandSetSend, soreplaceTrackin either direction restores the correct path.kCustommatchesRTCAudioSource::SourceTypein the Windows/LinuxlibwebrtcAPI.webrtc::CustomAudioSource(pc/custom_audio_source.h): aLocalAudioSourcethe application pushes interleaved int16 PCM into. Synchronous mode delivers on the caller's thread. Buffered mode paces 10 ms frames from a task queue, feeds silence on underrun, and offers a completion callback for back-pressure.RTCCustomAudioSourceand-[RTCPeerConnectionFactory customAudioSourceWithSampleRate:channels:queueSizeMs:], accepting NSData, AVAudioPCMBuffer and CMSampleBuffer.Capture-side audio processing is bypassed for custom sources by design. Push audio that needs no such processing.
Review guide
Four commits, each builds and passes its tests on its own. The first is the only fork-wide behavior change and is inert until something overrides
source_type(), which nothing in-tree does until the second commit.Testing
custom_audio_source_unittest: 18/18 under a simulated clock, plus the real-threadRemoveSinkfence test 10/10 under--gtest_repeat.AudioSendStream*38/38,RtpSenderReceiver*108/108,LocalAudioSource*2/2.*WebRtcVoiceEngine*inrtc_media_unittests: failure set identical to them150_releasebaseline.gn checkclean on the touched targets.sdk:peerconnectionfactory_base_objcbuilds for macOS, and the ObjC files are syntax-checked against the iOS SDK.