Repository navigation
Conversation
Add the LiveKit packet trailer ("LKTS"), byte-compatible with
client-sdk-js and rust-sdks, so that the Android SDK can send and read
per-frame metadata.
api/frame_metadata:
- FrameMetadataTransformer writes trailers on send, and strips and
caches them on receive, including the AV1 metadata OBU path. The send
side only writes values supplied with each captured frame, found by
capture time in ms. It generates none itself.
- FrameTransformerChain runs any two transformers in the order given,
and can nest.
- BuildTrailer drops user data, or the whole trailer, if the XORed
bytes would form an H.264 start code.
Android:
- VideoFrame carries an optional FrameMetadata, and the RTP timestamp
of received frames.
- VideoSource links a source to its send transformers.
- Add RtpSender and RtpReceiver.setFrameTransformer(), FrameTransformer,
FrameTransformerChain, FrameMetadataTransformer and its factory.
- FrameCryptorFactory.createUninstalledFrameCryptor() creates a cryptor
without installing it, so that a cryptor and a metadata transformer
can be installed as one chain. The existing methods are unchanged.
RtpVideoStreamReceiver2: reset the old delegate when replacing a
transformer. Otherwise the old delegate and transformer keep each other
alive, which leaks them and, for a cryptor, its thread.
Nothing installs a transformer on its own and frame_crypto_transformer
is untouched, so the receive delegate fix is the only change that runs
in rust-sdks builds.
Tested with rtc_unittests, the RtpVideoStreamReceiver2 tests and an
arm64 debug AAR build. The Java and JNI code has not been run on a
device.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the LiveKit packet trailer (
LKTS) so the Android SDK can send and read per-frame metadata: a timestamp, a frame ID and user data. The wire format is byte-compatible with client-sdk-js and rust-sdks.Nothing is installed automatically; the SDK creates the transformers and installs them. Apart from the
RtpVideoStreamReceiver2fix below, none of this runs unless something calls the new APIs, so that fix is the only change rust-sdks builds pick up.Changes
api/frame_metadataFrameMetadataTransformerLKTS, and keeps the values by RTP timestamp (last 300 frames, cleared when the SSRC changes) for lookups from a video sink.metadata_type31) after the temporal delimiter and sequence header, as in rust-sdks.FrameTransformerChainruns two transformers in the order given, for example a cryptor and then the trailer on send. Chains can nest.BuildTrailerdrops the user data, or the whole trailer, if the XORed bytes would form an H.264 start code.Android
VideoFramecarries an optionalFrameMetadata, and received frames carry their RTP timestamp (getRtpTimestamp()).VideoProcessor.applyFrameAdaptationParameterskeeps the metadata.VideoSource.addFrameMetadataTransformerandremoveFrameMetadataTransformerlink a source to send transformers. The native source passes each captured frame's metadata to them.RtpSender.setFrameTransformer(),RtpReceiver.setFrameTransformer(), theFrameTransformerinterface,FrameTransformerChain,FrameMetadataTransformerandFrameMetadataTransformerFactory.FrameCryptorimplementsFrameTransformer.FrameCryptorFactory.createUninstalledFrameCryptor()creates a cryptor without installing it, so a cryptor and a metadata transformer can be installed as one chain. The existing factory methods are unchanged.RtpVideoStreamReceiver2Resets the old delegate when a transformer is replaced. Otherwise the old delegate and transformer keep each other alive. That leaks both, and for a cryptor, its thread too. With E2EE this already happens today when re-subscribing on a reused receiver. Covered by new tests.
Compatibility
CAP_PACKET_TRAILER.Testing
rtc_unittests: the new frame metadata tests and theRtpVideoStreamReceiver2tests.restartTrack, republishing, and full reconnects on either side🤖 Generated with Claude Code