Skip to content

Add packet trailer frame metadata transformer and Android bindings - #344

Open
davidliu wants to merge 1 commit into
m150_releasefrom
dl/video_frame_data
Open

davidliu wants to merge 1 commit into
m150_releasefrom
dl/video_frame_data

Conversation

@davidliu

Copy link
Copy Markdown
Contributor

Summary

Adds the LiveKit packet trailer (LKTS) so the Android SDK can send and read per-frame metadata: a timestamp, a frame ID and user data. The wire format is byte-compatible with client-sdk-js and rust-sdks.

Nothing is installed automatically; the SDK creates the transformers and installs them. Apart from the RtpVideoStreamReceiver2 fix below, none of this runs unless something calls the new APIs, so that fix is the only change rust-sdks builds pick up.

Changes

api/frame_metadata

  • FrameMetadataTransformer
    • Send: appends a trailer to each encoded frame, with the values supplied with its captured frame, found by capture time in ms. It doesn't generate any values itself.
    • Receive: strips the trailer from frames that end in LKTS, and keeps the values by RTP timestamp (last 300 frames, cleared when the SSRC changes) for lookups from a video sink.
    • AV1: the trailer goes in a metadata OBU (metadata_type 31) after the temporal delimiter and sequence header, as in rust-sdks.
    • When disabled, it passes frames through unchanged. On receive, that means trailers aren't stripped.
  • FrameTransformerChain runs two transformers in the order given, for example a cryptor and then the trailer on send. Chains can nest.
  • BuildTrailer drops the user data, or the whole trailer, if the XORed bytes would form an H.264 start code.

Android

  • VideoFrame carries an optional FrameMetadata, and received frames carry their RTP timestamp (getRtpTimestamp()). VideoProcessor.applyFrameAdaptationParameters keeps the metadata.
  • VideoSource.addFrameMetadataTransformer and removeFrameMetadataTransformer link a source to send transformers. The native source passes each captured frame's metadata to them.
  • New: RtpSender.setFrameTransformer(), RtpReceiver.setFrameTransformer(), the FrameTransformer interface, FrameTransformerChain, FrameMetadataTransformer and FrameMetadataTransformerFactory.
  • FrameCryptor implements FrameTransformer. FrameCryptorFactory.createUninstalledFrameCryptor() creates a cryptor without installing it, so a cryptor and a metadata transformer can be installed as one chain. The existing factory methods are unchanged.

RtpVideoStreamReceiver2

Resets the old delegate when a transformer is replaced. Otherwise the old delegate and transformer keep each other alive. That leaks both, and for a cryptor, its thread too. With E2EE this already happens today when re-subscribing on a reused receiver. Covered by new tests.

Compatibility

  • Same trailer format as client-sdk-js and rust-sdks.
  • AV1 matches rust-sdks. client-sdk-js appends the raw trailer even for AV1, so AV1 metadata doesn't cross between JS and Android or Rust.
  • The SFU strips trailers before sending to subscribers that don't advertise CAP_PACKET_TRAILER.

Testing

  • rtc_unittests: the new frame metadata tests and the RtpVideoStreamReceiver2 tests.
  • On a Pixel 6 (Android 16), the Android SDK's frame metadata branch with its instrumented tests against a LiveKit server, all passing:
    • VP8, H.264, VP9, and AV1 (L1T3)
    • E2EE with VP8 and H.264
    • user data at the size limit, oversized, and forming a start code
    • simulcast layer switches
    • VP9 SVC with a VP8 backup codec
    • restartTrack, republishing, and full reconnects on either side
  • Manually tested interop with client-sdk-js and rust-sdks across the 4 codecs and E2EE.

🤖 Generated with Claude Code

Add the LiveKit packet trailer ("LKTS"), byte-compatible with
client-sdk-js and rust-sdks, so that the Android SDK can send and read
per-frame metadata.

api/frame_metadata:
- FrameMetadataTransformer writes trailers on send, and strips and
  caches them on receive, including the AV1 metadata OBU path. The send
  side only writes values supplied with each captured frame, found by
  capture time in ms. It generates none itself.
- FrameTransformerChain runs any two transformers in the order given,
  and can nest.
- BuildTrailer drops user data, or the whole trailer, if the XORed
  bytes would form an H.264 start code.

Android:
- VideoFrame carries an optional FrameMetadata, and the RTP timestamp
  of received frames.
- VideoSource links a source to its send transformers.
- Add RtpSender and RtpReceiver.setFrameTransformer(), FrameTransformer,
  FrameTransformerChain, FrameMetadataTransformer and its factory.
- FrameCryptorFactory.createUninstalledFrameCryptor() creates a cryptor
  without installing it, so that a cryptor and a metadata transformer
  can be installed as one chain. The existing methods are unchanged.

RtpVideoStreamReceiver2: reset the old delegate when replacing a
transformer. Otherwise the old delegate and transformer keep each other
alive, which leaks them and, for a cryptor, its thread.

Nothing installs a transformer on its own and frame_crypto_transformer
is untouched, so the receive delegate fix is the only change that runs
in rust-sdks builds.

Tested with rtc_unittests, the RtpVideoStreamReceiver2 tests and an
arm64 debug AAR build. The Java and JNI code has not been run on a
device.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant