Skip to content

Add FlexFEC negotiation and subscriber FEC protection - #1071

Draft
cloudwebrtc wants to merge 3 commits into
mainfrom
duan/flexfec
Draft

cloudwebrtc wants to merge 3 commits into
mainfrom
duan/flexfec

Conversation

@cloudwebrtc

Copy link
Copy Markdown
Contributor

Note

Draft until livekit/protocol#1987 is merged. The protocol submodule currently points at duan/fec-subscribe-presets-android (3a115c89), which is #1987 cherry-picked onto the protocol commit main pins (30d320b7). Once #1987 lands, the submodule should be moved to the merged commit.

Adds FlexFEC negotiation and per-track subscriber FEC protection.

FlexFEC negotiation

LiveKit.enableFlexFec turns on the WebRTC-FlexFEC-03-Advertised and WebRTC-FlexFEC-03 field trials when libwebrtc is initialized, so flexfec-03 is negotiated. It is process-wide and must be set before the first Room is created. Disabled by default.

Subscriber FEC protection

RemoteTrackPublication.setFecProtection asks the server to send FlexFEC repair packets for a subscribed video track through UpdateTrackSettings.fec:

LiveKit.enableFlexFec = true
// ...
publication.setFecProtection(FecProtection.Minimum(FecProtectionLevel.MEDIUM))

FecProtection is a sealed class mirroring the protocol modes:

Kotlin Protocol
FecProtection.Disabled (default) FEC_MODE_DISABLED
FecProtection.Adaptive FEC_MODE_ADAPTIVE
FecProtection.Fixed(level) FEC_MODE_FIXED + level
FecProtection.Minimum(level) FEC_MODE_MINIMUM + level

FecProtectionLevel is LOW / MEDIUM / HIGH (about 15/25/35% repair overhead). Only the modes that use a level carry one, so invalid combinations cannot be expressed.

  • fec is only sent after setFecProtection has been called. That follows the optional semantics in #1987: omitting the field keeps the server's setting, so other settings updates do not reset it.
  • Unlike quality and dimensions, it also applies while adaptive stream manages the track.
  • The change takes effect without renegotiation.

Tests

  • Unit tests in RemoteTrackPublicationTest: fec is omitted until requested, it is kept on later updates, every mode and level maps to the right proto values, and it is sent under adaptive stream.
  • FlexFecOnDeviceTest (androidTest): subscribes on a real server and checks the repairs received in each mode through the inbound-rtp FEC counters. It is skipped unless the lkUrl and lkToken instrumentation arguments are given. On-device results are in the comment below.

LiveKit.enableFlexFec turns on the FlexFEC field trials when libwebrtc is
initialized, so flexfec-03 is negotiated. Codec preferences already keep
unmatched codecs, so the repair codec is offered without further changes.

RemoteTrackPublication.setFecProtection requests FlexFEC repair packets from
the server for a subscribed video track through the new
UpdateTrackSettings.fec field, which requires livekit/protocol#1987.
FecProtection is a sealed class mirroring the protocol's modes: Disabled,
Adaptive, and Fixed and Minimum, which carry a FecProtectionLevel (low,
medium or high). The protection is only sent once requested, so other
settings updates leave the server's setting unchanged. Unlike quality and
dimensions it also applies while adaptive stream manages the track.

FlexFecOnDeviceTest subscribes to a video track on a real server and checks
the repairs received for each mode through the inbound-rtp FEC counters. It
is skipped unless the lkUrl and lkToken instrumentation arguments are given.

The protocol submodule points at livekit/protocol#1987 applied on top of the
current protocol commit.
@changeset-bot

changeset-bot Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 8a1a900

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
client-sdk-android Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@cloudwebrtc

cloudwebrtc commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

End-to-end test results

Ran FlexFecOnDeviceTest on a real device against a local livekit-server built from source.

Setup

Server livekit/livekit#5195 (branch duan/flexfec-downstream) @ cc316561 (Support downstream FEC protection modes). The PR's later commit 2f17766b only points go.mod at livekit/protocol#1987, which the test build already used through go.work.
Protocol livekit/protocol#1987 head de2bab40, used by the server through go.work; this PR's submodule is the same change on 30d320b7
Publisher lk room join --publish-demo (simulcast video)
Subscriber Xiaomi M367FC, Android 17, over Wi-Fi; LiveKit.enableFlexFec = true, adaptiveStream = false, highest layer subscribed
libwebrtc io.github.webrtc-sdk:android-prefixed:150.7871.04, the version this PR builds with

Server config:

rtc:
  flexfec:
    upstream_enabled: true
    downstream_enabled: true

Run with:

./gradlew :livekit-android-test:connectedDebugAndroidTest \
  -Pandroid.testInstrumentationRunnerArguments.class=io.livekit.android.room.track.FlexFecOnDeviceTest \
  -Pandroid.testInstrumentationRunnerArguments.lkUrl=ws://<server-ip>:7880 \
  -Pandroid.testInstrumentationRunnerArguments.lkToken=<subscriber token>

The server's own FlexFEC tests (go test ./pkg/sfu/ ./pkg/rtc/ -run 'FEC|Fec|FlexFEC' and go test ./test/ -run FlexFEC) also pass at that commit.

Results

Each row is a 6 s window, after a 2 s settle following setFecProtection. Counts come from the subscriber's video inbound-rtp stats (packetsReceived, fecPacketsReceived, fecPacketsDiscarded).

Protection Media packets FEC packets FEC / media FEC discarded
not requested 1083 0 0 0
Fixed(LOW) 1156 191 16.5% 0
Fixed(MEDIUM) 1176 219 18.6% 0
Fixed(HIGH) 1311 330 25.2% 0
Minimum(MEDIUM) 1230 225 18.3% 0
Adaptive 1072 0 0 0
Disabled 971 0 0 0

An earlier run gave the same picture (LOW 17.1%, HIGH 25.1%, Minimum MEDIUM 18.6%, Adaptive and Disabled 0).

What this shows:

  • flexfec-03 is negotiated, and the repairs are received and accepted by libwebrtc (none discarded).
  • No repairs are sent until FEC is requested.
  • Mode changes apply right away without renegotiation, and Disabled stops the repairs.
  • Without loss, Minimum(MEDIUM) matches Fixed(MEDIUM) and Adaptive sends no repairs.

Note on the ratios

Fixed(HIGH) measures about 25%, not 35%. This is expected from the server's per-frame NumFecPackets rounding, max(1, round(n * factor / 256)) with factor 89 for HIGH: a 4-packet frame gets 1 repair (25%) and a 7-packet frame gets 2 (29%). The nominal 15/25/35% is only reached for larger frames. The test therefore checks ordering and loose lower bounds rather than the nominal percentages.

libwebrtc version

WebRTC Android 150.7871.06 has not been released yet; 150.7871.04 is the latest on Maven Central. webrtc-sdk/webrtc#329 (RtpParameters::fec_protection) was merged into m150_release after .04, so it is expected to ship in .06. This PR and the results above use .04: negotiating flexfec-03 through the field trials and receiving the server's repairs do not depend on #329. Once .06 is published, the dependency should be bumped and the on-device test rerun.

Not covered

No loss was simulated, so recovery of lost packets and adaptive protection rising with loss were not exercised on device. The server's test/flexfec_downstream_test.go covers both.

Upstream (publish) results

Ran FlexFecPublishOnDeviceTest (added in 8a1a900) on the same device and server, publishing a synthetic 640x480 30 fps video track with LiveKit.enableFlexFec = true.

Negotiation

Without simulcast With simulcast (SDK default)
Publisher offer flexfec-03 and a=ssrc-group:FEC-FR no FlexFEC SSRC
Server answer accepts flexfec-03 n/a

libwebrtc only protects a single media stream with FlexFEC (media_session.cc: "Our FlexFEC implementation only supports protecting a single media stream"), so tracks published with simulcast do not send FlexFEC. The LiveKit.enableFlexFec KDoc now says so (c91fdee).

With .04 the publisher uses libwebrtc's loss-based FEC rate, which corresponds to FEC_MODE_ADAPTIVE; no repairs are sent without loss. Publish-side fixed levels need webrtc-sdk/webrtc#329, i.e. .06.

Under loss

Upstream loss was added with a userspace UDP proxy between the device and the server that drops only RTP. With 5% and 20% upstream loss the server's livekit_fec_total{direction="incoming"} shows FlexFEC packets received and lost packets recovered from them.

While doing this I found that the server's receiver reports to the publisher always carried fractionLost 0: packets restored by RTX or FEC cancelled the loss, so libwebrtc's loss-based FEC backed off. libwebrtc receivers report loss before repair. I have a local server fix for this on top of livekit/livekit#5195 (not pushed yet) that counts repaired packets as lost in the RR. 20% upstream loss, 30 s publish, without simulcast:

Before fix After fix
fractionLost in RR seen by publisher 0 ~0.15
FEC packets received by server 64 204
Packets recovered from FEC 11 61
Retransmissions 261 221

Run with:

./gradlew :livekit-android-test:connectedDebugAndroidTest \
  -Pandroid.testInstrumentationRunnerArguments.class=io.livekit.android.room.track.FlexFecPublishOnDeviceTest \
  -Pandroid.testInstrumentationRunnerArguments.lkUrl=ws://<server-ip>:7880 \
  -Pandroid.testInstrumentationRunnerArguments.lkPublisherToken=<publisher token> \
  -Pandroid.testInstrumentationRunnerArguments.lkPublishSeconds=30

@github-actions

Copy link
Copy Markdown
Contributor

Diffuse output:

OLD: diffuse-source-file
NEW: livekit-android-sdk-release.aar

 AAR      │ old      │ new      │ diff      
──────────┼──────────┼──────────┼───────────
      jar │  3.1 MiB │  3.1 MiB │ +20.6 KiB 
 manifest │  1.5 KiB │  1.5 KiB │       0 B 
 lint-jar │ 12.7 KiB │ 12.7 KiB │       0 B 
    other │  2.4 KiB │  2.4 KiB │       0 B 
──────────┼──────────┼──────────┼───────────
    total │  3.1 MiB │  3.2 MiB │ +20.6 KiB 

 JAR     │ old   │ new   │ diff             
─────────┼───────┼───────┼──────────────────
 classes │  1794 │  1810 │  +16 (+16 -0)    
 methods │ 22661 │ 22820 │ +159 (+562 -403) 
  fields │  5984 │  6033 │  +49 (+49 -0)
AAR
 size    │ diff      │ path          
─────────┼───────────┼───────────────
 3.1 MiB │ +20.6 KiB │ ∆ classes.jar 
─────────┼───────────┼───────────────
 3.1 MiB │ +20.6 KiB │ (total)
JAR
CLASSES:

   old  │ new  │ diff         
  ──────┼──────┼──────────────
   1794 │ 1810 │ +16 (+16 -0) 
  
  + io.livekit.android.room.track.FecProtection_Adaptive
  + io.livekit.android.room.track.FecProtection_Disabled
  + io.livekit.android.room.track.FecProtection_Fixed
  + io.livekit.android.room.track.FecProtection_Minimum
  + io.livekit.android.room.track.FecProtection
  + io.livekit.android.room.track.FecProtectionLevel_WhenMappings
  + io.livekit.android.room.track.FecProtectionLevel
  + livekit.LivekitRtc_FECProtection_Builder
  + livekit.LivekitRtc_FECProtection
  + livekit.LivekitRtc_FECProtectionLevel_1
  + livekit.LivekitRtc_FECProtectionLevel_FECProtectionLevelVerifier
  + livekit.LivekitRtc_FECProtectionLevel
  + livekit.LivekitRtc_FECProtectionMode_1
  + livekit.LivekitRtc_FECProtectionMode_FECProtectionModeVerifier
  + livekit.LivekitRtc_FECProtectionMode
  + livekit.LivekitRtc_FECProtectionOrBuilder
  

METHODS:

   old   │ new   │ diff             
  ───────┼───────┼──────────────────
   22661 │ 22820 │ +159 (+562 -403) 
  
  + io.livekit.android.LiveKit getEnableFlexFec() → boolean
  + io.livekit.android.LiveKit getEnableFlexFec_annotations()
  + io.livekit.android.LiveKit setEnableFlexFec(boolean)
  + io.livekit.android.dagger.RTCModule access_getFieldTrials(RTCModule) → String
  + io.livekit.android.dagger.RTCModule getFieldTrials() → String
  + io.livekit.android.room.SignalClient sendUpdateTrackSettings(String, boolean, Track_Dimensions, LivekitModels_VideoQuality, Integer, LivekitRtc_FECProtection)
  + io.livekit.android.room.SignalClient sendUpdateTrackSettings_default(SignalClient, String, boolean, Track_Dimensions, LivekitModels_VideoQuality, Integer, LivekitRtc_FECProtection, int, Object)
  + io.livekit.android.room.track.FecProtection <init>()
  + io.livekit.android.room.track.FecProtection <init>(DefaultConstructorMarker)
  + io.livekit.android.room.track.FecProtection toProto_livekit_android_sdk_release() → LivekitRtc_FECProtection
  + io.livekit.android.room.track.FecProtection_Adaptive <clinit>()
  + io.livekit.android.room.track.FecProtection_Adaptive <init>()
  + io.livekit.android.room.track.FecProtection_Adaptive equals(Object) → boolean
  + io.livekit.android.room.track.FecProtection_Adaptive hashCode() → int
  + io.livekit.android.room.track.FecProtection_Adaptive toString() → String
  + io.livekit.android.room.track.FecProtection_Disabled <clinit>()
  + io.livekit.android.room.track.FecProtection_Disabled <init>()
  + io.livekit.android.room.track.FecProtection_Disabled equals(Object) → boolean
  + io.livekit.android.room.track.FecProtection_Disabled hashCode() → int
  + io.livekit.android.room.track.FecProtection_Disabled toString() → String
  + io.livekit.android.room.track.FecProtection_Fixed <init>(FecProtectionLevel)
  + io.livekit.android.room.track.FecProtection_Fixed component1() → FecProtectionLevel
  + io.livekit.android.room.track.FecProtection_Fixed copy(FecProtectionLevel) → FecProtection_Fixed
  + io.livekit.android.room.track.FecProtection_Fixed copy_default(FecProtection_Fixed, FecProtectionLevel, int, Object) → FecProtection_Fixed
  + io.livekit.android.room.track.FecProtection_Fixed equals(Object) → boolean
  + io.livekit.android.room.track.FecProtection_Fixed getLevel() → FecProtectionLevel
  + io.livekit.android.room.track.FecProtection_Fixed hashCode() → int
  + io.livekit.android.room.track.FecProtection_Fixed toString() → String
  + io.livekit.android.room.track.FecProtection_Minimum <init>(FecProtectionLevel)
  + io.livekit.android.room.track.FecProtection_Minimum component1() → FecProtectionLevel
  + io.livekit.android.room.track.FecProtection_Minimum copy(FecProtectionLevel) → FecProtection_Minimum
  + io.livekit.android.room.track.FecProtection_Minimum copy_default(FecProtection_Minimum, FecProtectionLevel, int, Object) → FecProtection_Minimum
  + io.livekit.android.room.track.FecProtection_Minimum equals(Object) → boolean
  + io.livekit.android.room.track.FecProtection_Minimum getLevel() → FecProtectionLevel
  + io.livekit.android.room.track.FecProtection_Minimum hashCode() → int
  + io.livekit.android.room.track.FecProtection_Minimum toString() → String
  + io.livekit.android.room.track.FecProtectionLevel _values() → FecProtectionLevel[]
  + io.livekit.android.room.track.FecProtectionLevel <clinit>()
  + io.livekit.android.room.track.FecProtectionLevel <init>(String, int)
  + io.livekit.android.room.track.FecProtectionLevel getEntries() → EnumEntri
...✂

libwebrtc only protects a single media stream with FlexFEC, so it signals no
FlexFEC SSRC for a sender with simulcast layers.
FlexFecPublishOnDeviceTest publishes a synthetic video track to a real server
and checks the publisher negotiation: without simulcast the offer includes
flexfec-03 and a FEC-FR SSRC and the server accepts flexfec-03, and with
simulcast no FlexFEC SSRC is signaled. The track can be held for a while to
watch the repairs arrive at the server under upstream loss. It is skipped
unless the lkUrl and lkPublisherToken instrumentation arguments are given.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant