Repository navigation
feat: report the media start and end of LiveKit recordings - #51
Merged
Merged
Conversation
prlanzarin
force-pushed
the
bwr/feat/new-ts
branch
from
November 6, 2026 12:56
3b075e5 to
588ed6a
Compare
record-and-playback places a WebM's first frame at the time of the recording's start event. The controller stamps that event when it handles the Redis message. That can be seconds before a camera's first key frame. The recorder now takes each packet with an arrival time. A caller passes when it read the packet, before its jitter buffer held it. MediaSpan returns where the written media sits on the wall clock. Start is the arrival of the first frame written, from its earliest packet. End is Start plus the media written after that frame, to the end of the last frame. A repeated timestamp, as after an Opus step back, gets its own arrival. The first reads can find frames that waited in the receive buffers, and a camera can start behind and catch up. So a frame that arrived sooner than its PTS implies moves Start back. It moves at most 1 s, since bbb-webrtc-sfu ignores a start more than 1 s before its start request. Only frames in a file's first 2 s of media, audio or video, move Start. So a microphone and a camera on one path both drop a delay that clears within 2 s of their first frames. A step over 120 ms between audio frames ends this early, so a gap MediaClock inserts cannot move Start. Video has no such step, since a camera that catches up sends frames up to 700 ms apart.
The jitter buffer keeps each packet's arrival time but returns only the packets. So the recorder takes them as arriving when the buffer releases them, up to 2 s late for video. Nothing tells the session when the media starts. The track buffer now releases each packet with its arrival, and the read loop passes it to the recorder. After each write and on each watchFlow tick, the adapter reads the recorder's media span. It sends the start to a new media start callback once: when the recorder settles it, or after a 4.5 s timeout. The timeout covers a track that pauses inside the recorder's window and writes no frame to settle the start. It serves audio and video alike. The callback goes through the flow callback's queue, after the first flowing event, so the read loop does not wait on it. The adapter reads the span with its lock released, as the recorder takes that lock while holding its own.
bbb-webrtc-sfu stamps a recording's start and stop events with its own clock, when it handles the recorder's flowing status and recordingStopped. The recorder's event timestamps hold the time the event was built and a duration, so they cannot replace it. record-and-playback places each file's first frame at the start event's timestamp, the host's CLOCK_MONOTONIC in ms. The recorder now sends the adapter's media span in that clock and in epoch ms, so the controller can write those times instead. recordingMediaStarted carries the start once it is settled or 4.5 s old. recordingStopped adds the start and end, for a start event the controller missed, read after Close writes the frames the adapter still holds. getRecordings entries add the start. recorderStatus lists the mediaTimestamps capability, so a controller knows to wait for recordingMediaStarted. Go does not expose a time.Time's monotonic reading, so MonotonicMs reads CLOCK_MONOTONIC next to a time.Now() and adds a time's offset from that pair.
No scenario checks the media timestamps in a recording's start and stop events. record-and-playback plays a file's first frame at its start event's time. A start taken from the flowing event shifts the whole file by the time from that event to the first frame written. TestMediaTimestampsPlaceEachFrame records a camera and a microphone from one publisher, which writes each frame's index into its payload. The test reads the index from each WebM block, so it needs no mapping of RTP timestamps across LiveKit. The first key frame the recorder gets loses a packet, so the camera's file must start after the flowing event. TestMediaTimestampsTakeOffAStartDelay starts both recordings while netem delays the host's path to the servers by 300 ms. The delay clears within each track's first 2 s of media, so both starts must drop it. It uses the NACK scenario's namespaces, so it needs root; CI runs it in the root step. In both, every block must land within 40 ms of its send time, and the two files within 15 ms of each other. A frame's send time is when its first packet was due, so one late wake of the send loop does not count against the recorder. Both also check the span in recordingStopped, and the first checks mediaStartHR against the host's CLOCK_MONOTONIC.
The corpus replays its seeds through the read loop and the writer, but nothing checks the media span those runs produce. A span that took a frame's write time instead of its arrival would pass every expect file. Each single-track replay now checks the span against its trace. It must cover the file's media and start within 1 ms of the read of the first frame written. For Opus, that read is the first frame's, less the backlog the lag oracle finds. For VP8, it is the earliest read of the first key frame whose packets are all read. A seed pins its own start in mediaStartUs when its jitter buffer drops that frame. So does a seed whose later frames, inside a camera's 2 s window, arrived sooner than their PTS imply.
The media timestamps scenario sends frames on a schedule the test generates. A browser's capture, encoders and pacing delay its audio and video differently. Only a real one shows whether placing each file by its first frame's arrival keeps them in sync. TestBrowserMediaTimestampsKeepLipSync records Chromium's camera and microphone, published as BBB's client publishes them. Chromium's sender reports map both tracks to one capture clock, which on the recorder's host is the wall clock. The two files must play their media equally late, and every block as late as its file's first. It runs once per camera whose first frames Chromium sends late. The cameras are BBB's low profile, the hidden low-u30 profile at 30 kbps, and low captured at 5 fps.
The protocol docs call timestampHR and startTimeHR monotonic system time, and timestampUTC and startTimeUTC a frame's wall clock time. Next to the CLOCK_MONOTONIC mediaStartHR and mediaEndHR, that reads as if the older fields could serve as a recording event's timestamp. They cannot. timestampUTC is the time the event was built, as an RFC 3339 string, and startTimeUTC the time of the first flowing event. timestampHR and startTimeHR hold durations or media times, which differ by adapter. The docs now give each field's content per adapter and say none of these fields is a CLOCK_MONOTONIC time.
The e2e scenarios publish every camera as one VP8 stream. BBB's client publishes a camera as simulcast: with the default profile, q at 451x338 and h at 640x480. LiveKit forwards one layer to the recorder. On a switch, it rewrites the new layer's timestamps through the publisher's sender reports. No scenario checks that the media timestamps still place the frames after a switch. TestSimulcastMediaTimestampsPlaceEachFrame publishes those two layers and a microphone, and stops h for 3 s midway, so LiveKit moves the recorder to q and back. Every block must land within 40 ms of when its frame was sent, whichever layer it came from. The publisher gains PublishVP8Simulcast, with a frame tag per layer so the recording shows which layer a frame came from. A track now ignores RTCP for other SSRCs, since pion hands a compound packet to every SSRC it names.
The browser scenarios publish one VP8 stream per camera. BBB's client publishes the default camera profile as simulcast, with dynacast on. So no browser scenario sees a layer switch, or a camera that dynacast paused before recording starts. TestBrowserSimulcastMediaTimestamps publishes the default profile's layers. Midway, the page stops sending h. The test waits for LiveKit to ask h for a key frame, which shows LiveKit took h back. The file must hold both switches and pass the lip sync and recordingStopped checks. LiveKit can forward one captured frame twice at a switch up, so two blocks may share a PTS only if they hold that frame. A second case starts recording once dynacast has paused the unwatched camera.
prlanzarin
force-pushed
the
bwr/feat/new-ts
branch
from
November 9, 2026 14:16
588ed6a to
f58970f
Compare
This was referenced Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
record-and-playback plays each recorded file's first frame at the
timestampof its start event, the host's CLOCK_MONOTONIC in ms, and the rest by the file's own timestamps. For LiveKit recordings, bbb-webrtc-sfu stamps that event with its own clock when it handles the recorder'sflowingstatus. That can be far from the file's first frame, for example when a camera waits for its first key frame.The recorder now measures where each file's media sits on the wall clock, from the arrival of the frames it writes, taken before the jitter buffer. The start is the first frame's arrival. When the first frames reached the recorder late and then caught up, the start moves back to where they should have played, by up to 1 s. It sends the start and the end of the media in epoch ms and in CLOCK_MONOTONIC ms:
recordingMediaStarted, a new message, carries the start once it is settled or 4.5 s old.recordingStoppedadds the start and the end;getRecordingsentries add the start.recorderStatuslists themediaTimestampscapability, so a controller knows to wait forrecordingMediaStarted.The older fields keep their form, so older controllers behave as before. bbb-webrtc-sfu needs a matching change to write the new times into the recording events.
The tests check where every frame plays.
In detail:
How to test
TestMediaTimestampsPlaceEachFrameandTestBrowserMediaTimestampsKeepLipSynclog where a start stamped onflowingwould put each file's first frame, next to where the recorder's media start puts it.TestMediaTimestampsTakeOffAStartDelayneeds root (tests/e2e/README.md).E2E_RECORDER_BINpointing at a recorder built from develop, the media timestamp scenarios fail: it sends norecordingMediaStarted.Changelog
Technical — LiveKit recordings report when their media starts and ends, in the recording events' clock, for the controller to stamp start and stop events with.
User-facing — None on its own; with a matching bbb-webrtc-sfu, recorded cameras and microphones stay in sync with each other and with the rest of the recording more reliably, low-quality cameras included.