Repository navigation
test: check where every frame plays under network faults - #52
Merged
prlanzarin merged 27 commits intoNov 10, 2026
Merged
Conversation
The e2e publisher writes a tag into each frame's first packet, so a test can tell which frame each WebM block holds. The helpers live in tests/e2e behind the e2e build tag, so the in-process replay cannot use them. Its VP8 oracle has to match blocks to frames by order. Move the helpers to internal/corpus so the e2e suite and the replay share one tag format. The VP8 picture ID reader and setter move too, for operators that renumber frames. VP8PictureID now also returns the ID's length in bits, so a caller can require 15 bits. The tag is a uint32, so replay can write an RTP timestamp. The e2e publisher still writes the frame index.
The corpus operators cannot build some cases the media span depends on. Delay holds a span of reads and then drains it. It cannot remove a span or read it all at one instant. Nor can it give each frame its own delay, and no operator renumbers what follows a pause. Four operators cover those cases: - Blackout removes the originals read in a span. The gaps it leaves are what the sender sent meanwhile. - Stall reads a span's records in order at its end, like a receiver that reads nothing and then a burst. - Pause removes the frames first sent in a span. Later sequence numbers and picture IDs follow on with no gap, and timestamps keep the sender's clock, as LiveKit sends across a pause and resume. - DelayFrames delays each frame by its own amount, keyed by RTP timestamp, and keeps the read order.
The WebM oracle checks lag for Opus only, and it pairs blocks with frames by order, which a dropped frame breaks. No check says where the file plays a VP8 frame against when it was read. A gap the writer collapses, or a start that leaves the first frames late, passes every expect file. Each VP8 block can now name its frame. SummarizeWebM reads the tag corpus.TagFrame writes, and VideoPlacement pairs each block with the frame read at that RTP timestamp. TimelineLag now reads its frames the same way, so both skip a record with no payload. VideoPlacement fails on a block with no tag, a tag no frame read has, or tags that do not increase. It reports the range of the blocks' timeline errors: a block's PTS since the first block, less its RTP time since that block. It also reports the ideal start, the latest one that plays no block after its read, and the frames read but not written.
The replay checks a VP8 file's block count, duplicates and order, but not where each block plays. The media span check sees only the first key frame. So a writer that collapses the gap after a loss, or caps a long frame step, passes every seed. Replay now tags each VP8 frame's first packet with its RTP timestamp, so oracle.VideoPlacement can pair each block with its frame. A key frame's tag starts after its 10-byte header, and a delta frame's after its 3-byte one. So with one track the tag changes nothing the writer does. In an audio and video file, the writer can read a delta frame's tagged bytes as the video's dimensions, which no check reads. Every block must name a frame read, in increasing order. The blocks' timeline errors must agree within about 1 ms. The writer floors each PTS to ms, so the check bounds the spread of the errors, not each one. A single-track VP8 seed's span must start at the placement's ideal start, so c01-px-ml-ab-1-reccam drops its mediaStartUs pin. An Opus seed whose lag oracle fails must pin mediaStartUs. c02-opus-tail-blank pins 0, as the writer skips its +1-tick frame.
No seed has a long gap inside a file or a stall before its first read. A writer that collapses a 9 s gap, caps frame steps at 1 s, or takes arrival gaps for long steps passes the corpus. No seed shows how far back the media start can move. Four synthetic seeds cover them. Two are 9 s camera blackouts, one of them past the jitter buffer's new-stream threshold. One is a 4 s pause and resume that only the timestamp carries. In the last, every read of a camera stalls for the first 2.5 s. The start moves back at most 1 s, so the stall pins a 1.5 s start. A unit test checks the same limit for a microphone. The blackouts must keep their 9.6 s PTS gap and the resume its 4 s. The first frames after a gap often come late, as from a paced key frame or a draining queue. So the gap seeds read them 300 ms late. A writer that takes the arrival gap for the step then plays the rest of the file late.
A camera has a 2 s start window and no jump step because its first frames can arrive late. TestAdapterReportsALowFrameRateCameraStart checks that with a 5 fps camera, whose first on-time frame comes 600 ms in. A 1 s window passes it. The test now also replays a 30 kbps camera, as Chromium sends one at BBB's low-u30 profile. Three frames come about 450 ms late, then none until the first on-time frame at PTS 1001. A 1 s window settles before that frame and leaves the start 450 ms late.
The writer counts PTS from the first key frame it sees, even one it then rejects, such as an 8x8 blank frame. The first block written can then fall a fraction of a ms past a whole ms, and its PTS is floored. Measured from the first block's RTP time, later blocks then play up to that fraction late, or up to the rest of the ms early. No seed starts this way, so a placement check that assumes a whole ms there passes the corpus. c15-start-blank-key is a 30 fps camera whose key frame 0 is 8x8. The writer rejects it, asks for a key frame once, and writes from key frame 20. Its blocks then play up to 0.333 ms either side of their RTP time since block 0. The seed also pins what follows the rejection. A writer that keeps its key frame state would write the 19 deltas after it, which no key frame decodes.
TestMediaTimestampsPlaceEachFrame checks each block's placement and the media span in its own loop. TestSimulcastMediaTimestampsPlaceEachFrame and TestMediaTimestampsTakeOffAStartDelay repeat that loop with their own copies of the bounds, and the fault scenarios need the same checks. More copies would drift apart. One helper does it for every scenario. checkPlacement reads a file's tagged blocks and gets its recordingMediaStarted from the Driver's new WaitMediaStarted, which the browser tests now use too. It then checks each block's placement and the span in recordingStopped. TestMediaTimestampsPlaceEachFrame keeps its mediaStartHR check, within 2 ms. A preemption between the monotonic and wall-clock reads would skew the host clock offset. So hostClockOffset reads the wall clock on both sides, takes the midpoint, and keeps the narrowest of five tries. A missing recordingMediaStarted now fails the simulcast test without stopping it, as it already did in the other two.
TestUnrepairedLossRequestsKeyframe checks that the recorder asks for a key frame and writes frames again, not where those frames play. A writer that shrinks the PTS gap across the outage, or a start that moves, passes it. The camera now tags its frames, and the test runs checkPlacement on the file. Each block must play within 40 ms of when its frame was due, and recordingStopped must carry the file's span. The outage is checked by tag. The lossy frame and at least one more must be missing, and the first block after them must be the PLI's key frame. A run that wrote no WebM skips these checks. They would stop the test before its log, whose key-frame requests show whether the recorder looped. TestPacedKeyframesNoLoop shares runLoss and gets tagged frames too, but no placement check: its paced key frames send the frames behind them late.
TestNACKRepairUnderLoss checks that LiveKit repaired the loss and that nearly every frame was written, not where those frames play. A frame that waits for its retransmission is written late, so a writer that times frames by when it writes them passes the test. The camera now tags its frames. runNACKScenario checks that each block plays within 40 ms of when its frame was due, under loss and in TestNACKRepairControl. recordingStopped must carry the file's span. A positive control shows that the loss held frames back. At least one repaired gap must have waited over 40 ms, with the WebM holding every frame it repaired. server-sdk-go sends a NACK only when it reads a packet 20 ms or more after the gap: at 15 fps, the next frame's. So a repaired frame waits over 65 ms. traceFrames maps each trace timestamp to the publisher's frame by the frame sent last before each read. LiveKit moves timestamps by a constant, and the file can start after the trace's first key frame.
The signal-drop scenarios count frames and PTS gaps, but not where the
frames play. A resume that re-bases timestamps keeps every count and gap
within bounds. TestSignalRefusedSplitsFile also has the only
recordingStopped that no stopRecording asked for ("closed"), and the
only reused session ID. Neither scenario checks the media span in its
recordingStopped.
Both tracks now tag their frames, and each file goes through
checkPlacement. Each frame must play within 40 ms of when it was sent,
and recordingStopped must carry the file's span. The camera and the
microphone recorded together must land within 15 ms of each other.
After a refused redial, the first file is checked against its closed
recordingStopped, whose recordingMediaStarted must come before it. That
event names no file, so a start missing for the first file would
otherwise match the second file's. The second file must hold only later
frames, and its first frame must be sent after the refusal ended.
After a cut that resumes, the test logs what the resume cost.
readBlocks reads a WebM through mkvcore's block reader, which stops at the end of the first cluster. The recorder's writer starts a cluster when a block would sit 32.767 s or more into the current one. So a longer file loses its later blocks with no error, and the placement checks pass on part of the file. The oracle already reads the whole segment. Its walk becomes oracle.ReadBlocks, which returns the tracks and every block of every cluster. SummarizeWebM and readBlocks both read through it, so the e2e checks and the replay read a file the same way, TimecodeScale included.
readFrames joins the packets of one RTP timestamp into one frame, for Opus too. On a mute, LiveKit's first silence frame can reuse the timestamp of the frame before it. readFrames then joins the two into a frame that no block holds. The recorder writes the first packet and skips the second, whose timestamp does not advance. pairBlocks pairs the block with a later frame of the same bytes, or fails on a block with none left. An Opus frame is now one packet, so each block pairs with the packet it holds. The skipped packet is a frame that no block holds, which pairBlocks allows.
The browser runner publishes its microphone with createLocalAudioTrack, and cannot mute, unpublish or stop a track. So no browser scenario can make the new tracks, and new files, that BBB makes in a meeting. BBB's client publishes a track of its own capture, so a mute only disables it. It unpublishes a microphone muted for 5 s, and publishes the same capture again on unmute, as a new track. Stopping the camera unpublishes it, and sharing it again publishes a new one. The runner now does what BBB does. PublishBBBMic publishes a user-provided microphone with BBB's options and runs BBB's 5 s unpublish timer in the page. It replaces PublishMic, so every browser scenario publishes its microphone as BBB does. UnmuteMic unmutes with LiveKit's MutePublishedTrack, as bbb-webrtc-sfu does with BBB's default settings. Once the timer has unpublished the microphone, the page publishes it again, as BBB's LiveKit audio bridge does. UnpublishCamera stops the capture and unpublishes the camera. CameraCapture.Preview holds the capture before the publish, as BBB's preview does.
Every scenario starts its files together, at publish. An error that two files share in how they start then cancels out in the lip sync check. In a meeting, BBB's client makes a new microphone file on almost every unmute and a new camera file on every re-share. Each starts while a file of the other kind has run for a while. TestBrowserMidMeetingFilesKeepLipSync publishes as BBB's client does and records each new track as bbb-webrtc-sfu does. A 3 s mute keeps the microphone's file; a mute past 5 s makes BBB unpublish it, so the unmute starts a new one. The camera, published with low-u30's late first frames, is stopped and shared again with the low profile. All four files must keep the lip sync scenario's bounds on the browser's capture clock. Positive controls show that the mutes and the re-share went as planned, and that each new file started while the other kind ran. The host's wall clock must not step.
No scenario impairs a browser's network. The NACK plan shapes only the servers' link to the recorder. A browser on the host shares the test's link to the servers. Shaping that link, as the start-delay scenario does, impairs the driver and LiveKit's API calls too. newBrowserNetemPlan adds a third namespace, linked only to the servers. Its BrowserNet holds the servers' address on that link and a wrapper that runs a command in the namespace. Both ends of the link get a netem qdisc, and shaping changes both. The recorder, the driver and the RoomService calls stay on links netem does not touch. So the fault cannot impair the recorder's recovery or the test's measurements. The netem scenarios now share their gate, requireNetem, and their stack setup, newNetemStack. Deleting a namespace leaves what runs in it running, and Chromium can outlive the runner's process group. So the cleanup kills every process left in the plan's namespaces before it deletes them, also when it removes a dead run's namespaces. TestNetemBrowserPlan runs the browser's wrapper against a stand-in for ip, so it needs no iproute2.
NewBrowser runs node on the host, and the page dials LiveKit's URL, the test's address of the servers. A browser in a namespace of its own cannot use either. StackOptions.Browser takes a BrowserNet. With one, the runner starts through its wrapper, like a server does through ServerNet, and the page dials LiveKit on the BrowserNet's host. A stack without one runs as before. node is resolved to an absolute path, which the wrapper runs as it is. The runner states Playwright's default of no Chromium sandbox: as root, the sandbox refuses to start.
A scenario that cuts the browser's network cannot tell from the page what livekit-client did about it. It sees no resume or reconnect, no track a reconnect published again, and no NACK Chromium answered. CameraPLIs reads only simulcast layers, so a camera of one layer has no stats at all. These are the positive controls of a network fault: they show the fault reached the browser, and which of livekit-client's paths it took. The page subscribes to the room's reconnect, local track and connection quality events before it connects, and keeps each with the host's time. It also reads each layer's outbound-rtp counters, keyed by rid or "" for one layer. SenderStats replaces CameraPLIs.
placeOnCaptureClock maps each frame through the sender report nearest its timestamp. After a blackout, LiveKit can re-base a downtrack's timestamps. A report from before the blackout then maps the frames after it by the old base, and the oracle would blame the recorder. placeOnCaptureClockCut maps each frame by the reports read on its own side of a blackout. A file that keeps the publisher's timing then passes, whatever LiveKit does to the base. A segment with blocks and no report fails instead of borrowing one. The browser scenarios compare the files' lateness with each other. So a start error that every file shares passes them, while mediaStart stays within the 1 s before the first frame's read. The helper now also bounds each file's lateness on loopback, and each block's drift from it. Lateness includes the path's delay, which the recorder cannot see. So the helper also checks mediaStart against the reads of the frames in its start window.
No scenario checks placement through a loss, a blackout or a reconnect on a browser's network. Any of them can move a file's start or shrink its PTS gap, and nothing would fail. record-and-playback plays each file from its start and never stretches it, so every file must sit on Chromium's capture clock and keep lip sync. TestBrowserNetemKeepsPlacement shapes only the browser's link, so the recorder's recovery and the oracle stay clean. It runs loss and jitter, and blackouts of 3 s, 13.5 s and 25 s. The 13.5 s one takes livekit-server's ICE to disconnected. After the 25 s one, livekit-client reconnects with new tracks. A follower records each track LiveKit lists, as bbb-webrtc-sfu does. Across a blackout, the PTS step must equal the capture step. Positive controls show that each fault took effect. A camera's sender reports place its capture more closely than a microphone's, so cameras get their own report error, browserCameraReportError. This tightens a camera's lower bound in every browser scenario. TestBrowserNetemControl runs the 3 s case on loopback with a blackout that changes nothing, to show the blackout checks can tell it apart.
TestBrowserNetemKeepsPlacement needs root and a browser. The browser leg runs its scenarios unprivileged, where the browser netem scenario skips. The root step of the NACK and start-delay scenarios skips the browser leg. So CI never runs it. The root step now runs on the browser leg too, with only that scenario. So a change that moves placement behind a bad network fails a PR. E2E_BROWSER_REQUIRED makes a broken browser setup fail it instead of skipping. go test passes when -test.run selects no test, so the step first counts the tests its pattern lists. On the browser leg, the step runs only if the browser install passed. sudo gets PATH, for setup-node's node, and PLAYWRIGHT_BROWSERS_PATH, since Playwright under sudo looks for Chromium in root's home.
Every blackout, resume and pause seed is a camera, and no microphone seed that checks placement has a step over 260 ms. So a writer that turns a long Opus step into one 20 ms frame passes the corpus and every unit test. Only the browser scenarios catch it, and they need Chromium. c24-mic-blackout-9s reads nothing for 9 s, and the jitter buffer counts the loss. c24-mic-resume-4s sends nothing for 4 s: the sequence numbers follow on, and only the timestamp carries the gap. Each reads the frames sent in the 300 ms after its gap in one burst, as a draining queue would. A writer that takes that arrival gap for the step plays the rest of the file late, so the PTS must keep the gap. PauseOpus is Pause for an Opus track, which has no picture ID. A new audioMaxGapMs bounds an Opus track's largest PTS step. c02-opus-tail-blank, which the lag oracle cannot place, pins its 41 ms close-out step with it.
Every stack's livekit-server offers ICE over TCP: it gets a tcp_port, and LiveKit's fallback to TCP is on by default. BBB's packaged livekit.yaml turns both off. So a test's client can take a path no BBB server offers. Under netem, media on TCP sees loss as delay, and the netem checks read only the selected pair's address. With liveKitICETCP false, the stacks set tcp_port 0 and allow_tcp_fallback false, as BBB does. Setting it to true gives each stack a TCP port and the fallback again, should BBB turn them on.
BBB's away mode mutes a camera in place. The track stays published, so its file runs on while LiveKit forwards nothing. No live scenario mutes a camera. A writer that caps a video gap's step passes every unprivileged scenario's placement check; live, only the root browser netem scenario's long blackout catches it. TestMediaTimestampsPlaceEachFrame now mutes its camera for 3 s midway. Each block must still land where its frame was sent. The file must also hold a gap over maxVideoStep, which shows the mute reached the recorder. A writer that caps the gap at 1 s plays the rest of the camera's file 2.3 s early.
checkPlacement lets a block land up to 40 ms from where its frame was sent. A correct recorder's worst on CI is about 1.3 ms. The lip sync check compares medians, and the browser scenarios let a block drift 30 ms. So a recorder that puts some blocks 16-30 ms off, such as one simulcast layer's, passes every scenario. A file places all its blocks with one offset: mediaStart comes from the least delayed of its first frames, and each PTS from RTP time. On loopback that offset stays within about 3 ms. framePlacementBound is now 15 ms, under one 20 ms Opus frame. The NACK scenario's frames held for a retransmission still wait longer than that, 65 ms or more.
The e2e suite hard-codes the recorder's media span limits: its own maxVideoStep and maxAudioStep, and startByReads' 120 ms jump step and 1 s backlog. It also converts NTP times and a trace's read times with code of its own, beside corpus.SenderNTP. A change on either side leaves the copy behind. The recorder exports MediaSpanAudioJumpStep, MediaSpanMaxBacklog, MaxAudioStep and MaxVideoStep, as it does MediaSpanWindow. corpus gains NTP, which SenderNTP now uses, its inverse NTPTime, and Header.Time, a read time on the wall clock. The e2e tests use them in place of their copies.
The replay accepts any media span from the file's media to 1 s more. So a writer that hands mediaSpan a wrong last-frame duration, such as 0, passes every seed. Only the e2e scenarios' mediaEnd check catches it. Every single-track seed's span must now last the file's media plus its last frame within 2 ms, as assertMediaEnd requires. The last frame lasts the PTS step before it, capped at MaxVideoStep or MaxAudioStep. A writer that passes 0 for an Opus frame's duration fails five microphone seeds.
prlanzarin
force-pushed
the
bwr/feat/new-ts-pr2
branch
from
November 10, 2026 19:06
df13fce to
41095fa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Test-only follow-up to #51. #51 made the recorder report where each LiveKit file's media sits on the wall clock. Its tests check that every frame plays where it was sent: on loopback, behind a start-up delay and across simulcast layer switches.
This PR checks the same under faults that can break A/V sync. A recorder change that moves a file's start or shrinks a gap now fails a test:
The replay corpus now checks where every seed plays each VP8 frame, and how long each single-track seed's media span lasts. New seeds add blackouts and pauses for a camera and a microphone, a camera stall before its first read, and a first key frame the writer rejects. CI runs the browser netem scenario as root, and the test LiveKit offers no ICE over TCP, as BBB's does.
In detail:
How to test
TestNACKRepairUnderLoss,TestMediaTimestampsTakeOffAStartDelayandTestBrowserNetemKeepsPlacementskip unless run as root withE2E_NETEM=1, the last withE2E_BROWSER=1too. AGENTS.md shows how to run them without sudo. CI runs all three as root.Changelog
Technical — Tests check where every frame of a LiveKit recording plays under loss, blackouts, a camera mute and reconnects, and in files that start mid-meeting. CI runs the browser netem scenario as root.
User-facing — None.