Fix latency issue with rl pacing#1649
Draft
Sakoram wants to merge 4 commits into
Draft
Conversation
ST20 hardware-RL sessions could transmit the first real packet of a frame up to ~1.15ms before its scheduled target: the transmitter enqueued all warm-up pads and then the real packet immediately, trusting the NIC's rate-limiter to consume exactly warm_pkts * trs of wall-clock time before draining the real packet. The patched ICE driver's 2KB token bucket lets the pad train depart faster than that under real hardware timing, so the real packet could reach the wire early. RTP timestamps stayed correct, so this showed up purely as negative RX-timestamp-minus-RTP-timestamp latency. The transmitter now tracks explicit per-port RL state (IDLE/WAIT_WARMUP/WAIT_TARGET): pads still start early, but the first real packet is retained and released only once its own target TSC is reached, with failed warm-up pads still retried cooperatively before the target check. `st_tx_video_transmitter_state_cleanup()` centralizes resetting/releasing this retained state (and prior in-flight/pad bookkeeping) across attach, teardown, and fatal queue recovery so a NIC hang can't leave stale retained mbufs or state behind. The RL RTP-timestamp compensation shift (MTL_LATENCY_COMPENSATION_PACKET_SHIFT) is removed now that the real packet is genuinely synchronized to its target instead of relying on timestamp adjustment to paper over early transmission. `st20_tx_get_pacing_params_ex()`/`st20p_tx_get_pacing_params_ex()` expose the pacing parameters needed to verify this end-to-end. Signed-off-by: Kasiewicz, Marek <marek.kasiewicz@intel.com>
Adds a white-box harness for st_video_transmitter.c (st_video_transmitter_harness.c/.h) that mocks the TSC clock and NIC burst call to exercise _video_trs_rl_tasklet()/video_trs_rl_warm_up() directly, plus extends the existing st20_tx harness with accessors for tv_update_rtp_time_stamp() and the RL RTP-offset/pacing-cursor state. rl_warm_up_test.cpp pins the warm-up pad count staying bounded to the planned amount and the first real packet never being released before its target TSC (including already-late targets, zero warm packets, failed-pad retry ordering, and state reset across frames). transmitter_bugs_test.cpp pins the *ret_status inflight-retry bookkeeping, the pad-burst hang-detector update, the signed target/cur TSC delta, and the transmitter recovery cleanup helper. rtp_timestamp_test.cpp pins that round-to-nearest media-clock conversion can overshoot the real TAI time by up to half a tick now that the RL compensation shift is gone, and that user-visible RTP timestamps never go negative relative to the real send time. Signed-off-by: Kasiewicz, Marek <marek.kasiewicz@intel.com>
St20pUserTimestamp (and its exact-pacing/redundant-latency subclasses) now records, per received frame, the delta between actual receive time and the epoch-snapped expected transmit time, plus the RX-timestamp-minus-RTP-timestamp latency. assertTimingWithinBudget() flags any single frame (after frame 0, excluded as session warm-up) whose delta exceeds a 15us bound; a rare single-frame scheduling jitter is tolerated via a small (1%) outlier budget, but a systemic pacing regression still fails fast. assertRlLatencyWithinBounds() has no outlier budget: it fails on any frame past frame 0 with negative latency (a frame arriving before its own RTP timestamp) or latency >= 20us (RL warm-up overshoot). The average latency across all frames is always included in the failure message so a one-off outlier is distinguishable from a systemic regression. These are the regression checks that caught the target-gating bug fixed separately. getPacingParameters() pulls tr_offset/trs/vrx from st20p_tx_get_pacing_params() so the expected-transmit-time and expected-media-clock calculations track production instead of duplicating its constants. Signed-off-by: Kasiewicz, Marek <marek.kasiewicz@intel.com>
subprocess.run(..., text=True) only decodes stdout/stderr on normal return; on TimeoutExpired the partially-read buffers attached to the exception can still be raw bytes. Concatenating that with a str raised "can't concat str to bytes", masking the real timeout behind an unrelated tool error. _decode() coerces both paths to str before use. Build/install tools (build_mtl, dpdk_build, ice_driver_rebuild, install_dependencies, build_ffmpeg_plugin, build_gstreamer_plugin, build_ebpf_xdp) can each produce thousands of lines of output; returning that directly from an MCP tool bloated the agent's context for no benefit once saved to disk. _run_logged() now saves the full output under build/logs/ and returns only its last 40 lines plus the log path; log_tail(source='saved', log_path=...) reads the rest back on demand, with the path validated to stay under build/logs/. run_gtest now refuses NoCtxTest filters up front: each NoCtxTest case needs its own KahawaiTest process (DPDK EAL cannot reinit within one), which run_gtest doesn't support — point callers at run_noctx_tests instead of letting the run fail or hang. Signed-off-by: Kasiewicz, Marek <marek.kasiewicz@intel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.