ST40p RX: move frame assembly into the transport + RFC 8331 codec dedup

.github/agents/mtl-developer.agent.md

@@ -1,5 +1,5 @@
 ---
-description: "Writes production MTL C/C++ AND the gtest cases that pin its behavior, in one context window. Walks a six-gate test-first loop (knowledge → failing test → implement → green test → self-review → hand off). Use for: any code change to lib/, include/, app/, plugins/, ecosystem/, or tests/** — bug fixes, new features, regressions, behavior changes; building (./build.sh / ninja -C build / ./format-coding.sh); running unit gtest (./build/tests/unit/UnitTest). Do NOT use for: running KahawaiTest against real VFs (→ MTL System Admin); host setup (→ MTL System Admin); pytest under tests/validation (→ main agent per validation-tests instruction); read-only Q&A (→ Explore); multi-subsystem orchestration (→ MTL Planner). Tools: editFiles, read, search, usages, problems, testFailure, todo, memory, execute, agent (Explore only). Requires: execute."
+description: "Writes production MTL C/C++ AND the gtest cases that pin its behavior, in one context window. Walks a six-gate test-first loop (knowledge → failing test → implement → green test → self-review → hand off). Use for: any code change to lib/, include/, app/, plugins/, ecosystem/, or tests/** — bug fixes, new features, regressions, behavior changes; building (./build.sh / ninja -C build / ./format-coding.sh); running unit gtest (./build.sh unit). Do NOT use for: running KahawaiTest against real VFs (→ MTL System Admin); host setup (→ MTL System Admin); pytest under tests/validation (→ main agent per validation-tests instruction); read-only Q&A (→ Explore); multi-subsystem orchestration (→ MTL Planner). Tools: editFiles, read, search, usages, problems, testFailure, todo, memory, execute, agent (Explore only). Requires: execute."
 name: "MTL Developer (TDD)"
 tools: ['editFiles', 'read', 'codebase', 'search', 'usages', 'problems', 'testFailure', 'todo', 'memory', 'execute', 'agent']
 agents: ['Explore']
@@ -39,7 +39,7 @@ C coding rules, KB routing, gtest conventions, and the test-tier picker auto-loa
 |---|---|
 | Edit `lib/`, `include/`, `app/`, `plugins/`, `ecosystem/`, `tests/unit/`, `tests/integration_tests/` | Configure host — hugepages, VFs, drivers (delegate to MTL System Admin) |
 | Build (`./build.sh`, `ninja -C build`, `./format-coding.sh`) | Run integration `KahawaiTest` against real VFs (delegate to MTL System Admin) |
-| Run unit gtest (`./build/tests/unit/UnitTest`) | Run pytest under `tests/validation/` (return to main agent) |
+| Run unit gtest (`./build.sh unit`) | Run pytest under `tests/validation/` (return to main agent) |
 
 ## The six-gate loop
 

.github/agents/mtl-system-admin.agent.md

@@ -1,5 +1,5 @@
 ---
-description: "Configures MTL hosts via MCP tools (no shell). Use for: post-reboot setup (hugepages, VFs, MtlManager), driver install/rebuild (DPDK, ICE), NIC bind/unbind, building MTL via `build_mtl`, running integration gtest `KahawaiTest` via `run_gtest` MCP tool against real VFs. This is the **enforced exit gate** for data-plane / session-lifecycle changes — MTL Developer (TDD) hands off here at Gate 6. Do NOT use for: running unit gtest `./build/tests/unit/UnitTest` (→ MTL Developer (TDD)); editing source code (→ MTL Developer (TDD)); arbitrary shell commands. Tools: `tool_search` (loads MCP tools), `todo`, MCP `mcp_mtl-system-se_*`. Requires: MCP server reachable."
+description: "Configures MTL hosts via MCP tools (no shell). Use for: post-reboot setup (hugepages, VFs, MtlManager), driver install/rebuild (DPDK, ICE), NIC bind/unbind, building MTL via `build_mtl`, running integration gtest `KahawaiTest` via `run_gtest` MCP tool against real VFs. This is the **enforced exit gate** for data-plane / session-lifecycle changes — MTL Developer (TDD) hands off here at Gate 6. Do NOT use for: running unit gtest `./build.sh unit` (→ MTL Developer (TDD)); editing source code (→ MTL Developer (TDD)); arbitrary shell commands. Tools: `tool_search` (loads MCP tools), `todo`, MCP `mcp_mtl-system-se_*`. Requires: MCP server reachable."
 name: "MTL System Admin"
 tools: ['tool_search', 'todo', 'mtl-system-setup/*']
 user-invocable: true

.github/copilot-instructions.md

@@ -43,7 +43,7 @@ Use this table to pick the right subagent. Resolve ambiguity in order:
 | Multi-step work crossing 2+ subsystems (code + host + manager + plugins…) | **MTL Planner** | Decomposes and routes; no execution |
 | Edit any of `lib/`, `include/`, `app/`, `plugins/`, `ecosystem/`, `tests/unit/`, `tests/integration_tests/` | **MTL Developer (TDD)** | Owns code + tests + the six-gate TDD loop in one context window |
 | Build (`./build.sh`, `ninja -C build`, `./format-coding.sh`) | **MTL Developer (TDD)** | Build is Gate 4 of its loop |
-| Run `./build/tests/unit/UnitTest` (unit gtest, no NIC) | **MTL Developer (TDD)** | Test runs are Gate 2 (fail) and Gate 4 (pass) |
+| Run `./build.sh unit` (unit gtest, no NIC) | **MTL Developer (TDD)** | Test runs are Gate 2 (fail) and Gate 4 (pass) |
 | Run `KahawaiTest` (integration gtest, real VFs) | **MTL System Admin** | Only one with `run_gtest` MCP tool; enforced Gate 6 for data-plane changes |
 | Host setup (hugepages, VFs, drivers, MtlManager) | **MTL System Admin** | MCP-only, no shell |
 | Adversarial review of a saved diff | **MTL Reviewer** | Read-only; enforced Gate 5; refuses if diff empty |

.github/instructions/mtl-system-setup.instructions.md

@@ -20,17 +20,24 @@ decision logic and domain knowledge that tool descriptions alone don't convey.
 4. If MTL not built or rebuild needed → `build_mtl` or `mtl_clean_rebuild`
 
 ### "Run integration tests"
-1. Verify prerequisites: DPDK installed, VFs bound, MtlManager running
+1. Verify prerequisites **before** the first `run_gtest` call: `dpdk_devbind_status` (ports bound as VFs, not bare PFs — see below), MtlManager running (`manager_start` if not).
 2. `run_gtest` with appropriate `gtest_filter` (e.g. `St20p*`, `St30*`)
 3. If tests segfault → see "Test crashed" below
-4. Report pass/fail counts
+4. If RX session creation fails with multicast-join/IGMP errors → see "Multicast join / IGMP failure" below
+5. Report pass/fail counts
 
 ### "Test crashed / segfault"
 1. **SEGFAULT in `iavf_tm_node_add`** → `ice_driver_status` — almost always stock ICE
 2. **SEGFAULT elsewhere** → `dmesg_tail` for kernel errors
 3. `dpdk_status` → version mismatch?
 4. `mtl_clean_rebuild` → stale build?
 
+### "Multicast join / IGMP failure on RX session creation"
+1. `dpdk_devbind_status` first — check if the port is a bare PF bound to `vfio-pci` with 0 VFs. E810/E830 must always run through VFs (see `nic_bind_pmd`'s own tool description: prefer `nic_create_vf` for E800 series); a bare PF-to-vfio-pci binding causes exactly this symptom.
+2. Fix: `nic_bind_kernel` on the PF, then `nic_create_vf` (use `trusted=true` if the test needs multicast/promiscuous features).
+3. Re-run `run_gtest` with the newly created VF BDFs as `p_port`/`r_port`.
+4. This is an environment issue, not a code regression — confirm by reproducing on one unrelated, already-passing suite (e.g. `St30_rx.create_free_single`) only if you need extra certainty; don't use it as the primary diagnostic path, step 1 above is sufficient and faster.
+
 ### "Build failed"
 1. **Permission errors / CMakeCache.txt** → `mtl_clean_rebuild`
 2. **librte_*.so not found** → `sudo ldconfig` (or `dpdk_build` if DPDK not installed)

.github/mcp/mtl_mcp_server.py

@@ -115,6 +115,23 @@ def _save_test_log(name: str, content: str) -> Path:
     return log_path
 
 
+def _summarize_output(name: str, out: str, tail_lines: int = 40) -> str:
+    """Save large command output to a log file, returning a short summary.
+
+    Full build/install output can be thousands of lines — dumping it into a
+    tool result burns context for little benefit. This saves it to disk and
+    returns the log path, total line count (like `wc -l`), and just the
+    tail, so the caller can grep/read the file directly if more is needed.
+    """
+    log_path = _save_test_log(name, out)
+    total_lines = len(out.splitlines())
+    tail = "\n".join(out.splitlines()[-tail_lines:])
+    return (
+        f"- Full log: `{log_path}` ({total_lines} lines)\n"
+        f"### Output (last {tail_lines} lines)\n```\n{tail}\n```"
+    )
+
+
 def _load_versions() -> dict[str, str]:
     """Parse versions.env into a dict."""
     result: dict[str, str] = {}
@@ -788,7 +805,8 @@ def ice_driver_rebuild() -> str:
     )
 
     return (
-        f"## ICE Driver Build\n{out}\n\n## Reload\n{reload_out}\n\n"
+        f"## ICE Driver Build\n{_summarize_output('ice_driver_rebuild', out)}\n\n"
+        f"## Reload\n{reload_out}\n\n"
         "## ⚠ Important: VFs Destroyed\n"
         "The ICE driver reload destroyed all existing VFs.\n"
         "You MUST re-create VFs before running any tests:\n"
@@ -846,7 +864,10 @@ def dpdk_build() -> str:
     ver_after = _run_output(
         "pkg-config --modversion libdpdk 2>/dev/null || echo 'not installed'"
     )
-    return f"## DPDK Build\n{out}\n\n## Result\nInstalled version: {ver_after}"
+    return (
+        f"## DPDK Build\n{_summarize_output('dpdk_build', out)}\n\n"
+        f"## Result\nInstalled version: {ver_after}"
+    )
 
 
 # ===================================================================
@@ -899,7 +920,7 @@ def build_mtl(mode: str = "debugonly") -> str:
     libmtl = _run_output("ldconfig -p 2>/dev/null | grep libmtl || echo 'not found'")
 
     return (
-        f"## MTL Build ({mode})\n{out}\n\n"
+        f"## MTL Build ({mode})\n{_summarize_output('build_mtl', out)}\n\n"
         f"## Artifacts\n- RxTxApp: {rxtxapp}\n- MtlManager: {manager}\n- libmtl: {libmtl}"
     )
 
@@ -1248,7 +1269,7 @@ def build_ffmpeg_plugin() -> str:
         "bash .github/scripts/setup_environment.sh",
         timeout=600,
     )
-    return f"## FFmpeg Plugin Build\n{out}"
+    return f"## FFmpeg Plugin Build\n{_summarize_output('ffmpeg_plugin_build', out)}"
 
 
 @mcp.tool()
@@ -1265,7 +1286,9 @@ def build_gstreamer_plugin() -> str:
         "bash .github/scripts/setup_environment.sh",
         timeout=600,
     )
-    return f"## GStreamer Plugin Build\n{out}"
+    return (
+        f"## GStreamer Plugin Build\n{_summarize_output('gstreamer_plugin_build', out)}"
+    )
 
 
 # ===================================================================
@@ -1291,7 +1314,7 @@ def install_dependencies() -> str:
         "bash .github/scripts/setup_environment.sh",
         timeout=300,
     )
-    return f"## Install Dependencies\n{out}"
+    return f"## Install Dependencies\n{_summarize_output('install_dependencies', out)}"
 
 
 # ===================================================================
@@ -1316,7 +1339,7 @@ def build_ebpf_xdp() -> str:
         "bash .github/scripts/setup_environment.sh",
         timeout=300,
     )
-    return f"## eBPF/XDP Build\n{out}"
+    return f"## eBPF/XDP Build\n{_summarize_output('ebpf_xdp_build', out)}"
 
 
 # ===================================================================
@@ -1495,14 +1518,8 @@ def run_gtest(
     if failures:
         summary += "\n### Failed Tests\n" + "\n".join(f"- {f}" for f in failures)
 
-    # Save full log
-    log_path = _save_test_log("gtest", out)
-    summary += f"\n- Full log: `{log_path}`"
     summary += sim_hint
-
-    # Include last 40 lines for context
-    tail = "\n".join(out.splitlines()[-40:])
-    return f"{summary}\n\n### Output (last 40 lines)\n```\n{tail}\n```"
+    return f"{summary}\n\n{_summarize_output('gtest', out)}"
 
 
 @mcp.tool()

.github/skills/mtl-build/SKILL.md

@@ -14,6 +14,7 @@ Run from the repository root:
 | `./build.sh` | Release | Default. Optimized build for testing and deployment |
 | `./build.sh debug` | Debug + ASAN | Debugging with AddressSanitizer (`-O0 -g`). Slower but catches memory bugs |
 | `./build.sh debugonly` | Debug only | Debug symbols without ASAN. Faster than `debug` for general development |
+| `./build.sh unit` | Unit gtest | Configures `build_unit/` with `-Denable_unit_tests=true`, builds it, and runs `tests/unit/UnitTest`. Combine with `debug` (e.g. `./build.sh debug unit`) to run under ASan via `LD_PRELOAD` |
 
 ## Format Code
 

.github/skills/mtl-write-test/SKILL.md

@@ -11,7 +11,7 @@ The hard part of testing MTL is **picking the right tier** and **asserting the r
 
 | Tier | Binary | Hardware | Can observe |
 |---|---|---|---|
-| **Unit gtest** — `tests/unit/` | `UnitTest` (built only with `-Denable_unit_tests=true`; **not** built by `./build.sh`) | None — runs as regular user under ASan | RX-side session logic fed synthetic mbufs at the RFC 4175 / RTP payload layer: frame assembly, bitmap dedup, redundancy merge, stat counters, pipeline accounting |
+| **Unit gtest** — `tests/unit/` | `UnitTest` (built with `-Denable_unit_tests=true`; via `./build.sh unit`) | None — runs as regular user under ASan | RX-side session logic fed synthetic mbufs at the RFC 4175 / RTP payload layer: frame assembly, bitmap dedup, redundancy merge, stat counters, pipeline accounting |
 | **Integration gtest** — `tests/integration_tests/` | `KahawaiTest` | Real VFs (≥2; ≥4 for redundant cases) | Full session lifecycle, TX+RX through the NIC, every pacing mode, multi-session, callbacks at real timing, SHA digests, DMA, RSS, kernel-socket, AF_XDP, virtio-user |
 | **NoCtx gtest** — `tests/integration_tests/noctx/` | `KahawaiTest --no_ctx` | Real VFs (per-case; redundant cases require ≥4) | Same as integration plus behaviors that depend on `mtl_init()` flags/callbacks the shared test context cannot supply. **One case per process** (DPDK EAL cannot re-init) — runner script enforces this |
 | **pytest single-host** — `tests/validation/tests/single/` | RxTxApp / FFmpeg / GStreamer driven through pytest | Full host: VFs, MtlManager, NFS media, plugins, venv, SSH-to-localhost | End-to-end app behavior on one host: CLI flags, config files, real media bytes, PTP convergence |
@@ -67,7 +67,7 @@ Trap: a library bug whose easiest reproducer is RxTxApp still belongs in a **gte
 1. **Pick a neighbour test in the same directory** — closest to your behavior first, else the most recently modified file. Avoid `DISABLED_` cases and any file noticeably older than its siblings. **Copy its conventions, including its marker set** — markers gate CI selection ([tests/validation/pytest.ini](tests/validation/pytest.ini)) and an unmarked or wrongly-marked pytest runs in no job.
 2. **Modify assertions only** until the test encodes your one requirement.
 3. **Register in the build** for gtest tiers (add to the relevant `*_sources` list in the nearest `meson.build`). pytest needs no registration.
-4. **Build:** `./build.sh` for integration binaries; for unit, `meson setup build_unit -Denable_unit_tests=true` once, then `ninja -C build_unit`. See [`/mtl-build`](.github/skills/mtl-build/SKILL.md).
+4. **Build:** `./build.sh` for integration binaries; `./build.sh unit` for the unit suite (builds `build_unit/` and runs `UnitTest`). See [`/mtl-build`](.github/skills/mtl-build/SKILL.md).
 5. **Run** with the binary / pytest invocation from the tier's auto-loaded instructions.
 6. **Confirm** the test fails before the fix (regression) or passes deterministically across 3+ consecutive runs (new coverage).
 

build.sh

@@ -8,7 +8,7 @@ set -e
 user=$(whoami)
 
 function usage() {
-	echo "Usage: $0 [debug|debugonly|debugoptimized|plain|release] [enable_fuzzing]"
+	echo "Usage: $0 [debug|debugonly|debugoptimized|plain|release] [enable_fuzzing] [unit]"
 	exit 0
 }
 
@@ -17,6 +17,7 @@ enable_asan=false
 enable_tap=false
 enable_usdt=true
 enable_fuzzing=false
+build_unit=false
 
 : "${MTL_BUILD_ENABLE_ASAN:=false}"
 : "${MTL_BUILD_ENABLE_TAP:=false}"
@@ -67,6 +68,9 @@ while [ $# -gt 0 ]; do
 		enable_fuzzing=true
 		echo "Enable fuzzers"
 		;;
+	"unit")
+		build_unit=true
+		;;
 	*)
 		usage
 		;;
@@ -82,6 +86,22 @@ PLUGINS_BUILD_DIR=${WORKSPACE}/build/plugins
 LD_PRELOAD_BUILD_DIR=${WORKSPACE}/build/ld_preload
 MANAGER_BUILD_DIR=${WORKSPACE}/build/manager
 RXTXAPP_BUILD_DIR=${WORKSPACE}/tests/tools/RxTxApp/build
+UNIT_BUILD_DIR=${WORKSPACE}/build_unit
+
+# build and run the unit gtest suite (tests/unit), then exit
+if [ "$build_unit" == "true" ]; then
+	meson setup "${UNIT_BUILD_DIR}" -Dbuildtype="$buildtype" -Denable_asan="$enable_asan" -Denable_unit_tests=true
+	ninja -C "${UNIT_BUILD_DIR}"
+
+	if [ "$enable_asan" == "true" ]; then
+		ASAN_LIBRARY=$(cc -print-file-name=libasan.so)
+		LD_PRELOAD="${ASAN_LIBRARY}${LD_PRELOAD:+:${LD_PRELOAD}}" \
+			"${UNIT_BUILD_DIR}/tests/unit/UnitTest"
+	else
+		"${UNIT_BUILD_DIR}/tests/unit/UnitTest"
+	fi
+	exit 0
+fi
 
 if [ -n "${MTL_INSTALL_PREFIX:-}" ]; then
 	MTL_PREFIX_ARGS="--prefix=$MTL_INSTALL_PREFIX"

doc/design.md

@@ -354,7 +354,18 @@ st40p_rx_put_frame
 ```
 
 - **TX pipeline (`st40p_tx_*`)** wraps the lower-level `st40_tx_*` APIs while honoring redundancy, user timestamps, and user-managed frame buffers (for example when `ST40P_TX_FLAG_EXT_FRAME` is enabled). The helper also exposes the maximum user data word (UDW) size via `st40p_tx_max_udw_buff_size()` so that producers can pack metadata deterministically.
-- **RX pipeline (`st40p_rx_*`)** assembles ancillary packets into user buffers and can optionally dump metadata/UDW buffers directly to files for debugging through `st40p_rx_set_dump`. Each RX callback receives both the metadata blocks and the captured payload, mirroring the TX layout to simplify round-trip validation.
+- **RX pipeline (`st40p_rx_*`)** is a thin shim over the **FRAME_LEVEL transport**
+  (`ST40_TYPE_FRAME_LEVEL` on `st40_rx_ops`). The transport owns the frame-buffer pool,
+  parses RFC 8331 packets, and assembles full frames using a 64-bit per-frame
+  **session-merged sequence bitmap** that natively absorbs cross-port redundancy.
+  Once a frame is complete the transport invokes
+  `notify_frame_ready(priv, addr, struct st40_rx_frame_meta*)`; the pipeline records
+  the buffer pointer plus all stats (timestamps, per-port and session-level
+  `seq_lost`/`seq_discont`, `pkts_total`, `pkts_recv[]`, `status`) into its slot —
+  zero-copy. The app returns ownership through `st40p_rx_put_frame`, which calls
+  `st40_rx_put_framebuff(transport, addr)` under the hood. Apps that still want raw
+  RTP can fall back to `ST40_TYPE_RTP_LEVEL` and pull mbufs via
+  `st40_rx_get_mbuf`/`st40_rx_put_mbuf`.
 
 Split-mode ancillary (packet-split) is supported via `tx_split_anc_by_pkt` on TX and
 `rx_rtp_ring_size` plus `frame_info_path` on RX. Enable `tx_split_anc_by_pkt=true`

doc/stats_guide.md

@@ -337,6 +337,18 @@ ST40 anc) a fully-lost frame is absorbed — the delivered frame count drops
 rather than emitting an extra incomplete frame — so under heavy contiguous
 loss `stat_frames_corrupted` is a lower bound on frame-integrity impact.
 
+> **ST40p RX seq stats are bitmap-derived (transport-side).** The
+> `seq_lost` / `seq_discont` fields on `st40_frame_info` (and the underlying
+> `st40_rx_frame_meta`) are computed in the RX session from a per-frame
+> 64-bit **session-merged** sequence bitmap. Each accepted packet (after
+> the per-port redundancy filter) flips its bit; at marker time
+> `seq_lost = (max_offset + 1) - popcount(bitmap)`. Packets that arrive on
+> the redundant port and fill a gap left by the primary port flip the same
+> bit, so cross-port redundancy is reflected directly in the count — there
+> is no false positive `seq_discont` from cross-port reorder. The
+> `port_seq_lost[]` / `port_seq_discont[]` arrays still report the per-port
+> intra-frame view for observability.
+
 **TX (any type).** `stat_epoch_drop` = app handed frame too late, slots were
 skipped (counter advanced by the number of skipped slots).
 `stat_epoch_onward` = system clock ran past `max_onward_epochs` ahead of the

ecosystem/gstreamer_plugin/README.md

@@ -16,7 +16,7 @@ To install the required GStreamer packages on Ubuntu or Debian, run the followin
 ```bash
 sudo apt update
 
-sudo apt install -y gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-tools gstreamer1.0-libav libgstreamer1.0-dev
+sudo apt install -y gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-tools gstreamer1.0-libav libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev
 ```
 
 > For MTL installation instructions please refer to  [Build Documentation](../../doc/build.md).