perf(reshard): deduplicate replica reads and batch receiver installation

[KavinKrishnan]

What this changes, and why it is worth 10x

A reshard refit on main today is not bandwidth-bound. On real Qwen3-30B with
Megatron EP8 publishers feeding a vLLM TP2 receiver, it moved 47.58 GiB per rank
at an effective 146 Gbps on a 400 Gbps rail, and took 5102.6 ms.

The cause is range count, not bytes. Under DP/EDP every replica publishes the
same non-expert tensors, and merge_shard_tables keeps every offer it is given.
Nothing downstream removes the duplicates, so the planner cannot use its dim-0
partitioner and one fused transfer degenerates into 809,112 small ranges.
Picking one owner per byte-identical offer lets the partitioner work again and
the same payload crosses the wire as 19,011 large ranges.

The arithmetic is the point. Removing duplicate bytes is only a 20.5% reduction
and would predict a 1.26x speedup on its own. The measured wire leg improved
5.9x. The other 4.9x is entirely the number of ranges the receiver asks for.

That is why the dedup here is unconditional rather than a tunable: on any
topology with replicas it is closer to a planning fix than an optimization.

per rank	main	with these commits
wire bytes	47.58 GiB	37.82 GiB	−20.5%
wire segments	809,112	19,011	42.6x fewer
wire leg, median	2791.3 ms	469.5 ms	5.9x
refit e2e, median	5102.6 ms	480.7 ms	10.6x
effective wire rate	146.4 Gbps	692.0 Gbps	4.7x
stage attribution	54.5%	>97%	floor is 95%

Why dropping reads here is safe

The obvious worry about a change that removes reads is that it removes something
needed. Three things argue against that:

• Dedup only collapses offers that are byte-identical in geometry, and digest,
  tensor geometry and owner checks are all preserved. A shard that differs in
  any of those is not a duplicate and is kept.
• Every measured arm compared 435/435 receiver parameters byte-exact on both
  TP ranks, max_abs_err 0.0, with zero fallback and zero conversions.
  Deliberate corruption was detected and recovery generation agreement was 1.0.
• The deduplicated payload is 40,611,246,080 bytes per rank. That is the same
  wire volume an independent campaign measured on this model three weeks
  earlier through a different harness. The duplicates were redundant, not load
  that went missing.

Review guide

Three commits, ordered as validated. They are separately reviewable, and the
smallest one carries most of the win.

#	Commit	Impl	Tests	Focus
1	perf(reshard): batch the full-pull re-slice into one _foreach_copy_	42	271	receiver.py (+34) — replaces thousands of per-tensor destination copies with one _foreach_copy_. Re-slice 302.2 → 123.0 ms against the same 6,192 copies.
2	perf(reshard): deduplicate replica reads before planning	31	217	rendezvous.py only. Start here. This is the 6x, in 31 net lines.
3	perf(reshard): build read descriptors once per plan, and time the build	99	138	receiver.py (+114) — caches descriptor lists per stable plan, invalidating on plan rebuild or wire-arm change, and adds descriptor_build_s.

On commit 3, please read the invalidation conditions rather than the cache
itself. Its direct value is small and honestly reported: descriptor_build_s
falls from 12.6 ms to 0.01 ms, about 1.5% of an 800 ms refit. It is included
because rebuilding 413,772 descriptor objects per step was happening inside no
timed stage at all, which is what held stage attribution at 54.5% and made the
breakdown unreportable. The larger arm-to-arm deltas around it are run-to-run
variance, not the cache.

Source spreading was validated alongside these and is deliberately omitted.
It cost 165 ms on the pwal installer, with the damage in the wire leg, and did
nothing measurable on mdl. With single-rank legs already at ~1250 Gbps of a
1600 Gbps fabric there is no headroom for it to win, so spreading reads across
more peers adds contention instead of bandwidth. It should be re-judged on a
fabric with headroom, not landed on these numbers.

Evidence, and what it does not yet cover

Measured on real Qwen/Qwen3-30B-A3B-Instruct-2507 BF16 tensors — not synthetic
geometry — with Megatron EP8 publishers across two GB200 nodes and a vLLM TP2
receiver on a third, over NIXL/RDMA. One warm-up plus ten measured updates per
arm.

These figures are evidence for this implementation shape. They are not a
final benchmark row for this rebased PR head, and this is not yet a canonical
PASS: it still needs three independent cold starts, fifty synchronized updates
on the selected arm, multi-step KL sanity, framework E2E alongside receiver E2E,
and a run through the production GRPO recipe. Treat it as
PERF_PASS_CORRECTNESS_PENDING.

Note also that this topology is smaller than our earlier large-scale reference
runs, so the 480.7 ms figure should not be compared against those directly.

Test plan

• pytest tests/test_reshard_refit_replica_merge.py tests/test_reshard_refit_batch_install.py tests/test_reshard_refit_descriptor_cache.py tests/test_envs.py tests/test_reshard_refit_fused_wire.py -q — 32 passed
• full client suite in an image with the VMM extension built (local run could not load _alloc_ext; that is an environment gap, not a pass)
• arm64 image pinned by digest to this PR head
• re-run the exact EP8 → TP2 Qwen3-30B correctness gate against this head
• three cold starts plus a fifty-update final arm before any published benchmark row

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.