perf(deepseek-v4): select two-stage fused mHC launch by token count#661
Draft
Xiangyi1996 wants to merge 3 commits into
Draft
perf(deepseek-v4): select two-stage fused mHC launch by token count#661Xiangyi1996 wants to merge 3 commits into
Xiangyi1996 wants to merge 3 commits into
Conversation
The vendored fused_hc op exposes backend/tile_n/num_k_splits launch parameters that the wrapper previously hardcoded: allinone-fma for M<=32 and allinone-mma above. A 768-combination sweep on B200 (graph-replay timed, parity-checked against the composed two-stage path) found: * backend=1 (fma_ksplit + mhcBigFuseKernel), tile_n=2, num_k_splits=2 runs 15.2us/call at M=32 vs 22.2us for the current default (-31%). * the allinone-mma backend selected for M>32 runs 206-241us at every M; the routing cliff fixed by the previous commit is a bad backend choice, not a large-M property. backend=1 with tile_n=4 stays flat (18-24us). * the k-split backend accumulates num_k_splits x M partial rows into the y_acc/r_acc workspaces. The fused workspace sized them at M rows, so the spill corrupted small-M outputs and crashed a production p8 run with cudaErrorIllegalAddress; accumulators now reserve FUSED_HC_MAX_K_SPLITS x max_bs rows (a few KB per buffer set). Certified on DeepSeek-V4-Pro TP8 SWE-smith golden p8 (b200-79, order- reversed pairs vs a 7-run baseline plateau): TPOT -4~6%, steady throughput +6~8%, acceptance rate unchanged, 189/189 completions in all runs. An exact decode trace shows the mHC kernel category alone moving (3.43 -> 2.87 ms/step) with every other category flat. Signed-off-by: Xiangyi Zhang <xiangyiz@nvidia.com>
Xiangyi1996
force-pushed
the
xiangyi/v4-mhc-fused-backend-select
branch
from
a826592 to
fad2253
Compare
Sweeping bigfuse_block_size (fixed at the kernel default in the first
sweep) over the selected launch configs on B200: 512 improves the
two-stage path at every production decode shape (graph-replay
microbench): 15.1 -> 14.7us at M=32, 11.9 -> 9.6us at M=16,
17.9 -> 17.2us at M=48. Endpoint-neutral by magnitude (~0.1 ms/step at
p8); claims are kernel-level only.
The same sweep ran the full backend=0 (atomic-routing GEMM) tile grid
at M in {32, 64}: nothing beats the backend=1 table at production
shapes, so the atomic path stays unselected.
Signed-off-by: Xiangyi Zhang <xiangyiz@nvidia.com>
…ounts With contract-sized accumulator workspaces the two-stage k-split path wins at every measured M, not just M>=13: 7.6-9.2us vs 9.7-11.4us for the allinone default at M in [1, 12] (graph-replay microbench, parity within 0.5% of the composed reference at every point). The earlier small-M gate guarded against wrong values that were actually the workspace undersizing corrupting the k-split spill, so it can go. Low-concurrency decode (p1/p2, M=4/8) sits on this path and also drops the per-call fhcZeroWorkspacesKernel prologue the allinone backend launches; estimated ~0.3-0.45 ms/step at p1/p2 (kernel-level claim, to be verified by decode-trace kernel counts). Signed-off-by: Xiangyi Zhang <xiangyiz@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #622 (and transitively #600). Replaces the hardcoded fused-mHC backend selection with a measured launch table, and fixes the fused-workspace accumulator sizing that the k-split backend requires.
What the sweep found
The vendored
mhc_fused_hcop exposesbackend/tile_n/num_k_splitslaunch parameters that the wrapper hardcoded (allinone-fma for M≤32, allinone-mma above). Sweeping all 768 combinations on B200 (graph-replay timed, parity-checked against the composed two-stage path):backend=1 is the two-stage
fused_pmap_gemm_fma_ksplit+mhcBigFuseKernelorganization (profiler-verified). The M>32 cliff that #622 routes around is a bad backend choice — allinone-mma is 206-241µs at every M — not a large-M property.The workspace contract
The k-split backend accumulates
num_k_splits × Mpartial rows intoy_acc/r_acc. The fused workspace sized them at M rows; the spill silently corrupted small-M outputs in isolation and crashed a production p8 run withcudaErrorIllegalAddress. Accumulators now reserveFUSED_HC_MAX_K_SPLITS × max_bsrows (a few KB per buffer set), with the contract documented on the exported constant.Endpoint certification (DeepSeek-V4-Pro TP8, SWE-smith golden p8, b200-79)
Order-reversed same-day pairs against a 7-run baseline plateau (TPOT 12.48-12.88ms):
TPOT −4
6%, steady throughput +68%, acceptance unchanged. An exact decode trace confirms single-variable causality: only the mHC kernel category moves (3.43 → 2.87 ms/step), every other category flat; steady-window cadence −4.6% matches the endpoint TPOT delta.Tests
Remaining headroom
In-snapshot best (15.2µs @m=32) vs the TRT two-stage reference (13.4µs) leaves ~0.24ms/step — pursued separately via the atomic-GEMM backend tile tuning / vendor sync.
Update 2026-07-13: full-curve endpoint data + big-fuse block tuning
Full golden-protocol curve with this PR's launch table (b200-79, same protocol/dataset as the baselines; C = base stack + this PR, measured before the big-fuse commit):
p4 and p8 both benefit (decode M≥13 hits the new table); p1/p2 are unchanged as expected — their decode shapes (M=4/8) stay on the default backend, confirming the launch table only moves the paths it claims to.
Second commit (
a3b42c0):bigfuse_block_size=512over the kernel default — 15.1→14.7µs at M=32, 11.9→9.6µs at M=16, 17.9→17.2µs at M=48 (graph-replay microbench). Endpoint-neutral by magnitude (~0.1 ms/step at p8); kernel-level claim only. The same sweep ran the full backend=0 (atomic-routing GEMM) tile grid at M∈{32,64}: nothing beats the backend=1 table at production shapes, which narrows the remaining vendor-sync headroom to ~1.3µs/call at M=32 (~0.16 ms/step).