fix(spec): restore sentinel padding + context clamps for every drafter#618
Open
jasl wants to merge 1 commit into
Open
fix(spec): restore sentinel padding + context clamps for every drafter#618jasl wants to merge 1 commit into
jasl wants to merge 1 commit into
Conversation
Upstream lightseekorg#594 narrowed four MTP/spec safety mechanisms to DFLASH only, regressing every other drafter: - Decode-graph padding rows went back to req-pool slot 0, aliasing the live first request. Harmless for the plain verify decode, but a drafter in the captured graph re-derives per-row state from the pool index: Eagle/MTP padding rows read request 0's committed length and resolve draft-KV write slots through request 0's live frontier -- a write-write race plus wild paged-indexer accesses whenever the batch pads (bs not in capture_bs, i.e. exactly at mid-serve completions and admission ramps). On GB10 2-node DSv4-Flash MTP2 this crashed the arthur conc8 recall gate with a CUDA illegal memory access (conc1 and nomtp conc8 pass; eager MTP2 conc8 passes; pre-lightseekorg#594 passed 16/16). Restore the unconditional sentinel (max_req_pool_size) padding. - _cap_accept_to_context_len (both verify sites), the committed-length clamp, and draft_block_reservation_slack were DFLASH-gated. All three guard any drafter near context_len; restore them for every spec algorithm (inert mid-context, prevents accept/commit overshoot and reservation overflow at the limit). Signed-off-by: jasl <jasl9187@hotmail.com>
4629d45 to
437d828
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Disclaimer
The PR body and the code were written by Claude and reviewed by me.
I'm hacking TokenSpeed to support SM10x, and I believe this patch is valid for all archs.
Summary
#594 narrowed four MTP/spec safety mechanisms to
spec_algo == "DFLASH", regressing every other drafter (Eagle/MTP/EAGLE3):seq_len == 1), but any drafter inside the captured graph re-derives per-row state from the pool index: Eagle/MTP padding rows read request 0's committed length and resolve draft-KV write slots through request 0's live frontier — a write-write race plus wild paged-indexer accesses whenever the batch pads (bs ∉ capture_bs, i.e. exactly at mid-serve completions and admission ramps). The pre-fix(perf): fix qwen3.5 tp8 agentic ci perf regression #594 comment documented this hazard verbatim. Restored: unconditional sentinel (max_req_pool_size) padding._cap_accept_to_context_len(both verify sites),_clamp_committed_to_context_len, anddraft_block_reservation_slackwere DFLASH-gated. All three guard any drafter nearcontext_len(accept/commit overshoot, reservation overflow → KV drop). Restored for every spec algorithm — arithmetically inert mid-context, load-bearing at the limit.Evidence
Observed on a 2-node DGX Spark (GB10, TP=2) serving DeepSeek-V4-Flash with MTP2 (
num_speculative_tokens=2, capture_bs=[1,2,4,7,8]):illegal memory access→ NCCL watchdog kills the process group, mid-way through an 8-concurrent long-context recall gate (reproduced twice, both times right as batch sizes went ragged: 3/5/6 from completions/admissions).Validation
With this fix, on the same 2-node GB10 MTP2 serve:
context_len).Notes