Skip to content

fix(spec): restore sentinel padding + context clamps for every drafter#618

Open
jasl wants to merge 1 commit into
lightseekorg:mainfrom
jasl:fix/spec-sentinel-padding
Open

fix(spec): restore sentinel padding + context clamps for every drafter#618
jasl wants to merge 1 commit into
lightseekorg:mainfrom
jasl:fix/spec-sentinel-padding

Conversation

@jasl

@jasl jasl commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Disclaimer

The PR body and the code were written by Claude and reviewed by me.
I'm hacking TokenSpeed to support SM10x, and I believe this patch is valid for all archs.

Summary

#594 narrowed four MTP/spec safety mechanisms to spec_algo == "DFLASH", regressing every other drafter (Eagle/MTP/EAGLE3):

  1. Decode-graph padding rows went back to req-pool slot 0, aliasing the live first request. Harmless for the plain verify decode (seq_len == 1), but any drafter inside the captured graph re-derives per-row state from the pool index: Eagle/MTP padding rows read request 0's committed length and resolve draft-KV write slots through request 0's live frontier — a write-write race plus wild paged-indexer accesses whenever the batch pads (bs ∉ capture_bs, i.e. exactly at mid-serve completions and admission ramps). The pre-fix(perf): fix qwen3.5 tp8 agentic ci perf regression #594 comment documented this hazard verbatim. Restored: unconditional sentinel (max_req_pool_size) padding.
  2. _cap_accept_to_context_len (both verify sites), _clamp_committed_to_context_len, and draft_block_reservation_slack were DFLASH-gated. All three guard any drafter near context_len (accept/commit overshoot, reservation overflow → KV drop). Restored for every spec algorithm — arithmetically inert mid-context, load-bearing at the limit.

Evidence

Observed on a 2-node DGX Spark (GB10, TP=2) serving DeepSeek-V4-Flash with MTP2 (num_speculative_tokens=2, capture_bs=[1,2,4,7,8]):

  • Crash: CUDA illegal memory access → NCCL watchdog kills the process group, mid-way through an 8-concurrent long-context recall gate (reproduced twice, both times right as batch sizes went ragged: 3/5/6 from completions/admissions).
  • Isolation matrix: conc1 ✅ · nomtp conc8 ✅ 16/16 · eager MTP2 conc8 ✅ 16/16 · graphed MTP2 conc8 ❌ IMA — pinning the fault to MTP × decode-graph replay at padded batch sizes.
  • Pre-fix(perf): fix qwen3.5 tp8 agentic ci perf regression #594 (same serving stack on the older base) the identical gate passed 16/16 repeatedly.

Validation

With this fix, on the same 2-node GB10 MTP2 serve:

  • arthur 900-line recall gate conc8: 16/16, twice consecutively, zero IMA; conc1 2/2.
  • Decode/prefill throughput unchanged (pinned llama-benchy standard, 3 depths — all within noise of the pre-fix(perf): fix qwen3.5 tp8 agentic ci perf regression #594-regression numbers).
  • nomtp + DFLASH paths unaffected (sentinel row is zero-init; the clamps are identity ops away from context_len).

Notes

  • The padding-row hazard reproduces for any non-DFLASH drafter whose draft runs inside the decode graph; DSv3/GLM MTP configurations are exposed the same way.

@jasl jasl requested a review from a team as a code owner July 9, 2026 03:40
Upstream lightseekorg#594 narrowed four MTP/spec safety mechanisms to DFLASH only,
regressing every other drafter:

- Decode-graph padding rows went back to req-pool slot 0, aliasing the
  live first request. Harmless for the plain verify decode, but a
  drafter in the captured graph re-derives per-row state from the pool
  index: Eagle/MTP padding rows read request 0's committed length and
  resolve draft-KV write slots through request 0's live frontier -- a
  write-write race plus wild paged-indexer accesses whenever the batch
  pads (bs not in capture_bs, i.e. exactly at mid-serve completions and
  admission ramps). On GB10 2-node DSv4-Flash MTP2 this crashed the
  arthur conc8 recall gate with a CUDA illegal memory access (conc1 and
  nomtp conc8 pass; eager MTP2 conc8 passes; pre-lightseekorg#594 passed 16/16).
  Restore the unconditional sentinel (max_req_pool_size) padding.

- _cap_accept_to_context_len (both verify sites), the committed-length
  clamp, and draft_block_reservation_slack were DFLASH-gated. All three
  guard any drafter near context_len; restore them for every spec
  algorithm (inert mid-context, prevents accept/commit overshoot and
  reservation overflow at the limit).

Signed-off-by: jasl <jasl9187@hotmail.com>
@jasl jasl force-pushed the fix/spec-sentinel-padding branch from 4629d45 to 437d828 Compare July 9, 2026 03:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant