feat: Qwen3.5 / Qwen3.5-MoE MTP speculative decoding

[sufubao]

Summary

Adds Multi-Token Prediction (MTP) speculative decoding for Qwen3.5 and Qwen3.5-MoE (hybrid full-attention + Gated Delta Net / qwen3next linear-attention models). The draft head proposes mtp_step tokens per step and the base model verifies them in a single fused forward, giving exact greedy output with single-stream/low-concurrency latency speedups.

Implemented in 5 logical layers:

• Linear-attn cache state split — separate widened-GPU vs narrow-persisted conv/SSM state so the verify path can snapshot/restore committed state.
• qwen3next GDN spec-decode verify path — _gdn_verify_kernel + hybrid dispatch and a vendored spec-decode causal_conv1d_spec Triton kernel.
• Basemodel MTP decode CUDA graphs + verify dispatch — decode graph captured with the MTP verify layout; padding grouped by lcm(unit, mtp_step+1); capture-safe (no D2H syncs).
• Scheduler MTP verify backend + accept-len transport — canonical accept_len pointer plumbing; accept-count carry committed in phase 2 (pre-forward-release) to avoid a one-step-stale read under the overlap scheduler.
• Draft models — qwen3_5_mtp / qwen3_5_moe_mtp packages (full-attn draft layer, mrope, inline mtp.* weights), non-colliding draft KV slots, backend registration.

Test Plan

• 40 new unit tests pass (cache-state split, GDN verify equivalence, spec conv kernel, draft layer/slots, MTP decode CUDA graph): pytest unit_tests/common unit_tests/models/qwen3_5 unit_tests/models/qwen3next
• E2E greedy parity: MTP output token-identical to MTP-off baseline (spec decode is exact)
• GSM8K parity gate on Qwen3.5-27B (TP4, conc-16): off 0.905 / mtp1 0.895 / mtp2 0.915
• black clean

Notes

• MTP is a low-concurrency/latency accelerator: single-stream decode speedup up to ~1.57x (mtp2, accept_len 2.43) on 27B; crossover ~conc-8 where the GPU saturates.

[gemini-code-assist]

Code Review

This pull request adds support for Qwen3.5 and Qwen3.5-Moe MTP models, integrating speculative decoding with CUDA graph support and linear attention. It introduces a spec-decode capable causal_conv1d update kernel, refactors CUDA graph capture and warmup logic to support both normal and MTP verify decode layouts, and adjusts memory management to handle dedicated draft KV slots and widened GPU conv buffers. Feedback on the changes suggests correcting a typo in the function name _nomarl_prefill_att to _normal_prefill_att in fp.py to improve code readability and maintainability.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

There appears to be a typo in the function name. _nomarl_prefill_att should likely be _normal_prefill_att. This should be corrected for better code readability and maintainability. You'll also need to update the call site for this function.

Suggested change

	def _nomarl_prefill_att(
	self, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, att_control: AttControl, alloc_func=torch.empty
	self,
	q: torch.Tensor,
	k: torch.Tensor,
	v: torch.Tensor,
	att_control: AttControl,
	alloc_func=torch.empty,
	) -> torch.Tensor:
	def _normal_prefill_att(
	self,
	q: torch.Tensor,
	k: torch.Tensor,
	v: torch.Tensor,
	att_control: AttControl,
	alloc_func=torch.empty,
	) -> torch.Tensor: