feat(qwen35): add transactional speculative verifier#667
Conversation
|
Follow-up split plan:
Each PR will keep the default Qwen3.5 path unchanged unless DFlash is enabled. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c65ada7ecd
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| req.token_ids.len() >= 2, | ||
| "Qwen3.5 speculative verify request {} needs [current, draft...]", | ||
| req.request_id.get() |
There was a problem hiding this comment.
Allow single-token verify spans at the budget edge
When a speculative step is clamped to the last remaining output token, the verify span can legitimately contain only the current token; the existing speculative scheduler does exactly this with token_ids.truncate(remaining) when remaining == 1 (openinfer-qwen3/src/scheduler/plan.rs:188-198). A one-token target verify should consume the current token and return the model's single posterior token, and the rest of this verifier/commit path already handles span.len() == 1; this guard would instead turn the final speculative step into an error when Qwen3.5 is wired into the same flow.
Useful? React with 👍 / 👎.
Summary
This PR is the target-verifier slice split out of #626. It makes Qwen3.5 speculative verification reviewable independently from the draft model and serving integration.
It verifies
[current token, draft tokens...]in one target forward pass, then commits the longest greedy-matching prefix plus one target token. Qwen3.5 hybrid state is handled as one transaction: paged KV, recurrent state, convolution state, and CUDA Graph slot state.What changed
Validation
On one RTX 5090 with Qwen3.5-4B:
Scope
This PR does not add the DFlash draft model, scheduler/server opt-in wiring, sampling support, or a performance claim. Those parts remain in follow-up PRs split from #626.