Skip to content

feat(qwen35): add transactional speculative verifier#667

Draft
CAICAIIs wants to merge 5 commits into
openinfer-project:mainfrom
CAICAIIs:feat/qwen35-speculative-verifier
Draft

feat(qwen35): add transactional speculative verifier#667
CAICAIIs wants to merge 5 commits into
openinfer-project:mainfrom
CAICAIIs:feat/qwen35-speculative-verifier

Conversation

@CAICAIIs

Copy link
Copy Markdown
Collaborator

Summary

This PR is the target-verifier slice split out of #626. It makes Qwen3.5 speculative verification reviewable independently from the draft model and serving integration.

It verifies [current token, draft tokens...] in one target forward pass, then commits the longest greedy-matching prefix plus one target token. Qwen3.5 hybrid state is handled as one transaction: paged KV, recurrent state, convolution state, and CUDA Graph slot state.

What changed

  • Adds batched target verification and reusable verification buffers.
  • Adds commit, partial replay, and rollback for Qwen3.5 hybrid state.
  • Uses checked projection calls so launch failures reach rollback.
  • Covers accept-all, reject-first, prefix acceptance, KV page boundaries, injected failures, and replay of a graph captured before the transaction.
  • Keeps the verifier on one CUDA stream and rejects stream overrides before mutation.

Validation

On one RTX 5090 with Qwen3.5-4B:

  • Qwen3.5 feature check and Clippy passed.
  • Speculative verifier tests: 11 passed.
  • HF golden gate: 2 passed, including the 4097/8192-token fixtures.
  • Scheduler E2E: 1 passed.
  • Local formatting and diff checks passed.

Scope

This PR does not add the DFlash draft model, scheduler/server opt-in wiring, sampling support, or a performance claim. Those parts remain in follow-up PRs split from #626.

@CAICAIIs

Copy link
Copy Markdown
Collaborator Author

Follow-up split plan:

  • Drafter PR: add DFlash draft-model loading and draft-forward correctness gates on top of this verifier.
  • Serving integration PR: wire the opt-in scheduler/server flow, explicit fallback and admission rules, then add same-host benchmark and profile evidence.

Each PR will keep the default Qwen3.5 path unchanged unless DFlash is enabled.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c65ada7ecd

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread openinfer-qwen35-4b/src/speculative.rs Outdated
Comment on lines +537 to +539
req.token_ids.len() >= 2,
"Qwen3.5 speculative verify request {} needs [current, draft...]",
req.request_id.get()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Allow single-token verify spans at the budget edge

When a speculative step is clamped to the last remaining output token, the verify span can legitimately contain only the current token; the existing speculative scheduler does exactly this with token_ids.truncate(remaining) when remaining == 1 (openinfer-qwen3/src/scheduler/plan.rs:188-198). A one-token target verify should consume the current token and return the model's single posterior token, and the rest of this verifier/commit path already handles span.len() == 1; this guard would instead turn the final speculative step into an error when Qwen3.5 is wired into the same flow.

Useful? React with 👍 / 👎.

@CAICAIIs CAICAIIs marked this pull request as draft July 13, 2026 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant