Fix(spec decode): catch up trim bug#335
Conversation
… NextN NaN) Signed-off-by: rjzhb <rjzhb222@163.com>
…ch-up-trim-bug Signed-off-by: rjzhb <rjzhb222@163.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c87743f7ec
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0c7d1ef3b5
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
a0a7a59 to
20d1c61
Compare
…ion_trim flag Signed-off-by: rjzhb <rjzhb222@163.com>
20d1c61 to
edac557
Compare
LorrinWWW
left a comment
There was a problem hiding this comment.
This is a correctness fix for spec. And we will have a follow-up refactoring to reduce / remove the extra if-else branching.
Summary
fix #217 that broke spec-decode catch-up on Qwen NextN and DeepSeek V3 NextN:
seq_lenstrim indrafter/eagle.pyllama_eagle3.pyThe trim assumed q was sliced to 1 row before the kernel. Llama Eagle3 had the matching q-slice; Qwen NextN and DeepSeek V3 NextN didn't — trim alone left
q_len > cache_seqlens, giving negative kernel positions, all-masked softmax, NaN, andaccept_rate=0.Solution is to add a temp flag: pre_attention_trim.
NOTICE: this flag would be removed in the future refactor PR once adding pre_attention_trim feature to both Qwen NextN and DeepSeek V3 NextN.
Test Plan
accept_rate > 0restored.