DeepSeek V4 : Failed to predict at token position

[alex-ie]

IIRC:

1. Release notes 1.117.1 did not explicitly state DeepSeek V4 support, but "new models support"
2. Release date was after DeepSeek V4 merge in llama.cpp

Should kcpp support DeepSeek V4 already or not?

Describe the Issue
Prefill is processed, 1st response generated, but at processing of 2nd turn prompt I got (both times when I have tried to run DS4 Flash via kcpp):

Failed to predict at token position

Prompt processing debug output stopped at this message.

Additional Information:
1.117.1, Linux

[LostRuins]

Yes it should work although it has not been specifically tested for v4 flash. Which download did you use, and can I get a bit more of the error log

[alex-ie]

MXFP4 bartowski. I have run it via llama-server and there were no issues (2nd turn prompt got expected response). The kcpp log contains (typed manually, may contain typos but I have checked the numbers in the quote):

...
n_ctx=64256
...
This model does not use SWA
....

Note: SWA context cannot be reused (Desired n_past=940, SWA lowest n_past=943), to avoid this, disable SWA or increase SWA padding), output may degrade.

[Debug: embd_inp=896 n_past=48 Context size = 48]


Processing prompt (0 / 896 tokens)
Processing prompt (896 / 896 tokens)init: the tokens of sequence 0 in the input batch have inconsistent sequence position:
 - the last position stored in the memory module of the context (i.e. the KV cache) for sequence 0 is X = 943
 - the tokens for sequence 0 in the input batch have a starting position of Y = 48
 it is required that the sequence positions remain consecutive: Y = X + 1
decode: failed to initialize batch
llama_decode: failed to decode, ret = -1

Failed to predict at token position 48! Check you context buffer sizes!

[alex-ie]

I have tried to run it with --noswa - same error on 2nd turn. Very short prompts used, got embd_inp=9, n_past=Context Size=X=Y=38.

P.S. SWA mismatch noted earlier might be another issue, might it not?

[DilanRG]

I traced the position mismatch in the log. This is not caused by SWA: DeepSeek V4's cache rejects the partial rollback KCPP 1.117.1 attempted, but KCPP ignored that failure and then decoded at Y=48 while the cache still ended at X=943.\n\nConcedo's commit a25af27 on concedo_experimental adds a fallback for this path: if the cache cannot be rewound, it clears the cache and reprocesses the full prompt from position 0.\n\nCould you test a build containing that commit with both cases you reported (the 943 -> 48 case and the short X=Y=38 case), with and without --noswa? The fallback should log:\n\n\nKV cache could not be rewound for prompt reuse; reprocessing full prompt instead\n\n\nand the second turn should then complete normally.

[LostRuins]

Can you try the latest rolling builds and let me know if its fixed? https://github.com/LostRuins/koboldcpp/releases/tag/rolling

[Tacx79]

Rolling build seems to work, but it's reprocessing the prompt on every generation. Also I'm not sure if this is the usual behaviour, but using SWA halves the processing speed

[LostRuins]

Yeah, this model doesn't support partial rewinds, so it requires a full reprocess.

[alex-ie]

Yeah, this model doesn't support partial rewinds, so it requires a full reprocess.

But why were "partial rewinds" even attempted while I have not changed (no edits) previous turns? 2nd turn prompt does not seems to need to trigger "include only newest thinking in context" editing either.

I also recall that Qwen models does not support rewind so cache slots or something are used to avoid full reprocessing. Could we use same for DS4?

[gpez-git]

I'm having this same issue. Sometimes it doesn't require a full reprocess, sometimes it gets stuck with the "Failed to predict at token" error and is unable to continue. Can confirm --noswa allows it to work, but as mentioned every turn is a full reprocess making it almost unusable at high contexts.

Have not yet tried the rolling build, happy to do so or wait for 1.118.

[LostRuins]

Rolling build is here https://github.com/LostRuins/koboldcpp/releases/tag/rolling