Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/configuration/server.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ the values accepted by the bundled `tokenspeed-smg` package.
| Parameter | Purpose |
| --- | --- |
| `--speculative-config` | JSON speculative decoding configuration. |
| `--speculative-algorithm` | Speculative algorithm, such as `EAGLE3` or `MTP`. |
| `--speculative-algorithm` | Speculative algorithm, such as `EAGLE3`, `MTP`, or `DFLASH`. |
| `--speculative-draft-model-path` | Draft model path or repo ID. |
| `--speculative-draft-model-quantization` | Draft model quantization. Defaults to `unquant`. |
| `--speculative-num-steps` | Number of draft model steps. Defaults to `3`. |
Expand Down
32 changes: 32 additions & 0 deletions docs/recipes/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,38 @@ tokenspeed serve nvidia/Kimi-K2.5-NVFP4 \
For K2.6, keep the same parameter shape and change the checkpoint and parser
only if the model card requires a different value.

To enable a compatible DFlash draft model, keep the target launch shape and add
the draft model path plus DFlash speculative decoding options:

```bash
tokenspeed serve nvidia/Kimi-K2.6-NVFP4 \
--served-model-name kimi-k2.6 \
--trust-remote-code \
--max-model-len 262144 \
--kv-cache-dtype fp8 \
--quantization nvfp4 \
--tensor-parallel-size 4 \
--enable-expert-parallel \
--chunked-prefill-size 8192 \
--max-num-seqs 256 \
--attention-backend tokenspeed_mla \
--moe-backend flashinfer_trtllm \
--reasoning-parser kimi_k25 \
--tool-call-parser kimik2 \
--speculative-algorithm DFLASH \
--speculative-draft-model-path /path/to/kimi-k2.6-dflash \
--speculative-num-draft-tokens 8 \
--speculative-num-steps 7 \
--drafter-attention-backend fa4 \
--host 0.0.0.0 \
--port 8000
```

Known limitation: native TokenSpeed DFlash currently uses full-history draft
attention. It does not yet expose an equivalent of SGLang's
`--speculative-dflash-draft-window-size`; add such a flag before relying on
bounded draft attention for long-context deployments.

## GLM5 / GLM5.2

GLM5 launches usually need remote code, long context, expert parallelism, FP8 KV
Expand Down
5 changes: 5 additions & 0 deletions python/tokenspeed/runtime/execution/cuda_graph_wrapper.py
Original file line number Diff line number Diff line change
Expand Up @@ -422,6 +422,11 @@ def run_once():
torch.cuda.synchronize()
dist.barrier()

# Warmups can switch a backend back to eager metadata objects. Restore
# the graph-backed metadata immediately before capture so replay-time
# metadata refreshes update the same tensors recorded by the graph.
self._init_capture_metadata(bs)

# Fill sampler buffers OUTSIDE the capture so RNG ops aren't recorded.
if self.sampling_backend is not None:
self.sampling_backend.prepare_capture(
Expand Down
Loading
Loading