Skip to content
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/configuration/server.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ the values accepted by the bundled `tokenspeed-smg` package.
| Parameter | Purpose |
| --- | --- |
| `--speculative-config` | JSON speculative decoding configuration. |
| `--speculative-algorithm` | Speculative algorithm, such as `EAGLE3` or `MTP`. |
| `--speculative-algorithm` | Speculative algorithm, such as `EAGLE3`, `MTP`, or `DFLASH`. |
| `--speculative-draft-model-path` | Draft model path or repo ID. |
| `--speculative-draft-model-quantization` | Draft model quantization. Defaults to `unquant`. |
| `--speculative-num-steps` | Number of draft model steps. Defaults to `3`. |
Expand Down
32 changes: 32 additions & 0 deletions docs/recipes/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,38 @@ tokenspeed serve nvidia/Kimi-K2.5-NVFP4 \
For K2.6, keep the same parameter shape and change the checkpoint and parser
only if the model card requires a different value.

To enable a compatible DFlash draft model, keep the target launch shape and add
the draft model path plus DFlash speculative decoding options:

```bash
tokenspeed serve nvidia/Kimi-K2.6-NVFP4 \
--served-model-name kimi-k2.6 \
--trust-remote-code \
--max-model-len 262144 \
--kv-cache-dtype fp8 \
--quantization nvfp4 \
--tensor-parallel-size 4 \
--enable-expert-parallel \
--chunked-prefill-size 8192 \
--max-num-seqs 256 \
--attention-backend tokenspeed_mla \
--moe-backend flashinfer_trtllm \
--reasoning-parser kimi_k25 \
--tool-call-parser kimik2 \
--speculative-algorithm DFLASH \
--speculative-draft-model-path /path/to/kimi-k2.6-dflash \
--speculative-num-draft-tokens 8 \
--speculative-num-steps 7 \
--drafter-attention-backend fa4 \
--host 0.0.0.0 \
--port 8000
```

Known limitation: native TokenSpeed DFlash currently uses full-history draft
attention. It does not yet expose an equivalent of SGLang's
`--speculative-dflash-draft-window-size`; add such a flag before relying on
bounded draft attention for long-context deployments.

## GLM5 / GLM5.2

GLM5 launches usually need remote code, long context, expert parallelism, FP8 KV
Expand Down
5 changes: 5 additions & 0 deletions python/tokenspeed/runtime/execution/cuda_graph_wrapper.py
Original file line number Diff line number Diff line change
Expand Up @@ -430,6 +430,11 @@ def run_once():
torch.cuda.synchronize()
dist.barrier()

# Warmups can switch a backend back to eager metadata objects. Restore
# the graph-backed metadata immediately before capture so replay-time
# metadata refreshes update the same tensors recorded by the graph.
self._init_capture_metadata(bs)

# Fill sampler buffers OUTSIDE the capture so RNG ops aren't recorded.
if self.sampling_backend is not None:
self.sampling_backend.prepare_capture(
Expand Down
Loading
Loading