iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[Doc] Clarify supported keys for --speculative-config (#40455)

Browse Source 
Signed-off-by: Wangxiaoxiaoa <Wangxiaoxiaoa@users.noreply.github.com>
Co-authored-by: Wangxiaoxiaoa <Wangxiaoxiaoa@users.noreply.github.com>
This commit is contained in:
xiao
2026-04-22 04:36:17 -07:00
committed by GitHub
co-authored by Wangxiaoxiaoa
parent a250f1bd5f
commit ecbe42e991
4 changed files with 105 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/features/speculative_decoding/draft_model.md
+8 -4
View File
@@ -33,9 +33,9 @@ vllm serve Qwen/Qwen3-4B-Thinking-2507 \
--port 8000 \
--seed 42 \
-tp 1 \
--max_model_len 2048 \
--gpu_memory_utilization 0.8 \
--speculative_config '{"model": "Qwen/Qwen3-0.6B", "num_speculative_tokens": 5, "method": "draft_model"}'
--max-model-len 2048 \
--gpu-memory-utilization 0.8 \
--speculative-config '{"model": "Qwen/Qwen3-0.6B", "num_speculative_tokens": 5, "method": "draft_model"}'
```
The code used to request as completions as a client remains unchanged:
@@ -77,4 +77,8 @@ The code used to request as completions as a client remains unchanged:
```
!!! warning
Note: Please use `--speculative_config` to set all configurations related to speculative decoding. The previous method of specifying the model through `--speculative_model` and adding related parameters (e.g., `--num_speculative_tokens`) separately has been deprecated.
Note: Please use `--speculative-config` to set all configurations related
to speculative decoding. The previous method of specifying the model
through `--speculative-model` and adding related parameters such as
`--num-speculative-tokens` separately has been deprecated. For supported
keys and examples, see the [`--speculative-config` schema](README.md#--speculative-config-schema).