[MRV2][Spec Decode] Quantized draft lm_head for drafters that share the target head (#59973)

Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
This commit is contained in:
Stefano Castagnetta
2026-10-15 22:06:40 +00:00
committed by GitHub
co-authored by Kevin H. Luu
parent d92a4f25f2
commit 784d1fff0c
11 changed files with 613 additions and 3 deletions
@@ -74,3 +74,12 @@ vllm serve XiaomiMiMo/MiMo-7B-Base \
is a good default to start with.
- If your model does not support MTP, use another method such as EAGLE or draft
model speculation.
- When the MTP head shares the target's unquantized lm_head (most MTP models),
`"draft_lm_head_quantization": "fp8"` or `"nvfp4"` in `--speculative-config`
gives the drafter a quantized copy of that head (Model Runner V2). The target
still verifies with its own head, so outputs are unchanged; only the
acceptance length can move. It helps most on bandwidth-bound GPUs with a
large vocabulary. The copy costs about one byte (FP8) or 0.56 bytes (NVFP4)
per lm_head weight, e.g. 0.6 or 0.35 GiB for a 248k x 2560 head; it is
counted in the model weights, so with `--kv-cache-memory-bytes` leave room
for it.