From ffdde24f546c503c7e47c5264b180c16fd668888 Mon Sep 17 00:00:00 2001 From: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com> Date: Fri, 9 Oct 2026 13:54:40 -0700 Subject: [PATCH] [Docs] Document adaptive verification without a confidence head (#60913) Signed-off-by: Giancarlo Delfin --- .../adaptive_verification.md | 25 ++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/docs/features/speculative_decoding/adaptive_verification.md b/docs/features/speculative_decoding/adaptive_verification.md index e9314c0e09..d560b5af3e 100644 --- a/docs/features/speculative_decoding/adaptive_verification.md +++ b/docs/features/speculative_decoding/adaptive_verification.md @@ -4,7 +4,7 @@ Speculative decoding buys fewer decode steps with more compute. At batch size 1 That matters because per-position acceptance decays fast. While the GPU is memory-bound that slot is effectively free and worth the gamble; once it saturates the gamble has a real throughput cost. The crossover moves with load and with workload-dependent acceptance rates, so no static `num_speculative_tokens` is right across concurrencies. -Adaptive verification decides per step how much of the draft to verify instead. Every (request, position) draft slot is scored by its *survival probability*, the running product of that request's per-position confidences, and the highest-scoring slots are admitted until a global budget is spent. Slots compete across requests: position 5 of a confident request can outrank position 1 of a doubtful one, so one request keeps its full block while another could be trimmed after a token or two. +Adaptive verification decides per step how much of the draft to verify instead. Every (request, position) draft slot is scored by its *survival probability*, the running product of that request's per-position acceptance estimates, and the highest-scoring slots are admitted until a global budget is spent. Slots compete across requests: position 5 of a confident request can outrank position 1 of a doubtful one, so one request keeps its full block while another could be trimmed after a token or two. The budget itself comes from a cost model profiled at startup. vLLM measures what a step costs at each shape, then picks the token count that maximizes expected accepted tokens per second. @@ -12,11 +12,14 @@ The practical effect is that one configuration holds up across the whole load ra ## Support -Adaptive verification needs per-position acceptance estimates, so today it is only supported for DSpark with a **confidence head**. +Adaptive verification is supported for model-based drafters (EAGLE, MTP, DFlash, DSpark and standalone draft models). The per-position acceptance estimates come from one of two sources: + +- **Confidence head**: DSpark checkpoints that ship a trained confidence head use its predictions directly. +- **Online acceptance estimator**: every other drafter predicts acceptance from its draft logits with a small logistic model, recalibrated during serving against the target's verification results. ## Usage -It is off by default. Enable it in the speculative config: +It is off by default. Enable it in the speculative config with `enable_adaptive_verification`. For example, with DSpark: ```bash vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark \ @@ -30,6 +33,21 @@ vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark \ }' ``` +Or with DFlash on MiMo-V2.5-Pro, whose drafter ships in the checkpoint's `dflash/` subdirectory: + +```bash +hf download XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash --local-dir MiMo-V2.5-Pro-FP4-DFlash + +vllm serve MiMo-V2.5-Pro-FP4-DFlash \ + --tensor-parallel-size 4 --trust-remote-code --max-model-len auto \ + --speculative-config '{ + "method": "dflash", + "model": "MiMo-V2.5-Pro-FP4-DFlash/dflash", + "num_speculative_tokens": 8, + "enable_adaptive_verification": true + }' +``` + Set `enable_adaptive_verification: false` to verify the full block for every request. ## Requirements and limitations @@ -37,6 +55,7 @@ Set `enable_adaptive_verification: false` to verify the full block for every req - The attention backend must tolerate device-decided query lengths, since the CPU lengths only bound them from above. Backends that plan off the CPU lengths are excluded by the attention selector, and rejected at startup for models that hard-wire their backend. - Full cudagraphs are required: step costs are profiled from captured graphs, so `--enforce-eager` is rejected at startup. - Not supported with LoRA (the per-token LoRA mapping is built from CPU-side boundaries) or pipeline parallelism (cost curves and confidences exist only on the last rank). +- Without a confidence head, `use_local_argmax_reduction` is rejected. The estimator needs the full draft logits, which local argmax reduction never materializes. ## Tuning the cost profile