[Spec Decode] DSpark confidence-scheduled verification (#47808)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com> Signed-off-by: Lucas Wilkinson <wilkinson.lucas@gmail.com> Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: OpenAI Codex <noreply@openai.com>
This commit is contained in:
co-authored by
OpenAI Codex
Claude Opus 5
Benjamin Chislett
mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Nick Hill
OpenAI Codex
parent
fe889ac925
commit
7f7a32cfec
@@ -18,6 +18,7 @@ vLLM supports a variety of methods of speculative decoding. Model-based methods
|
||||
- [Hidden State Extraction](extract_hidden_states.md)
|
||||
- [Custom Proposer Backend (Experimental)](#custom-proposer-backend-experimental)
|
||||
- [Dynamic Speculative Decoding](dynamic_speculative_decoding.md)
|
||||
- [Adaptive Verification](adaptive_verification.md)
|
||||
|
||||
## Method Selection at a Glance
|
||||
|
||||
@@ -35,6 +36,7 @@ depend on your model family, traffic pattern, hardware, and sampling settings.
|
||||
| Suffix decoding | Low to medium gain | Medium gain | No extra draft model; dynamic speculation depth. |
|
||||
| Custom Proposer | Varies | Varies | Bring your own proposer class (experimental). |
|
||||
| Dynamic Speculative Decoding | High gain | Higher than base SD method | Useful for RL or workload with fluctuating QPS |
|
||||
| Adaptive Verification | High gain | Higher than base SD method | Sizes verification per request from drafter confidence; currently DSpark only. |
|
||||
|
||||
For reproducible measurements in your environment, use
|
||||
[`examples/features/speculative_decoding/spec_decode_offline.py`](../../../examples/features/speculative_decoding/spec_decode_offline.py)
|
||||
|
||||
Reference in New Issue
Block a user