[SpecDecode] Add LiLiCorr drafter (#57934)
Signed-off-by: Andrii Skliar <askliar@nvidia.com> Signed-off-by: Andrii Skliar <andreyws96@gmail.com> Co-authored-by: Andrii Skliar <askliar@nvidia.com> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
This commit is contained in:
co-authored by
Andrii Skliar
Codex
Benjamin Chislett
parent
25b0add7b8
commit
73a78e6f1f
@@ -9,6 +9,7 @@ To train your own draft models for optimized speculative decoding, see [vllm-pro
|
||||
vLLM supports a variety of methods of speculative decoding. Model-based methods such as EAGLE, MTP, draft models, PARD and MLP provide the best latency reduction, while simpler methods such as n-gram and suffix decoding provide modest speedups without increasing workload during peak traffic.
|
||||
|
||||
- [EAGLE](eagle.md)
|
||||
- [LiLiCorr](lilicorr.md)
|
||||
- [Multi-Token Prediction (MTP)](mtp.md)
|
||||
- [Draft Model](draft_model.md)
|
||||
- [Parallel Draft Model (PARD)](parallel_draft_model.md)
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
# LiLiCorr
|
||||
|
||||
LiLiCorr uses a DFlash backbone to produce per-position candidates, then scores
|
||||
candidate combinations with a learned correlator. It requires a compatible
|
||||
trained checkpoint and Model Runner V2, which is selected automatically.
|
||||
Both plain and grouped-convolution checkpoints are supported.
|
||||
|
||||
## Usage
|
||||
|
||||
For a checkpoint trained with block size 16:
|
||||
|
||||
```bash
|
||||
vllm serve /path/to/target --dtype bfloat16 \
|
||||
--speculative-config '{"method":"dflash","model":"/path/to/lilicorr","num_speculative_tokens":15,"draft_sample_method":"probabilistic"}'
|
||||
```
|
||||
|
||||
We strongly recommend explicitly setting `num_speculative_tokens` to the
|
||||
checkpoint's trained `block_size - 1` (for example, `15` for `block_size=16`).
|
||||
The trained block includes one anchor position. Other supported lengths are
|
||||
available, but may degrade acceptance length unpredictably; benchmark them with
|
||||
your checkpoint and workload before using them.
|
||||
|
||||
Shorter drafts, down to one token, use the corresponding prefix of the learned
|
||||
slot embeddings and attention bias without changing checkpoint parameter shapes.
|
||||
Lengths above `block_size - 1` are unsupported because the checkpoint has no
|
||||
learned slot embeddings for those positions.
|
||||
|
||||
Fixed-length drafting with shorter verification is a separate option provided by
|
||||
adaptive verification (`enable_adaptive_verification`). LiLiCorr shares the
|
||||
candidate-sampling and acceptance-estimation path, but this combination has not
|
||||
been validated end-to-end. Adaptive verification also requires compatible target
|
||||
attention backends and variable-length CUDA graph support; do not assume it works
|
||||
with Mamba2 targets.
|
||||
|
||||
## Proposal sampling
|
||||
|
||||
- `"draft_sample_method":"greedy"` selects the highest-scoring candidate at each
|
||||
step of the correlator's conditional walk.
|
||||
- `"draft_sample_method":"probabilistic"` samples from those conditional scores
|
||||
using the request's temperature and supplies the realized proposal
|
||||
distributions to the rejection sampler.
|
||||
|
||||
These settings control proposals; they do not replace the target model's
|
||||
sampling settings. With request temperature zero, probabilistic proposals also
|
||||
use argmax. For nonzero temperatures, benchmark both modes with your checkpoint
|
||||
and workload: compare acceptance length and end-to-end throughput or latency.
|
||||
Neither mode is universally faster.
|
||||
|
||||
The default rejection method is `"rejection_sample_method":"standard"`.
|
||||
The shared Model Runner V2 rejection sampler also exposes `"block"`, but
|
||||
LiLiCorr-specific GPU correctness and performance validation for that combination
|
||||
is still pending. Do not assume that changing the rejection method improves
|
||||
throughput; evaluate it separately from the proposal sampling mode.
|
||||
|
||||
## Checkpoint requirements
|
||||
|
||||
The checkpoint must declare `LiLiCorrDraftModel` and include all `lilicorr_*`
|
||||
geometry fields and trained head weights. Geometry is read from `dflash_config`.
|
||||
Convolution tensors must match the configured `conv_kernel_size` and
|
||||
`conv_group_size`. Target input embeddings must be available on the draft rank.
|
||||
By default, candidates use the target LM head, which must also be available there.
|
||||
A checkpoint with top-level `"has_own_lm_head": true` instead uses its own
|
||||
`lm_head` with the draft quantization configuration and exclusions. Its weights
|
||||
must be present; for ModelOpt NVFP4 this includes `lm_head.weight`,
|
||||
`lm_head.weight_scale`, and `lm_head.weight_scale_2`. The owned head is preserved
|
||||
and is not replaced by the target head. Candidate token embeddings still come
|
||||
from the target.
|
||||
|
||||
Correlator linear layers and convolution kernel projections use the draft
|
||||
quantization configuration and its module exclusions. The following parameters
|
||||
must remain in floating-point model dtype:
|
||||
|
||||
- Lattice QKV `in_proj_weight` and `in_proj_bias`, which retain the exported
|
||||
parameter layout.
|
||||
- `lilicorr.factor_input_proj`, `lilicorr.out_head`, and `lilicorr.in_head`, which
|
||||
are used by the split factor and fused edge projections. Exclude these modules
|
||||
when exporting a quantized checkpoint.
|
||||
Reference in New Issue
Block a user