[SpecDecode] Add LiLiCorr drafter (#57934)

Signed-off-by: Andrii Skliar <askliar@nvidia.com>
Signed-off-by: Andrii Skliar <andreyws96@gmail.com>
Co-authored-by: Andrii Skliar <askliar@nvidia.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
This commit is contained in:
Andrii Skliar
2026-09-25 10:08:28 -04:00
committed by GitHub
co-authored by Andrii Skliar Codex Benjamin Chislett
parent 25b0add7b8
commit 73a78e6f1f
21 changed files with 1944 additions and 110 deletions
@@ -9,6 +9,7 @@ To train your own draft models for optimized speculative decoding, see [vllm-pro
vLLM supports a variety of methods of speculative decoding. Model-based methods such as EAGLE, MTP, draft models, PARD and MLP provide the best latency reduction, while simpler methods such as n-gram and suffix decoding provide modest speedups without increasing workload during peak traffic.
- [EAGLE](eagle.md)
- [LiLiCorr](lilicorr.md)
- [Multi-Token Prediction (MTP)](mtp.md)
- [Draft Model](draft_model.md)
- [Parallel Draft Model (PARD)](parallel_draft_model.md)
@@ -0,0 +1,77 @@
# LiLiCorr
LiLiCorr uses a DFlash backbone to produce per-position candidates, then scores
candidate combinations with a learned correlator. It requires a compatible
trained checkpoint and Model Runner V2, which is selected automatically.
Both plain and grouped-convolution checkpoints are supported.
## Usage
For a checkpoint trained with block size 16:
```bash
vllm serve /path/to/target --dtype bfloat16 \
--speculative-config '{"method":"dflash","model":"/path/to/lilicorr","num_speculative_tokens":15,"draft_sample_method":"probabilistic"}'
```
We strongly recommend explicitly setting `num_speculative_tokens` to the
checkpoint's trained `block_size - 1` (for example, `15` for `block_size=16`).
The trained block includes one anchor position. Other supported lengths are
available, but may degrade acceptance length unpredictably; benchmark them with
your checkpoint and workload before using them.
Shorter drafts, down to one token, use the corresponding prefix of the learned
slot embeddings and attention bias without changing checkpoint parameter shapes.
Lengths above `block_size - 1` are unsupported because the checkpoint has no
learned slot embeddings for those positions.
Fixed-length drafting with shorter verification is a separate option provided by
adaptive verification (`enable_adaptive_verification`). LiLiCorr shares the
candidate-sampling and acceptance-estimation path, but this combination has not
been validated end-to-end. Adaptive verification also requires compatible target
attention backends and variable-length CUDA graph support; do not assume it works
with Mamba2 targets.
## Proposal sampling
- `"draft_sample_method":"greedy"` selects the highest-scoring candidate at each
step of the correlator's conditional walk.
- `"draft_sample_method":"probabilistic"` samples from those conditional scores
using the request's temperature and supplies the realized proposal
distributions to the rejection sampler.
These settings control proposals; they do not replace the target model's
sampling settings. With request temperature zero, probabilistic proposals also
use argmax. For nonzero temperatures, benchmark both modes with your checkpoint
and workload: compare acceptance length and end-to-end throughput or latency.
Neither mode is universally faster.
The default rejection method is `"rejection_sample_method":"standard"`.
The shared Model Runner V2 rejection sampler also exposes `"block"`, but
LiLiCorr-specific GPU correctness and performance validation for that combination
is still pending. Do not assume that changing the rejection method improves
throughput; evaluate it separately from the proposal sampling mode.
## Checkpoint requirements
The checkpoint must declare `LiLiCorrDraftModel` and include all `lilicorr_*`
geometry fields and trained head weights. Geometry is read from `dflash_config`.
Convolution tensors must match the configured `conv_kernel_size` and
`conv_group_size`. Target input embeddings must be available on the draft rank.
By default, candidates use the target LM head, which must also be available there.
A checkpoint with top-level `"has_own_lm_head": true` instead uses its own
`lm_head` with the draft quantization configuration and exclusions. Its weights
must be present; for ModelOpt NVFP4 this includes `lm_head.weight`,
`lm_head.weight_scale`, and `lm_head.weight_scale_2`. The owned head is preserved
and is not replaced by the target head. Candidate token embeddings still come
from the target.
Correlator linear layers and convolution kernel projections use the draft
quantization configuration and its module exclusions. The following parameters
must remain in floating-point model dtype:
- Lattice QKV `in_proj_weight` and `in_proj_bias`, which retain the exported
parameter layout.
- `lilicorr.factor_input_proj`, `lilicorr.out_head`, and `lilicorr.in_head`, which
are used by the split factor and fused edge projections. Exclude these modules
when exporting a quantized checkpoint.
+7
View File
@@ -1502,6 +1502,13 @@ _SPECULATIVE_DECODING_EXAMPLE_MODELS = {
max_model_len=8192, # Reduce max len to ensure test runs in low-VRAM CI env
max_num_seqs=32,
),
# LiLiCorr checkpoints are not published yet.
"LiLiCorrDraftModel": _HfExamplesInfo(
"Qwen/Qwen3-8B",
speculative_model="LiLiCorrDraftModel",
is_available_online=False,
use_original_num_layers=True,
),
"DFlash2DraftModel": _HfExamplesInfo(
"Qwen/Qwen3.8-27B",
speculative_model="z-lab/Qwen3.8-27B-DFlash2",
+14 -9
View File
@@ -582,7 +582,7 @@ def test_rocm_mrv1_default_yields_to_v1_unsupported_config(monkeypatch):
speculative_config=None,
)
config._dflash_needs_multi_kv_group = lambda: False
config._is_dflash2_draft = lambda: False
config._is_dflash_candidate_draft = lambda: False
config._get_v2_model_runner_unsupported_features = lambda: []
# The real predicate, so the test also pins where dspark lands in it.
config._get_v1_model_runner_unsupported_features = lambda: (
@@ -813,7 +813,8 @@ def test_v2_model_runner_supports_custom_logits_processors():
assert config._get_v2_model_runner_unsupported_features() == []
def test_dflash2_draft_forces_v2_model_runner():
@pytest.mark.parametrize("architecture", ["DFlash2DraftModel", "LiLiCorrDraftModel"])
def test_dflash_candidate_draft_forces_v2_model_runner(architecture):
"""A DFlash2 draft must reach the V2 speculator, the only one that runs its
candidate selector; on V1 it would draft as DFlash1 without raising."""
@@ -825,11 +826,15 @@ def test_dflash2_draft_forces_v2_model_runner():
)
)
assert VllmConfig._is_dflash2_draft(config("dflash", ["DFlash2DraftModel"]))
assert not VllmConfig._is_dflash2_draft(config("dflash", ["DFlashDraftModel"]))
assert not VllmConfig._is_dflash2_draft(config("eagle", ["DFlash2DraftModel"]))
assert not VllmConfig._is_dflash2_draft(SimpleNamespace(speculative_config=None))
assert not VllmConfig._is_dflash2_draft(
assert VllmConfig._is_dflash_candidate_draft(config("dflash", [architecture]))
assert not VllmConfig._is_dflash_candidate_draft(
config("dflash", ["DFlashDraftModel"])
)
assert not VllmConfig._is_dflash_candidate_draft(config("eagle", [architecture]))
assert not VllmConfig._is_dflash_candidate_draft(
SimpleNamespace(speculative_config=None)
)
assert not VllmConfig._is_dflash_candidate_draft(
SimpleNamespace(
speculative_config=SimpleNamespace(method="dflash", draft_model_config=None)
)
@@ -1243,7 +1248,7 @@ def test_v1_model_runner_rejects_v2_only_features():
model_config=None,
)
config._dflash_needs_multi_kv_group = lambda: False
config._is_dflash2_draft = lambda: False
config._is_dflash_candidate_draft = lambda: False
config._get_v1_model_runner_unsupported_features = lambda: (
VllmConfig._get_v1_model_runner_unsupported_features(config)
)
@@ -1430,7 +1435,7 @@ def test_v1_model_runner_rejects_pipeline_parallelism_with_async_scheduling():
model_config=None,
)
config._dflash_needs_multi_kv_group = lambda: False
config._is_dflash2_draft = lambda: False
config._is_dflash_candidate_draft = lambda: False
assert VllmConfig._get_v1_model_runner_unsupported_features(config) == []
+180 -5
View File
@@ -6,6 +6,9 @@ from types import SimpleNamespace
import pytest
import torch
from vllm.model_executor.models.qwen3_dflash import (
_add_global_draft_layer_exclusions,
)
from vllm.model_executor.models.qwen3_dflash2 import _grouped_conv, _score_edges
from vllm.platforms import current_platform
from vllm.v1.worker.gpu.spec_decode.dflash.speculator import DFlashSpeculator
@@ -85,6 +88,24 @@ def test_grouped_conv_triton_matches_reference(
)
def test_draft_quant_exclusions_include_global_layer_indices():
quant_config = SimpleNamespace(
exclude_modules=[
"layers.0.mlp_conv*",
"*layers.4.self_attn.q_proj",
"layers.88.already_global",
"lilicorr.layers.0.mlp.0",
]
)
_add_global_draft_layer_exclusions(quant_config, 88, 5)
assert "layers.88.mlp_conv*" in quant_config.exclude_modules
assert "*layers.92.self_attn.q_proj" in quant_config.exclude_modules
assert quant_config.exclude_modules.count("layers.88.already_global") == 1
assert "lilicorr.layers.88.mlp.0" not in quant_config.exclude_modules
def test_selector_edges_match_sequential_reference():
torch.manual_seed(1)
batch, steps, top_k, rank = 2, 4, 3, 5
@@ -169,10 +190,13 @@ def test_selector_asks_for_fp32_proposal_logits():
@pytest.mark.skip_global_cleanup
def test_dflash2_model_decoder_layer_cls(monkeypatch):
@pytest.mark.parametrize("variant", ["dflash2", "lilicorr", "lilicorr_plain"])
def test_candidate_model_decoder_layer_cls(monkeypatch, variant):
from types import SimpleNamespace
from vllm.config import set_current_vllm_config
from vllm.model_executor.models.lilicorr import LiLiCorr
from vllm.model_executor.models.qwen3_dflash import DFlashQwen3DecoderLayer
from vllm.model_executor.models.qwen3_dflash2 import (
DFlash2Qwen3DecoderLayer,
DFlash2Qwen3Model,
@@ -249,8 +273,17 @@ def test_dflash2_model_decoder_layer_cls(monkeypatch):
dflash_config={
"selector_rank": 4,
"selector_top_k": 3,
"conv_kernel_size": 3,
"conv_group_size": 2,
"conv_kernel_size": 0 if variant == "lilicorr_plain" else 3,
"conv_group_size": 0 if variant == "lilicorr_plain" else 2,
"block_size": 5,
"lilicorr_candidate_topk": 4,
"lilicorr_hidden_size": 8,
"lilicorr_num_layers": 2,
"lilicorr_num_heads": 2,
"lilicorr_mlp_ratio": 2.0,
"lilicorr_factor_dim": 4,
"lilicorr_vector_eps": 1e-6,
"lilicorr_logit_scale": 3.0,
"use_aux_hidden_state": False,
},
)
@@ -277,8 +310,150 @@ def test_dflash2_model_decoder_layer_cls(monkeypatch):
# 3. Instantiate the model under meta device to avoid parameter allocation issues
with set_current_vllm_config(mock_current_vllm_config), torch.device("meta"):
model = DFlash2Qwen3Model(vllm_config=vllm_config)
model_cls = DFlash2Qwen3Model if variant == "dflash2" else LiLiCorr
model = model_cls(vllm_config=vllm_config)
# 4. Assert that the layers are DFlash2Qwen3DecoderLayer (the subclass)
assert len(model.layers) == 2
assert isinstance(model.layers[0], DFlash2Qwen3DecoderLayer)
expected = (
DFlashQwen3DecoderLayer
if variant == "lilicorr_plain"
else DFlash2Qwen3DecoderLayer
)
assert type(model.layers[0]) is expected
def test_conv_projections_use_draft_quant_config(monkeypatch):
from torch import nn
from vllm.distributed import parallel_state
from vllm.model_executor.layers.quantization import modelopt
from vllm.model_executor.models.qwen3_dflash import DFlashQwen3DecoderLayer
from vllm.model_executor.models.qwen3_dflash2 import DFlash2Qwen3DecoderLayer
from vllm.model_executor.models.utils import AutoWeightsLoader
monkeypatch.setattr(
parallel_state, "_TP", SimpleNamespace(rank_in_group=0, world_size=1)
)
monkeypatch.setattr(
DFlashQwen3DecoderLayer,
"__init__",
lambda self, *a, **kw: nn.Module.__init__(self),
)
# Exercise the real quantized parameter allocation without selecting a GPU kernel.
monkeypatch.setattr(
modelopt,
"select_linear_kernel",
lambda *a, **kw: SimpleNamespace(input_quant_key=lambda: None),
)
quant_config = modelopt.ModelOptNvFp4Config(
quant_method="W4A16_NVFP4", is_checkpoint_nvfp4_serialized=True
)
layer = DFlash2Qwen3DecoderLayer(
SimpleNamespace(
speculative_config=SimpleNamespace(num_speculative_tokens=7),
model_config=SimpleNamespace(dtype=torch.bfloat16),
),
config=SimpleNamespace(
hidden_size=16,
dflash_config={"conv_kernel_size": 2, "conv_group_size": 2},
),
layer_idx=0,
prefix="model.layers.0",
quant_config=quant_config,
)
for name in ("attention_conv", "mlp_conv"):
module = getattr(layer, name)
projection = module.kernel_projection
assert projection.weight.dtype == torch.uint8
assert projection.weight.shape == (32, 8)
weight = torch.ones_like(projection.weight)
AutoWeightsLoader(module).load_weights([("kernel_projection.weight", weight)])
torch.testing.assert_close(projection.weight, weight)
assert module.base_kernel.dtype == torch.bfloat16
def test_context_kv_uses_quantized_projection_fallback(monkeypatch):
from torch import nn
from vllm.model_executor.models import qwen3_dflash
class Projection(nn.Module):
def __init__(self, packed_weight):
super().__init__()
self.register_buffer("packed_weight", packed_weight)
self.quant_method = object()
self.calls = 0
def forward(self, hidden_states):
self.calls += 1
return torch.nn.functional.linear(hidden_states, self.packed_weight), None
monkeypatch.setattr(
qwen3_dflash.ops,
"rms_norm",
lambda output, hidden_states, weight, eps: output.copy_(hidden_states),
)
context_states = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
projections = [
Projection(
torch.tensor(
[
[0.0, 0.0],
[0.0, 0.0],
[1.0, 0.0],
[0.0, 1.0],
[1.0, 1.0],
[1.0, -1.0],
]
)
),
Projection(
torch.tensor(
[
[0.0, 0.0],
[0.0, 0.0],
[2.0, 0.0],
[0.0, 2.0],
[-1.0, 0.0],
[0.0, -1.0],
]
)
),
]
model = SimpleNamespace(
hidden_norm=SimpleNamespace(weight=nn.Parameter(torch.ones(2))),
_rms_norm_eps=1e-6,
)
layers_attn = [
SimpleNamespace(
qkv_proj=projection,
q_size=2,
k_norm=SimpleNamespace(weight=nn.Parameter(torch.ones(2))),
)
for projection in projections
]
qwen3_dflash.DFlashQwen3Model._build_context_kv_buffers(
model, layers_attn, has_bias=False
)
assert model._fused_kv_weight is None
assert all(not hasattr(projection, "weight") for projection in projections)
actual_k, actual_v = qwen3_dflash.DFlashQwen3Model._project_context_kv(
model,
context_states,
num_ctx=2,
num_layers=2,
num_kv_heads=1,
head_dim=2,
)
expected_k = torch.tensor(
[[[1.0, 2.0], [3.0, 4.0]], [[2.0, 4.0], [6.0, 8.0]]]
).unsqueeze(2)
expected_v = torch.tensor(
[[[3.0, -1.0], [7.0, -1.0]], [[-1.0, -2.0], [-3.0, -4.0]]]
).unsqueeze(2)
torch.testing.assert_close(actual_k, expected_k)
torch.testing.assert_close(actual_v, expected_v)
assert [projection.calls for projection in projections] == [1, 1]
+777
View File
@@ -0,0 +1,777 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
from types import SimpleNamespace
from typing import Any
import pytest
import torch
import torch.nn.functional as F
from torch import nn
from vllm.config import (
CompilationConfig,
DeviceConfig,
LoadConfig,
VllmConfig,
set_current_vllm_config,
)
from vllm.distributed import parallel_state
from vllm.model_executor.layers import logits_processor
from vllm.model_executor.layers.linear import LinearBase
from vllm.model_executor.models.lilicorr import LiLiCorrHead
from vllm.transformers_utils.configs.eagle import EAGLEConfig
from vllm.v1.worker.gpu.spec_decode.dflash2.speculator import CandidateSampler
from vllm.v1.worker.gpu.spec_decode.lilicorr.speculator import LiLiCorrSpeculator
@pytest.fixture(autouse=True)
def head_tp_metadata(monkeypatch):
# Single-rank unit tests only need TP metadata.
monkeypatch.setattr(
parallel_state, "_TP", SimpleNamespace(rank_in_group=0, world_size=1)
)
def _config(**overrides):
config = dict(
candidate_topk=4,
hidden_size=8,
num_layers=2,
num_heads=2,
mlp_ratio=2.0,
factor_dim=4,
vector_eps=1e-6,
logit_scale=3.0,
)
config.update(overrides)
return {f"lilicorr_{key}": value for key, value in config.items()}
def _reference_scores(head, embeddings, log_probs, hidden, anchor, valid):
"""Unfused exported head: MHA attention, concatenated factors, separate edges."""
batch, slots, top_k = log_probs.shape
rank = torch.arange(top_k, device=log_probs.device) / max(top_k - 1, 1)
top1 = (torch.arange(top_k, device=log_probs.device) == 0).float()
features = torch.stack(
(
log_probs,
log_probs.exp(),
log_probs - log_probs.amax(-1, keepdim=True),
rank.expand_as(log_probs),
top1.expand_as(log_probs),
),
-1,
)
x = head.token_proj(embeddings) + head.pass_hidden_proj(hidden).unsqueeze(-2)
features = head.feature_norm(features.to(x.dtype))
x = x + head.feature_mlp.down_proj(F.silu(head.feature_mlp.up_proj(features)[0]))[0]
x = x + head.slot_embedding[:, 0, :slots] + head.rank_embedding[:, 0]
x = x.flatten(1, 2)
positions = torch.arange(slots, device=x.device).repeat_interleave(top_k)
relative = positions[:, None] - positions[None, :]
bias = head.relative_slot_bias[:, relative + head.block_size - 1]
bias = bias + (relative == 0) * head.same_slot_bias[:, None, None]
bias = bias.repeat(batch, 1, 1)
for layer in head.layers:
attention = nn.MultiheadAttention(
head.hidden_size,
head.num_heads,
batch_first=True,
dtype=x.dtype,
device=x.device,
)
attention.load_state_dict(layer.attn.state_dict())
normalized = layer.attn_norm(x)
x = (
x
+ attention(
normalized, normalized, normalized, attn_mask=bias, need_weights=False
)[0]
)
x = x + layer.mlp.down_proj(F.silu(layer.mlp.up_proj(layer.mlp_norm(x))[0]))[0]
x = head.output_norm(x).view(batch, slots, top_k, -1)
a = head.anchor_norm(head.context_proj(anchor) * valid[:, None])
expanded = a[:, None, None].expand_as(x)
factors = F.silu(head.factor_input_proj(torch.cat((x, expanded, x * expanded), -1)))
incoming = F.normalize(head.in_head(factors), dim=-1, eps=head.vector_eps)
outgoing = F.normalize(head.out_head(factors), dim=-1, eps=head.vector_eps)
a = F.normalize(head.anchor_out_head(a), dim=-1, eps=head.vector_eps)
start = (a[:, None] * incoming[:, 0]).sum(-1)
pairs = outgoing[:, :-1] @ incoming[:, 1:].transpose(-1, -2)
return (
head.logit_scale
* torch.cat((start[:, None, None].expand(-1, 1, top_k, -1), pairs), 1).float()
)
@pytest.mark.skipif(not torch.cuda.is_available(), reason="requires CUDA or ROCm")
@pytest.mark.parametrize(
"head_width,slots,dtype",
[
pytest.param(8, 3, torch.float32, id="projected-multi-slot"),
pytest.param(16, 1, torch.float32, id="identity-shortened-single-slot"),
pytest.param(8, 2, torch.bfloat16, id="bf16-shortened-multi-slot"),
],
)
def test_lilicorr_matches_exported_head(head_width, slots, dtype):
device = torch.device("cuda")
torch.manual_seed(7)
with (
torch.device(device),
set_current_vllm_config(
VllmConfig(
device_config=DeviceConfig("cuda"),
compilation_config=CompilationConfig(mode=0),
)
),
):
head = LiLiCorrHead(
model_hidden_size=16,
block_size=4,
rms_norm_eps=1e-6,
config=_config(hidden_size=head_width),
).to(dtype)
# vLLM linear parameters are initialized by the checkpoint loader.
with torch.no_grad(), torch.device(device):
for module in head.modules():
if isinstance(module, LinearBase):
for parameter in module.parameters():
parameter.weight_loader(
parameter, torch.randn_like(parameter) * 0.1
)
for layer in head.layers:
layer.attn.in_proj_weight.normal_(std=0.1)
layer.attn.in_proj_bias.normal_(std=0.1)
head.relative_slot_bias.normal_(std=0.1)
head.same_slot_bias.normal_(std=0.1)
head.slot_embedding.normal_(std=0.1)
head.rank_embedding.normal_(std=0.1)
head.materialize_inference_buffers(device, dtype)
inputs = (
torch.randn(2, slots, 4, 16, dtype=dtype),
torch.randn(2, slots, 23).log_softmax(-1).topk(4).values,
torch.randn(2, slots, 16, dtype=dtype),
torch.randn(2, 16, dtype=dtype),
torch.tensor([True, False]),
)
actual = head(*inputs)
expected = _reference_scores(head, *inputs)
tolerance = 0.04 if dtype == torch.bfloat16 else 2e-6
torch.testing.assert_close(actual, expected, atol=tolerance, rtol=tolerance)
def test_runtime_length_keeps_trained_checkpoint_geometry(monkeypatch):
from vllm.model_executor.models.lilicorr import LiLiCorr
from vllm.model_executor.models.qwen3_dflash import DFlashQwen3Model
def init_backbone(self, **kwargs):
nn.Module.__init__(self)
self.quant_config = None
monkeypatch.setattr(DFlashQwen3Model, "__init__", init_backbone)
spec = SimpleNamespace(
num_speculative_tokens=1,
draft_model_config=SimpleNamespace(
hf_config=SimpleNamespace(
hidden_size=16,
rms_norm_eps=1e-6,
dflash_config=_config() | {"block_size": 4},
)
),
)
with set_current_vllm_config(
VllmConfig(
device_config=DeviceConfig("cpu"),
compilation_config=CompilationConfig(mode=0),
)
):
for slots in (1, 3):
spec.num_speculative_tokens = slots
head = LiLiCorr(
vllm_config=SimpleNamespace(speculative_config=spec)
).lilicorr
assert head.slot_embedding.shape == (1, 1, 3, 1, 8)
assert head.relative_slot_bias.shape == (2, 7)
spec.draft_model_config.hf_config.dflash_config["lilicorr_enabled"] = False
with pytest.raises(ValueError, match="lilicorr_enabled"):
LiLiCorr(vllm_config=SimpleNamespace(speculative_config=spec))
spec.draft_model_config.hf_config.dflash_config["lilicorr_enabled"] = True
spec.num_speculative_tokens = 4
with pytest.raises(ValueError, match="trained block_size"):
LiLiCorr(vllm_config=SimpleNamespace(speculative_config=spec))
@pytest.mark.parametrize("tp_size", [1, 2])
def test_candidates_use_global_partition_and_exclude_padding(monkeypatch, tp_size):
"""Candidate features must include mass outside top-k and on other TP ranks."""
class Gather:
def __init__(self, calls):
self.calls = calls
self.index = 0
def __call__(self, value, dim=-1):
expected_input, result = self.calls[self.index]
assert dim == -1
torch.testing.assert_close(value, expected_input)
self.index += 1
return result
shards = [
torch.tensor([[1.0, 3.0, 2.0, 100.0], [4.0, 1.0, 2.0, 100.0]]),
torch.tensor([[6.0, 5.0, 0.0, 100.0], [3.0, 6.0, 5.0, 100.0]]),
][:tp_size]
for rank in range(tp_size):
fake = SimpleNamespace(
scale=1.0,
soft_cap=None,
_apply_head=lambda *args, rank=rank: shards[rank].clone(),
)
lm_head = SimpleNamespace(
tp_size=tp_size,
shard_indices=SimpleNamespace(
num_org_vocab_padding=1, org_vocab_start_index=rank * 3
),
)
valid_shards = [shard[:, :3] for shard in shards]
local_topk = [shard.topk(2) for shard in valid_shards]
gather_calls = [
(
valid_shards[rank].logsumexp(-1, keepdim=True),
torch.cat(
[shard.logsumexp(-1, keepdim=True) for shard in valid_shards],
-1,
),
),
(
local_topk[rank].values,
torch.cat([topk.values for topk in local_topk], -1),
),
(
local_topk[rank].indices + rank * 3,
torch.cat(
[
topk.indices + shard_rank * 3
for shard_rank, topk in enumerate(local_topk)
],
-1,
),
),
]
all_gather = Gather(gather_calls)
monkeypatch.setattr(
logits_processor,
"tensor_model_parallel_all_gather",
all_gather,
)
monkeypatch.setattr(logits_processor, "_topk", lambda x, k: x.topk(k, dim=-1))
ids, values = logits_processor.LogitsProcessor.get_top_k_tokens(
fake, lm_head, torch.empty(2, 1), 2, return_log_probs=True
)
expected = torch.cat(valid_shards, -1).log_softmax(-1).topk(2)
torch.testing.assert_close(ids, expected.indices)
torch.testing.assert_close(values, expected.values)
assert all_gather.index == (3 if tp_size > 1 else 0)
def test_context_anchor_is_last_committed_normalized_feature():
hidden = torch.arange(1, 25, dtype=torch.float32).view(6, 4)
norm = nn.RMSNorm(4)
spec = SimpleNamespace(
hidden_states=hidden,
model=SimpleNamespace(model=SimpleNamespace(hidden_norm=norm)),
anchor_hidden=torch.full((4, 4), 123.0),
anchor_valid=torch.ones(4, dtype=torch.bool),
)
batch = SimpleNamespace(num_reqs=2, query_start_loc=torch.tensor([0, 2, 6]))
LiLiCorrSpeculator.prepare_context_anchor(spec, batch, torch.tensor([0, 2]))
torch.testing.assert_close(spec.anchor_hidden[:2], norm(hidden[[1, 3]]))
assert spec.anchor_valid.tolist() == [True, True, False, False]
assert not spec.anchor_hidden[2:].any()
# A smaller reordered batch must not retain an old request's anchor.
batch = SimpleNamespace(num_reqs=1, query_start_loc=torch.tensor([0, 3]))
LiLiCorrSpeculator.prepare_context_anchor(spec, batch, torch.tensor([1]))
torch.testing.assert_close(spec.anchor_hidden[0], norm(hidden[1]))
assert not spec.anchor_hidden[1:].any()
@pytest.mark.parametrize("lilicorr", [False, True])
def test_candidate_generation_routes_scores_and_adaptive_inputs(monkeypatch, lilicorr):
"""Both heads share the walk, including a padded request and adaptive scores."""
from vllm.v1.worker.gpu.spec_decode.dflash.speculator import DFlashSpeculator
from vllm.v1.worker.gpu.spec_decode.dflash2.speculator import DFlash2Speculator
key = "lilicorr_candidate_topk" if lilicorr else "selector_top_k"
def init_base(self, config, device):
self.draft_model_config = SimpleNamespace(
hf_config=SimpleNamespace(hidden_size=4, dflash_config={key: 4})
)
self.max_num_reqs = 3
self.num_speculative_steps = 2
self.num_query_per_req = 3
self.dtype = torch.float32
monkeypatch.setattr(DFlashSpeculator, "__init__", init_base)
cls = LiLiCorrSpeculator if lilicorr else DFlash2Speculator
spec = cls(None, torch.device("cpu"))
hidden = torch.arange(36).float().view(9, 4)
spec._run_model = lambda *args: hidden
spec.sample_indices = torch.tensor([1, 2, 4, 5, 7, 8])
spec.sample_pos = torch.arange(6)
spec.sample_idx_mapping = torch.tensor([1, 1, 0, 0, -1, -1])
spec.sample_col = torch.tensor([0, 1, 0, 1, 0, 1])
spec.temperature = torch.ones(3)
spec.seeds = torch.arange(3)
spec.draft_tokens = torch.empty(3, 2, dtype=torch.long)
spec.draft_logits = None
spec.use_fp64_gumbel = False
spec.enable_adaptive_verification = True
spec.input_buffers = SimpleNamespace(input_ids=torch.arange(9))
candidates = torch.arange(24).view(6, 4)
log_probs = candidates.float().log_softmax(-1)
scores = torch.randn(3, 2, 4, 4)
scoring_inputs: list[torch.Tensor] = []
sampling_inputs: list[Any] = []
adaptive_inputs: list[torch.Tensor] = []
def score(*args):
scoring_inputs.extend(args)
return scores
def sample(*args):
sampling_inputs.extend(args)
spec.candidate_sampler.scores.copy_(scores[:, :, 0])
spec.model = SimpleNamespace(
compute_candidates=lambda h: (candidates, log_probs),
model=SimpleNamespace(lilicorr=score, candidate_selector=score),
)
spec.target_embeddings = lambda ids: ids.float().unsqueeze(-1).expand(*ids.shape, 4)
if lilicorr:
spec.anchor_hidden.copy_(torch.arange(12).view(3, 4))
spec.anchor_valid[:2] = True
spec.candidate_sampler.sample = sample
spec._maybe_predict_acceptance = lambda *args: adaptive_inputs.extend(args)
spec._generate_draft(3, 9, None, None, None)
expected_ids = candidates.view(3, 2, 4)
expected_first = spec.target_embeddings(expected_ids) if lilicorr else expected_ids
torch.testing.assert_close(scoring_inputs[0], expected_first)
torch.testing.assert_close(scoring_inputs[1], log_probs.view(3, 2, 4))
torch.testing.assert_close(
scoring_inputs[2], hidden[spec.sample_indices].view(3, 2, 4)
)
if lilicorr:
torch.testing.assert_close(scoring_inputs[3], spec.anchor_hidden)
torch.testing.assert_close(scoring_inputs[4], spec.anchor_valid)
else:
torch.testing.assert_close(scoring_inputs[3], torch.tensor([0, 3, 6]))
torch.testing.assert_close(sampling_inputs[0], expected_ids)
assert sampling_inputs[1] is scores
assert sampling_inputs[4] is spec.sample_idx_mapping
torch.testing.assert_close(adaptive_inputs[0], scores[:, :, 0].flatten(0, 1))
torch.testing.assert_close(adaptive_inputs[1], spec.sample_idx_mapping)
torch.testing.assert_close(adaptive_inputs[2], spec.sample_col)
def test_lilicorr_config_preserves_architecture():
from transformers import Qwen3Config
config = Qwen3Config(architectures=["LiLiCorrDraftModel"])
wrapped = EAGLEConfig(config, method="dflash")
assert wrapped.architectures == ["LiLiCorrDraftModel"]
@pytest.mark.skipif(not torch.cuda.is_available(), reason="requires CUDA")
@pytest.mark.parametrize("probabilistic", [False, True])
def test_candidate_walk_preserves_conditional_scores_and_padding(probabilistic):
"""Check the shared sampler's actual q, stale support, and graph padding."""
device = torch.device("cuda")
batch, steps, top_k, vocab = 3, 3, 4, 32
sampler = CandidateSampler(batch, steps, top_k, device)
candidates = (
torch.arange(steps * top_k, device=device)
.view(1, steps, top_k)
.repeat(batch, 1, 1)
)
scores = torch.randn(batch, steps, top_k, top_k, device=device)
tokens = torch.full((batch, steps), -1, dtype=torch.long, device=device)
cache = (
torch.full((batch, steps, vocab), -float("inf"), device=device)
if probabilistic
else None
)
positions = torch.arange(batch * steps * 2, device=device)[::2]
mappings = torch.tensor(
[2] * steps + [0] * steps + [-1] * steps, device=device
).repeat_interleave(2)[::2]
temperature = torch.tensor([0.0, 1.0, 0.8], device=device)
seeds = torch.tensor([11, 22, 33], device=device)
def run():
sampler.sample(
candidates,
scores,
batch,
positions,
mappings,
temperature,
seeds,
tokens,
cache,
False,
)
run()
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph):
run()
old_candidates = candidates.clone()
candidates.add_(12)
graph.replay()
for row, state in ((0, 2), (1, 0)):
previous = 0
for step in range(steps):
realized = scores[row, step, previous]
torch.testing.assert_close(sampler.scores[row, step], realized)
index = (candidates[row, step] == tokens[row, step]).nonzero().item()
if not probabilistic or temperature[state] == 0:
assert index == realized.argmax().item()
if cache is not None:
torch.testing.assert_close(
cache[state, step, candidates[row, step]], realized
)
assert cache[state, step, old_candidates[row, step]].isneginf().all()
previous = index
assert (tokens[2] == -1).all()
if cache is not None:
assert cache[1].isneginf().all()
@pytest.mark.parametrize(
("convolution", "mismatch", "quantized"),
[
pytest.param(False, None, False, id="valid-plain"),
pytest.param(True, None, False, id="valid-convolution"),
pytest.param(False, "missing_head", False, id="missing-head"),
pytest.param(False, "extra_head", False, id="extra-head"),
pytest.param(True, "missing_conv", False, id="missing-convolution"),
pytest.param(False, "extra_conv", False, id="unexpected-convolution"),
pytest.param(False, None, True, id="optional-w4a16-input-scale"),
pytest.param(False, "missing_weight", True, id="missing-quantized-weight"),
pytest.param(False, "missing_bias", True, id="missing-quantized-bias"),
pytest.param(False, "missing_weight_scale", True, id="missing-block-scale"),
pytest.param(False, "missing_weight_scale_2", True, id="missing-global-scale"),
],
)
def test_checkpoint_coverage_rejects_incomplete_or_wrong_heads(
monkeypatch, convolution, mismatch, quantized
):
from vllm.model_executor.models.lilicorr import LiLiCorr, LiLiCorrForCausalLM
wrapper = LiLiCorrForCausalLM.__new__(LiLiCorrForCausalLM)
nn.Module.__init__(wrapper)
wrapper.model = LiLiCorr.__new__(LiLiCorr)
nn.Module.__init__(wrapper.model)
wrapper.has_own_lm_head = False
with set_current_vllm_config(
VllmConfig(
device_config=DeviceConfig("cpu"),
compilation_config=CompilationConfig(mode=0),
)
):
wrapper.model.lilicorr = LiLiCorrHead(
model_hidden_size=16, block_size=4, rms_norm_eps=1e-6, config=_config()
)
layer = nn.Module()
if convolution:
layer.attention_conv = nn.Linear(2, 2, bias=False)
wrapper.model.layers = nn.ModuleList([layer])
if quantized:
from vllm.config.quantization import QuantSpec
from vllm.model_executor.layers.quantization.modelopt import (
CkptCtx,
ModelOptLinearMethod,
)
from vllm.model_executor.layers.quantization.utils.quant_utils import (
kNvfp4Static,
)
# Match the W4A16 NVFP4 checkpoint contract without GPU kernel setup.
projection = wrapper.model.lilicorr.pass_hidden_proj
projection.quant_method = ModelOptLinearMethod(
QuantSpec(weight=kNvfp4Static, activation=None), CkptCtx(group_size=16)
)
projection.weight = nn.Parameter(
torch.zeros(8, 8, dtype=torch.uint8), requires_grad=False
)
for name, value in (
("weight_scale", torch.ones(8, 1).to(torch.float8_e4m3fn)),
("weight_scale_2", torch.ones(1)),
("input_scale", torch.full((1,), torch.nan)),
):
projection.register_parameter(
name, nn.Parameter(value, requires_grad=False)
)
weights = dict(wrapper.model.state_dict())
if quantized:
del weights["lilicorr.pass_hidden_proj.input_scale"]
if mismatch == "missing_head":
del weights["lilicorr.slot_embedding"]
elif mismatch in (
"missing_weight",
"missing_bias",
"missing_weight_scale",
"missing_weight_scale_2",
):
del weights[f"lilicorr.pass_hidden_proj.{mismatch.removeprefix('missing_')}"]
elif mismatch == "extra_head":
weights["lilicorr.untrained.weight"] = torch.zeros(1)
elif mismatch == "missing_conv":
del weights["layers.0.attention_conv.weight"]
elif mismatch == "extra_conv":
weights["layers.0.mlp_conv.untrained"] = torch.zeros(1)
# Keep production loading and mapping, bypass only backbone GPU KV setup.
wrapper.model.use_aux_hidden_state = True
wrapper.model.has_separate_mask_embedding = False
monkeypatch.setattr(wrapper, "_read_mask_embedding", lambda: None)
monkeypatch.setattr(
wrapper.model, "_build_fused_kv_buffers", lambda: None, raising=False
)
supplied = [("model." + name, value) for name, value in weights.items()]
if mismatch:
message = (
"no module or parameter"
if mismatch.startswith("extra")
else "coverage mismatch"
)
with pytest.raises(ValueError, match=message):
wrapper.load_weights(supplied)
else:
wrapper.load_weights(supplied)
assert wrapper.model.lilicorr._attn_bias is not None
@pytest.fixture
def owned_head_model(monkeypatch):
from vllm.model_executor.layers.quantization import modelopt
from vllm.model_executor.models import qwen3_dflash
from vllm.model_executor.models.lilicorr import LiLiCorrForCausalLM
quant_config = modelopt.ModelOptNvFp4Config(
quant_method="W4A16_NVFP4", is_checkpoint_nvfp4_serialized=True
)
monkeypatch.setattr(
modelopt,
"select_linear_kernel",
lambda *a, **kw: SimpleNamespace(input_quant_key=lambda: None),
)
monkeypatch.setattr(qwen3_dflash, "get_draft_quant_config", lambda _: quant_config)
class Backbone(nn.Module):
def __init__(self, **kwargs):
super().__init__()
self.quant_config = quant_config
self.use_aux_hidden_state = True
self.has_separate_mask_embedding = False
self.lilicorr = LiLiCorrHead(
model_hidden_size=16, block_size=4, rms_norm_eps=1e-6, config=_config()
)
def _build_fused_kv_buffers(self):
pass
monkeypatch.setattr(LiLiCorrForCausalLM, "model_cls", Backbone)
monkeypatch.setattr(LiLiCorrForCausalLM, "_read_mask_embedding", lambda _: None)
def build(owned=True):
config = SimpleNamespace(
vocab_size=64,
draft_vocab_size=64,
hidden_size=16,
has_own_lm_head=owned,
num_hidden_layers=0,
dflash_config={},
)
vllm_config = SimpleNamespace(
speculative_config=SimpleNamespace(
draft_model_config=SimpleNamespace(hf_config=config),
draft_load_config=None,
attention_backend=None,
kv_cache_dtype=None,
),
model_config=SimpleNamespace(
get_total_num_hidden_layers=lambda: 0,
get_vocab_size=lambda: 64,
),
attention_config=SimpleNamespace(),
cache_config=SimpleNamespace(),
load_config=LoadConfig(),
)
with set_current_vllm_config(
VllmConfig(
device_config=DeviceConfig("cpu"),
compilation_config=CompilationConfig(mode=0),
)
):
model = LiLiCorrForCausalLM(vllm_config=vllm_config)
return model, vllm_config
return build
@pytest.mark.parametrize("missing", [None, "weight", "weight_scale", "weight_scale_2"])
def test_owned_nvfp4_lm_head_loads_required_checkpoint_tensors(
owned_head_model, missing
):
model, _ = owned_head_model()
assert model.lm_head.weight.dtype == torch.uint8
assert model.lm_head.weight.shape == (64, 8)
weights = {
name: torch.ones_like(value) for name, value in model.state_dict().items()
}
del weights["lm_head.input_scale"] # Deprecated and unused for W4A16.
if missing:
del weights[f"lm_head.{missing}"]
with pytest.raises(ValueError, match=f"lm_head.{missing}"):
model.load_weights(weights.items())
else:
model.load_weights(weights.items())
assert model.has_own_lm_head
for name in ("weight", "weight_scale", "weight_scale_2"):
torch.testing.assert_close(
getattr(model.lm_head, name), weights[f"lm_head.{name}"]
)
@pytest.mark.parametrize("owned", [False, True])
def test_selected_lm_head_survives_dflash_and_lilicorr_sharing(
monkeypatch, owned_head_model, owned
):
from vllm.model_executor.layers.vocab_parallel_embedding import (
VocabParallelEmbedding,
)
from vllm.v1.worker.gpu.spec_decode.dflash import utils as dflash_utils
from vllm.v1.worker.gpu.spec_decode.eagle import utils as eagle_utils
model, config = owned_head_model(owned)
own_head = model.lm_head
target_head = owned_head_model(owned)[0].lm_head
# Identical packed bytes do not imply identical quantized heads.
with torch.no_grad():
own_head.weight.fill_(1)
target_head.weight.fill_(1)
if owned:
own_head.weight_scale_2.fill_(1)
target_head.weight_scale_2.fill_(2)
embedding = VocabParallelEmbedding(64, 16)
target = SimpleNamespace(
model=SimpleNamespace(embed_tokens=embedding), lm_head=target_head
)
monkeypatch.setattr(
eagle_utils, "get_pp_group", lambda: SimpleNamespace(world_size=1)
)
monkeypatch.setattr(
dflash_utils, "replace", lambda obj, **kw: SimpleNamespace(**(vars(obj) | kw))
)
monkeypatch.setattr(dflash_utils, "get_pp_safe_draft_load_config", lambda c: c)
monkeypatch.setattr(dflash_utils, "get_model", lambda **kw: model)
if owned:
def no_weight_comparison(*args):
raise AssertionError("Explicit ownership must bypass raw-weight comparison")
monkeypatch.setattr(dflash_utils, "_should_share", no_weight_comparison)
spec = LiLiCorrSpeculator.__new__(LiLiCorrSpeculator)
spec.vllm_config = config
assert spec.load_draft_model(target, set()) is model
assert model.lm_head is (own_head if owned else target_head)
assert spec.target_embeddings is embedding
if owned:
target.lm_head = None
assert spec.load_draft_model(target, set()).lm_head is own_head
def test_quantized_head_calls_methods_without_reading_packed_weights(monkeypatch):
from vllm.model_executor.layers import linear
from vllm.model_executor.layers.linear import UnquantizedLinearMethod
calls = []
configured = []
quant_config = object()
class PackedMethod(UnquantizedLinearMethod):
def apply(self, layer, x, bias=None):
calls.append(layer.prefix)
# Packed storage has a different name/dtype from the activations.
assert x.dtype == torch.float32
return F.linear(x, layer.packed_weight.float() * layer.weight_scale, bias)
def resolve(config, layer, prefix):
assert config is quant_config
configured.append(prefix)
return PackedMethod()
monkeypatch.setattr(linear, "resolve_quant_method", resolve)
with set_current_vllm_config(
VllmConfig(
device_config=DeviceConfig("cpu"),
compilation_config=CompilationConfig(mode=0),
)
):
head = LiLiCorrHead(
model_hidden_size=16,
block_size=4,
rms_norm_eps=1e-6,
config=_config(hidden_size=8),
quant_config=quant_config,
prefix="model.lilicorr",
)
with torch.no_grad():
for module in head.modules():
if isinstance(module, LinearBase):
if module.quant_config is None:
module.weight.normal_(std=0.1)
module.bias.zero_()
continue
module.register_parameter(
"packed_weight",
nn.Parameter(
torch.randint(-4, 5, module.weight.shape, dtype=torch.int8),
requires_grad=False,
),
)
module.register_parameter(
"weight_scale",
nn.Parameter(
torch.tensor(0.05),
requires_grad=False,
),
)
del module.weight
module.bias.zero_()
for layer in head.layers:
layer.attn.in_proj_weight.normal_(std=0.1)
layer.attn.in_proj_bias.zero_()
head.materialize_inference_buffers(torch.device("cpu"), torch.float32)
result = head(
torch.randn(2, 3, 4, 16),
torch.randn(2, 3, 4).log_softmax(-1),
torch.randn(2, 3, 16),
torch.randn(2, 16),
torch.tensor([True, False]),
)
assert result.shape == (2, 3, 4, 4)
assert torch.isfinite(result).all()
for name in ("factor_input_proj", "out_head", "in_head"):
assert getattr(head, name).quant_config is None
assert f"model.lilicorr.{name}" not in configured
assert set(calls) == set(configured)
assert len(configured) == len(set(configured))
for name, module in head.named_modules():
if isinstance(module, LinearBase):
assert module.prefix == f"model.lilicorr.{name}"
+10 -6
View File
@@ -760,8 +760,8 @@ class VllmConfig:
return True
def _is_dflash2_draft(self) -> bool:
"""Whether the DFlash draft is a DFlash2 one, by the architecture the
def _is_dflash_candidate_draft(self) -> bool:
"""Whether the DFlash draft has a candidate head, by the architecture the
speculator selects on (v1/worker/gpu/spec_decode/__init__.py)."""
spec = self.speculative_config
if spec is None or spec.method != "dflash":
@@ -769,7 +769,11 @@ class VllmConfig:
draft_config = getattr(spec, "draft_model_config", None)
if draft_config is None:
return False
return "DFlash2DraftModel" in (draft_config.architectures or [])
return bool(
{"DFlash2DraftModel", "LiLiCorrDraftModel"}.intersection(
draft_config.architectures or []
)
)
def _dflash_needs_multi_kv_group(self) -> bool:
"""Whether a DFlash draft mixes sliding-window and full attention."""
@@ -3097,11 +3101,11 @@ class VllmConfig:
if self._dflash_needs_multi_kv_group():
unsupported.append("mixed sliding/full dflash drafts")
# The DFlash2 candidate selector exists only in the V2 speculator. On
# DFlash candidate heads exist only in the V2 speculator. On
# V1 the same checkpoint drafts through DFlashProposer, which never
# calls it, so the draft would degrade to DFlash1 silently.
if self._is_dflash2_draft():
unsupported.append("dflash2 drafts")
if self._is_dflash_candidate_draft():
unsupported.append("DFlash candidate-head drafts")
if self.model_config is not None and self.model_config.is_diffusion:
unsupported.append("diffusion models")
+21 -1
View File
@@ -256,6 +256,8 @@ class LogitsProcessor(PluggableLayer):
hidden_states: torch.Tensor,
k: int,
embedding_bias: torch.Tensor | None = None,
*,
return_log_probs: bool = False,
) -> tuple[torch.Tensor, torch.Tensor]:
"""Vocab-parallel top-k without all-gathering full logits.
@@ -263,7 +265,9 @@ class LogitsProcessor(PluggableLayer):
the values as well as the global ids. Communication is
O(batch * 2k * tp_size) rather than O(batch * vocab_size).
Scale and soft cap are applied to the k selected values rather than
With return_log_probs, values use the full-vocabulary log-partition.
Otherwise scale and soft cap are applied to the k selected values rather than
the whole vocabulary; both are monotonic, so the selection is the same
and only k entries are touched.
"""
@@ -274,12 +278,26 @@ class LogitsProcessor(PluggableLayer):
)
logits = self._apply_head(lm_head, hidden_states, embedding_bias)
if return_log_probs:
if self.soft_cap is not None:
logits = torch.tanh(logits / self.soft_cap) * self.soft_cap
if self.scale != 1.0:
logits = logits * self.scale
# Mask out padding entries beyond org_vocab_size on this shard.
num_pad = lm_head.shard_indices.num_org_vocab_padding
if num_pad > 0:
logits[..., -num_pad:] = -float("inf")
if return_log_probs:
log_partition = torch.logsumexp(logits.float(), dim=-1, keepdim=True)
if lm_head.tp_size > 1:
log_partition = torch.logsumexp(
tensor_model_parallel_all_gather(log_partition, dim=-1),
dim=-1,
keepdim=True,
)
values, ids = _topk(logits, k)
# Convert shard-local indices to global vocab indices.
ids = ids.to(torch.int64) + lm_head.shard_indices.org_vocab_start_index
@@ -291,6 +309,8 @@ class LogitsProcessor(PluggableLayer):
ids = ids.gather(-1, selected)
values = values.float()
if return_log_probs:
return ids, values - log_partition
if self.scale != 1.0:
values = values * self.scale
if self.soft_cap is not None:
+546
View File
@@ -0,0 +1,546 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# LiLiCorr head adapted from sgl-project/sglang PR #37462 (Apache-2.0).
"""DFlash backbone with the LiLiCorr candidate-lattice correlator."""
from collections.abc import Iterable
from typing import Any
import torch
import torch.nn.functional as F
from torch import nn
from vllm.compilation.backends import set_model_tag
from vllm.compilation.decorators import support_torch_compile
from vllm.config import VllmConfig
from vllm.model_executor.layers.linear import (
LinearBase,
ReplicatedLinear,
UnquantizedLinearMethod,
)
from vllm.model_executor.layers.logits_processor import LogitsProcessor
from vllm.model_executor.layers.quantization.base_config import QuantizationConfig
from vllm.model_executor.layers.quantization.modelopt import ModelOptLinearMethod
from vllm.model_executor.layers.quantization.utils.quant_utils import kNvfp4Static
from vllm.model_executor.layers.vocab_parallel_embedding import ParallelLMHead
from .nemotron_h import NemotronHMLP
from .qwen3_dflash import DFlashQwen3ForCausalLM, DFlashQwen3Model
from .qwen3_dflash2 import DFlash2Qwen3DecoderLayer
from .utils import AutoWeightsLoader, maybe_prefix
class LiLiCorrLatticeAttention(nn.Module):
def __init__(
self,
hidden_size: int,
num_heads: int,
quant_config: QuantizationConfig | None = None,
prefix: str = "",
) -> None:
super().__init__()
if num_heads <= 0 or hidden_size % num_heads != 0:
raise ValueError(
f"LiLiCorr hidden_size={hidden_size} must be divisible "
f"by num_heads={num_heads}."
)
self.hidden_size = hidden_size
self.num_heads = num_heads
self.head_dim = self.hidden_size // self.num_heads
# Exported lattice QKV parameters stay in the model's floating dtype.
self.in_proj_weight = nn.Parameter(
torch.empty(3 * hidden_size, hidden_size), requires_grad=False
)
self.in_proj_bias = nn.Parameter(
torch.empty(3 * hidden_size), requires_grad=False
)
self.out_proj = ReplicatedLinear(
hidden_size,
hidden_size,
return_bias=False,
quant_config=quant_config,
prefix=maybe_prefix(prefix, "out_proj"),
)
def forward(
self, hidden_states: torch.Tensor, attention_bias: torch.Tensor
) -> torch.Tensor:
bsz, seq_len, _ = hidden_states.shape
qkv = F.linear(hidden_states, self.in_proj_weight, self.in_proj_bias)
q, k, v = qkv.chunk(3, dim=-1)
shape = (bsz, seq_len, self.num_heads, self.head_dim)
q = q.view(shape).transpose(1, 2)
k = k.view(shape).transpose(1, 2)
v = v.view(shape).transpose(1, 2)
# Attention and MMEncoderAttention cannot accept this learned per-head
# additive bias. The full candidate lattice has no KV-cache state.
out = F.scaled_dot_product_attention(
q,
k,
v,
attn_mask=attention_bias,
)
return self.out_proj(
out.transpose(1, 2).reshape(bsz, seq_len, self.hidden_size)
)
class LiLiCorrLayer(nn.Module):
def __init__(
self,
hidden_size: int,
num_heads: int,
mlp_ratio: float,
rms_norm_eps: float,
quant_config: QuantizationConfig | None = None,
prefix: str = "",
) -> None:
super().__init__()
self.attn_norm = nn.RMSNorm(hidden_size, eps=rms_norm_eps)
self.attn = LiLiCorrLatticeAttention(
hidden_size, num_heads, quant_config, maybe_prefix(prefix, "attn")
)
self.mlp_norm = nn.RMSNorm(hidden_size, eps=rms_norm_eps)
mlp_hidden_size = int(hidden_size * mlp_ratio)
self.mlp = NemotronHMLP(
config=None,
hidden_size=hidden_size,
intermediate_size=mlp_hidden_size,
hidden_act="silu",
bias=True,
quant_config=quant_config,
prefix=maybe_prefix(prefix, "mlp"),
)
def forward(
self, hidden_states: torch.Tensor, attention_bias: torch.Tensor
) -> torch.Tensor:
hidden_states = hidden_states + self.attn(
self.attn_norm(hidden_states), attention_bias
)
return hidden_states + self.mlp(self.mlp_norm(hidden_states))
@support_torch_compile
class LiLiCorrHead(nn.Module):
num_candidate_features = 5
def __init__(
self,
*,
model_hidden_size: int,
block_size: int,
rms_norm_eps: float,
config: dict[str, Any],
quant_config: QuantizationConfig | None = None,
prefix: str = "",
) -> None:
super().__init__()
hidden_size = config["lilicorr_hidden_size"] or model_hidden_size
self.block_size = block_size
self.num_candidate_slots = block_size - 1
self.candidate_topk = config["lilicorr_candidate_topk"]
if (
self.candidate_topk <= 0
or self.candidate_topk > 16
or self.candidate_topk & (self.candidate_topk - 1)
):
raise ValueError("LiLiCorr candidate_topk must be a power of two <= 16.")
self.hidden_size = hidden_size
self.num_heads = config["lilicorr_num_heads"]
self.mlp_ratio = config["lilicorr_mlp_ratio"]
self.factor_dim = config["lilicorr_factor_dim"]
self.vector_eps = config["lilicorr_vector_eps"]
self.logit_scale = config["lilicorr_logit_scale"]
# Candidate IDs and embeddings are replicated across TP ranks, so each
# rank scores the same lattice; the MLPs use standard tensor parallelism.
self.token_proj = (
nn.Identity()
if model_hidden_size == hidden_size
else ReplicatedLinear(
model_hidden_size,
hidden_size,
return_bias=False,
quant_config=quant_config,
prefix=maybe_prefix(prefix, "token_proj"),
)
)
self.pass_hidden_proj = ReplicatedLinear(
model_hidden_size,
hidden_size,
return_bias=False,
quant_config=quant_config,
prefix=maybe_prefix(prefix, "pass_hidden_proj"),
)
self.feature_norm = nn.LayerNorm(self.num_candidate_features)
self.feature_mlp = NemotronHMLP(
config=None,
hidden_size=self.num_candidate_features,
intermediate_size=hidden_size,
hidden_act="silu",
output_size=hidden_size,
bias=True,
quant_config=quant_config,
prefix=maybe_prefix(prefix, "feature_mlp"),
)
self.slot_embedding = nn.Parameter(
torch.zeros(1, 1, self.num_candidate_slots, 1, hidden_size)
)
self.rank_embedding = nn.Parameter(
torch.zeros(1, 1, 1, self.candidate_topk, hidden_size)
)
self.relative_slot_bias = nn.Parameter(
torch.zeros(self.num_heads, 2 * self.block_size - 1)
)
self.same_slot_bias = nn.Parameter(torch.zeros(self.num_heads))
self.context_proj = ReplicatedLinear(
model_hidden_size,
hidden_size,
return_bias=False,
quant_config=quant_config,
prefix=maybe_prefix(prefix, "context_proj"),
)
self.layers = nn.ModuleList(
[
LiLiCorrLayer(
hidden_size=hidden_size,
num_heads=self.num_heads,
mlp_ratio=self.mlp_ratio,
rms_norm_eps=rms_norm_eps,
quant_config=quant_config,
prefix=maybe_prefix(prefix, f"layers.{i}"),
)
for i in range(config["lilicorr_num_layers"])
]
)
self.output_norm = nn.RMSNorm(hidden_size, eps=rms_norm_eps)
self.anchor_norm = nn.RMSNorm(hidden_size, eps=rms_norm_eps)
self.factor_input_proj = ReplicatedLinear(
hidden_size * 3,
hidden_size,
return_bias=False,
quant_config=None,
prefix=maybe_prefix(prefix, "factor_input_proj"),
)
# Each candidate gets outgoing and incoming vectors. After normalization,
# their dot product scores a transition from slot s to slot s+1:
# pair_score[s, i, j] = dot(out[s, i], incoming[s+1, j])
# The anchor supplies the outgoing vector for the first slot.
self.out_head = ReplicatedLinear(
hidden_size,
self.factor_dim,
return_bias=False,
quant_config=None,
prefix=maybe_prefix(prefix, "out_head"),
)
self.in_head = ReplicatedLinear(
hidden_size,
self.factor_dim,
return_bias=False,
quant_config=None,
prefix=maybe_prefix(prefix, "in_head"),
)
self.anchor_out_head = ReplicatedLinear(
hidden_size,
self.factor_dim,
return_bias=False,
quant_config=quant_config,
prefix=maybe_prefix(prefix, "anchor_out_head"),
)
self._fused_edge_weight: torch.Tensor | None = None
self._fused_edge_bias: torch.Tensor | None = None
self._factor_input_splits: tuple[torch.Tensor, ...]
self._attn_bias: torch.Tensor | None = None
self._rank_frac_col: torch.Tensor
self._is_top1_col: torch.Tensor
@torch.no_grad()
def materialize_inference_buffers(
self, device: torch.device, dtype: torch.dtype
) -> None:
"""Build inference-only views/copies after checkpoint weights are loaded."""
topk = self.candidate_topk
self._attn_bias = self._build_attention_bias(device=device, dtype=dtype)
# Both projections consume the same candidate features f:
# out = W_out @ f + b_out
# incoming = W_in @ f + b_in
# Stack weights and biases to compute both in one linear call. score()
# splits the result and normalizes the two vectors separately.
self._fused_edge_weight = (
torch.cat([self.out_head.weight, self.in_head.weight], dim=0)
.to(device=device, dtype=dtype)
.contiguous()
)
self._fused_edge_bias = (
torch.cat([self.out_head.bias, self.in_head.bias], dim=0)
.to(device=device, dtype=dtype)
.contiguous()
)
# For candidate state x and request anchor a, the trained projection is:
# W @ concat(x, a, x*a) + b
# Split W into input blocks to evaluate the same expression as:
# W_x @ x + W_a @ a + W_cross @ (x*a) + b
# Here x*a is elementwise. This avoids concatenating inputs and reuses
# W_a @ a across candidates, adding the original bias only once.
weight = self.factor_input_proj.weight
hdim = self.hidden_size
self._factor_input_splits = (
weight[:, :hdim].contiguous(),
weight[:, hdim : 2 * hdim].contiguous(),
weight[:, 2 * hdim :].contiguous(),
)
if topk > 1:
rank_frac = torch.arange(topk, device=device, dtype=torch.float32).view(
1, 1, topk
) / float(topk - 1)
else:
rank_frac = torch.zeros(1, 1, topk, device=device, dtype=torch.float32)
is_top1 = torch.zeros(1, 1, topk, device=device, dtype=torch.float32)
is_top1[..., 0] = 1.0
self._rank_frac_col = rank_frac.contiguous()
self._is_top1_col = is_top1.contiguous()
def _build_attention_bias(
self, *, device: torch.device, dtype: torch.dtype
) -> torch.Tensor:
topk = self.candidate_topk
slot_ids = torch.arange(
self.num_candidate_slots, device=device, dtype=torch.long
).repeat_interleave(topk)
# Attention flattens [slot, candidate] into one sequence. With topk=2,
# the candidate slot IDs are [0, 0, 1, 1, ...]. Each head learns:
# bias(q, k) = relative_bias[slot(q) - slot(k) + offset]
# + same_slot_bias * (slot(q) == slot(k))
# The same-slot term includes distinct candidates at the same position.
# This is an additive preference, not a causal mask. Keep the trained
# offset = block_size-1 when score() selects a shorter draft's prefix,
# so each distance continues to use the same learned weight.
rel = slot_ids.view(-1, 1) - slot_ids.view(1, -1)
bias = self.relative_slot_bias[:, rel + self.block_size - 1]
same_slot = slot_ids.view(-1, 1) == slot_ids.view(1, -1)
bias = bias + same_slot.unsqueeze(0).to(
dtype=bias.dtype
) * self.same_slot_bias.view(-1, 1, 1)
return bias.to(device=device, dtype=dtype).contiguous()
def score(
self,
*,
token_embeddings: torch.Tensor,
candidate_log_probs: torch.Tensor,
pass_hidden: torch.Tensor,
anchor_hidden: torch.Tensor,
anchor_valid: torch.Tensor,
) -> tuple[torch.Tensor, torch.Tensor]:
if self._attn_bias is None:
raise RuntimeError(
"Call materialize_inference_buffers() after loading the LiLiCorr head."
)
bsz, n_slots, topk = candidate_log_probs.shape
if not 1 <= n_slots <= self.num_candidate_slots:
raise ValueError(
f"LiLiCorr supports 1..{self.num_candidate_slots} candidate slots, "
f"got {n_slots}."
)
if topk != self.candidate_topk:
raise ValueError(
f"LiLiCorr expects candidate_topk={self.candidate_topk}, got {topk}."
)
proj_dtype = self.slot_embedding.dtype
if token_embeddings.dtype != proj_dtype:
token_embeddings = token_embeddings.to(proj_dtype)
if pass_hidden.dtype != proj_dtype:
pass_hidden = pass_hidden.to(proj_dtype)
token_states = self.token_proj(token_embeddings)
pass_states = self.pass_hidden_proj(pass_hidden).unsqueeze(-2)
log_probs = candidate_log_probs.float()
features = torch.stack(
[
log_probs,
log_probs.exp(),
log_probs - log_probs.max(dim=-1, keepdim=True).values,
self._rank_frac_col.expand_as(log_probs),
self._is_top1_col.expand_as(log_probs),
],
dim=-1,
)
hidden_states = token_states + pass_states
hidden_states = hidden_states + self.feature_mlp(
self.feature_norm(features.to(dtype=token_states.dtype))
)
hidden_states = hidden_states + self.slot_embedding[:, 0, :n_slots]
hidden_states = hidden_states + self.rank_embedding[:, 0]
hidden_states = hidden_states.reshape(bsz, n_slots * topk, self.hidden_size)
anchor_state = self.context_proj(anchor_hidden)
anchor_state = anchor_state * anchor_valid[:, None].to(anchor_state.dtype)
# Shorter drafts use the learned prefix; keep checkpoint parameter shapes.
lattice = n_slots * topk
attention_bias = self._attn_bias[None, :, :lattice, :lattice]
for layer in self.layers:
hidden_states = layer(hidden_states, attention_bias)
hidden_states = self.output_norm(hidden_states).reshape(
bsz, n_slots, topk, self.hidden_size
)
anchor_state = self.anchor_norm(anchor_state)
w_self, w_anchor, w_cross = self._factor_input_splits
anchor_row = anchor_state[:, None, None, :]
pre = F.linear(hidden_states, w_self, self.factor_input_proj.bias)
pre = pre + F.linear(anchor_row, w_anchor)
pre = pre + F.linear(hidden_states * anchor_row, w_cross)
factor_hidden = F.silu(pre)
edges = F.linear(factor_hidden, self._fused_edge_weight, self._fused_edge_bias)
out_vec, in_vec = F.normalize(
edges.unflatten(-1, (2, self.factor_dim)), dim=-1, eps=self.vector_eps
).unbind(-2)
anchor_out = F.normalize(
self.anchor_out_head(anchor_state), dim=-1, eps=self.vector_eps
)
# start_scores[b, j] scores anchor -> candidate j in slot zero.
# pair_scores[b, s, i, j] scores candidate i in slot s -> candidate j
# in slot s+1. Normalized factor dot products encode learned compatibility;
# forward() scales them into logits for the candidate-path selector.
start_scores = (anchor_out[:, None, :] * in_vec[:, 0, :, :]).sum(dim=-1)
pair_scores = torch.matmul(out_vec[:, :-1], in_vec[:, 1:].transpose(-1, -2))
return (start_scores, pair_scores)
def forward(
self,
token_embeddings: torch.Tensor,
candidate_log_probs: torch.Tensor,
pass_hidden: torch.Tensor,
anchor_hidden: torch.Tensor,
anchor_valid: torch.Tensor,
) -> torch.Tensor:
start, pairs = self.score(
token_embeddings=token_embeddings,
candidate_log_probs=candidate_log_probs,
pass_hidden=pass_hidden,
anchor_hidden=anchor_hidden,
anchor_valid=anchor_valid,
)
start = self.logit_scale * start.float()
pairs = self.logit_scale * pairs.float()
# The shared walk starts at predecessor index zero; all first rows agree.
first = start[:, None, None, :].expand(-1, 1, self.candidate_topk, -1)
return torch.cat((first, pairs), dim=1)
class LiLiCorr(DFlashQwen3Model):
def __init__(
self, *, vllm_config: VllmConfig, start_layer_id: int = 0, prefix: str = ""
):
spec = vllm_config.speculative_config
assert spec is not None
config = spec.draft_model_config.hf_config
draft_config = config.dflash_config
if not draft_config.get("lilicorr_enabled", True):
raise ValueError("LiLiCorr requires lilicorr_enabled=True.")
block_size = draft_config.get("block_size", getattr(config, "block_size", 0))
if not 1 <= spec.num_speculative_tokens < block_size:
raise ValueError(
"LiLiCorr requires 1 <= num_speculative_tokens < trained block_size."
)
taps = draft_config.get("conv_kernel_size", 0)
groups = draft_config.get("conv_group_size", 0)
if taps < 0 or groups < 0 or bool(taps) != bool(groups):
raise ValueError(
"LiLiCorr convolution geometry must enable both taps and group size."
)
if taps:
self.decoder_layer_cls = DFlash2Qwen3DecoderLayer
super().__init__(
vllm_config=vllm_config, start_layer_id=start_layer_id, prefix=prefix
)
with set_model_tag("lilicorr_head"):
self.lilicorr = LiLiCorrHead(
model_hidden_size=config.hidden_size,
block_size=block_size,
rms_norm_eps=config.rms_norm_eps,
config=draft_config,
quant_config=self.quant_config,
prefix=maybe_prefix(prefix, "lilicorr"),
)
class LiLiCorrForCausalLM(DFlashQwen3ForCausalLM):
model_cls = LiLiCorr
def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):
spec = vllm_config.speculative_config
assert spec is not None
super().__init__(vllm_config=vllm_config, prefix=prefix)
if self.draft_id_to_target_id is not None:
raise ValueError("LiLiCorr candidates require the full target vocabulary.")
self.has_own_lm_head = bool(getattr(self.config, "has_own_lm_head", False))
# The correlator consumes raw candidate-head log-softmax features.
self.candidate_logits_processor = LogitsProcessor(self.config.vocab_size)
def compute_candidates(
self, hidden_states: torch.Tensor
) -> tuple[torch.Tensor, torch.Tensor]:
return self.candidate_logits_processor.get_top_k_tokens(
self.lm_head,
hidden_states,
self.model.lilicorr.candidate_topk,
return_log_probs=True,
)
def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]):
quantized_metadata: set[str] = set()
for name, module in self.named_modules():
name = name.removeprefix("model.")
if not isinstance(module, (LinearBase, ParallelLMHead)) or isinstance(
module.quant_method, UnquantizedLinearMethod
):
continue
required = {"weight", "bias"}
method = module.quant_method
if (
isinstance(method, ModelOptLinearMethod)
and method.spec.weight == kNvfp4Static
):
# NVFP4 scales are trained checkpoint data. Only W4A16's
# deprecated, unused input_scale may be absent.
required.update(("weight_scale", "weight_scale_2"))
if method.spec.activation is not None:
required.add("input_scale")
quantized_metadata.update(
f"{name}.{parameter}"
for parameter, _ in module.named_parameters(recurse=False)
if parameter not in required
)
expected = {
name.removeprefix("model.")
for name, _ in self.named_parameters()
if (
name.startswith("model.lilicorr.")
or ".attention_conv." in name
or ".mlp_conv." in name
or (self.has_own_lm_head and name.startswith("lm_head."))
)
and name.removeprefix("model.") not in quantized_metadata
}
seen: set[str] = set()
head_weights: list[tuple[str, torch.Tensor]] = []
def normalized_weights():
for name, value in weights:
name = name.removeprefix("model.")
seen.add(name)
# The backbone stacks up_proj into gate_up_proj. Load the
# nongated correlator separately to avoid that transformation.
if name.startswith("lilicorr."):
head_weights.append((name.removeprefix("lilicorr."), value))
else:
yield name, value
super().load_weights(normalized_weights())
AutoWeightsLoader(self.model.lilicorr).load_weights(head_weights)
missing = expected - seen
if missing:
raise ValueError(
f"LiLiCorr checkpoint coverage mismatch: missing={sorted(missing)}"
)
head = self.model.lilicorr
parameter = head.slot_embedding
head.materialize_inference_buffers(parameter.device, parameter.dtype)
@@ -34,6 +34,7 @@ from vllm.model_executor.models.interfaces import (
HasInnerState,
IsHybrid,
MultiModalEmbeddings,
SupportsEagle3,
SupportsLoRA,
SupportsMultiModal,
SupportsMultiModalPruning,
@@ -928,6 +929,7 @@ class NemotronH_Nano_VL_V2(
nn.Module,
HasInnerState,
IsHybrid,
SupportsEagle3,
SupportsMultiModal,
SupportsMultiModalPruning,
SupportsLoRA,
+7 -4
View File
@@ -34,7 +34,7 @@ from vllm.config.parallel import ParallelConfig
from vllm.distributed import get_ep_group, get_tensor_model_parallel_world_size
from vllm.distributed.communication_op import tensor_model_parallel_all_gather
from vllm.distributed.parallel_state import get_pp_group
from vllm.model_executor.layers.activation import ReLUSquaredActivation
from vllm.model_executor.layers.activation import get_act_fn
from vllm.model_executor.layers.attention import Attention
from vllm.model_executor.layers.fused_moe import (
FusedMoEFactory,
@@ -93,7 +93,7 @@ from vllm.transformers_utils.configs.nemotron_h import NemotronHConfig
class NemotronHMLP(nn.Module):
def __init__(
self,
config: NemotronHConfig,
config: NemotronHConfig | None,
hidden_size: int,
intermediate_size: int,
quant_config: QuantizationConfig | None = None,
@@ -101,6 +101,9 @@ class NemotronHMLP(nn.Module):
reduce_results: bool = True,
is_sequence_parallel: bool = False,
prefix: str = "",
*,
hidden_act: str = "relu2",
output_size: int | None = None,
) -> None:
super().__init__()
@@ -114,14 +117,14 @@ class NemotronHMLP(nn.Module):
)
self.down_proj = RowParallelLinear(
input_size=intermediate_size,
output_size=hidden_size,
output_size=hidden_size if output_size is None else output_size,
bias=bias,
quant_config=quant_config,
reduce_results=reduce_results,
disable_tp=is_sequence_parallel,
prefix=f"{prefix}.down_proj",
)
self.act_fn = ReLUSquaredActivation()
self.act_fn = get_act_fn(hidden_act)
def forward(self, x: torch.Tensor):
x, _ = self.up_proj(x)
+70 -15
View File
@@ -4,6 +4,7 @@
import io
from collections.abc import Iterable
import regex as re
import torch
import torch.nn.functional as F
from torch import nn
@@ -23,6 +24,7 @@ from vllm.model_executor.layers.linear import (
QKVParallelLinear,
ReplicatedLinear,
RowParallelLinear,
UnquantizedLinearMethod,
)
from vllm.model_executor.layers.logits_processor import LogitsProcessor
from vllm.model_executor.layers.quantization.base_config import QuantizationConfig
@@ -54,6 +56,31 @@ logger = init_logger(__name__)
_SLIDING_ATTENTION = "sliding_attention"
_DRAFT_LAYER_PATTERN = re.compile(r"(?<![A-Za-z0-9_.])layers\.(\d+)")
def _add_global_draft_layer_exclusions(
quant_config: QuantizationConfig | None,
start_layer_id: int,
num_hidden_layers: int,
) -> None:
"""Add runtime layer aliases for checkpoint-local quant exclusions."""
if quant_config is None or start_layer_id == 0:
return
exclusions = getattr(quant_config, "exclude_modules", None)
if not isinstance(exclusions, list):
return
def offset_local_layer(match: re.Match[str]) -> str:
layer_idx = int(match.group(1))
if layer_idx >= num_hidden_layers:
return match.group(0)
return f"layers.{layer_idx + start_layer_id}"
for exclusion in tuple(exclusions):
global_exclusion = _DRAFT_LAYER_PATTERN.sub(offset_local_layer, exclusion)
if global_exclusion != exclusion and global_exclusion not in exclusions:
exclusions.append(global_exclusion)
def _dflash_layer_causal(config: Qwen3Config, layer_idx: int) -> bool:
@@ -394,6 +421,9 @@ class DFlashQwen3Model(nn.Module):
self.config = vllm_config.speculative_config.draft_model_config.hf_config
self.vocab_size = self.config.vocab_size
self.quant_config = get_draft_quant_config(vllm_config)
_add_global_draft_layer_exclusions(
self.quant_config, start_layer_id, self.config.num_hidden_layers
)
drafter_config = getattr(self.config, "eagle_config", {})
drafter_config.update(getattr(self.config, "dflash_config", {}))
@@ -473,14 +503,25 @@ class DFlashQwen3Model(nn.Module):
has_bias: bool,
) -> None:
self._hidden_norm_weight = self.hidden_norm.weight.data
self._context_qkv_projs = [a.qkv_proj for a in layers_attn]
self._context_q_sizes = [a.q_size for a in layers_attn]
# KV projection weights: [num_layers * 2 * kv_size, hidden_size]
kv_weights = [a.qkv_proj.weight[a.q_size :] for a in layers_attn]
self._fused_kv_weight = torch.cat(kv_weights, dim=0)
if has_bias:
kv_biases = [a.qkv_proj.bias[a.q_size :] for a in layers_attn]
self._fused_kv_bias: torch.Tensor | None = torch.cat(kv_biases, dim=0)
if all(
isinstance(proj.quant_method, UnquantizedLinearMethod)
for proj in self._context_qkv_projs
):
# KV projection weights: [num_layers * 2 * kv_size, hidden_size]
kv_weights = [a.qkv_proj.weight[a.q_size :] for a in layers_attn]
self._fused_kv_weight: torch.Tensor | None = torch.cat(kv_weights, dim=0)
if has_bias:
kv_biases = [a.qkv_proj.bias[a.q_size :] for a in layers_attn]
self._fused_kv_bias: torch.Tensor | None = torch.cat(kv_biases, dim=0)
else:
self._fused_kv_bias = None
else:
# Quantized linear weights may use packed storage that cannot be
# consumed by F.linear. Run each projection through its quant method.
self._fused_kv_weight = None
self._fused_kv_bias = None
# K-norm weights stacked into one contiguous [num_layers, head_dim]
@@ -548,17 +589,26 @@ class DFlashQwen3Model(nn.Module):
self._hidden_norm_weight,
self._rms_norm_eps,
)
all_kv_flat = F.linear(
normed_context_states, self._fused_kv_weight, self._fused_kv_bias
)
if self._fused_kv_weight is not None:
all_kv_flat = F.linear(
normed_context_states, self._fused_kv_weight, self._fused_kv_bias
)
all_kv = all_kv_flat.view(num_ctx, num_layers, 2, num_kv_heads, head_dim)
else:
layer_kv = []
for proj, q_size in zip(
self._context_qkv_projs, self._context_q_sizes, strict=True
):
qkv, _ = proj(normed_context_states)
layer_kv.append(
qkv[:, q_size:].view(num_ctx, 2, num_kv_heads, head_dim)
)
all_kv = torch.stack(layer_kv, dim=1)
# Single contiguous copy that separates K/V and transposes to
# layer-major layout. Result: [2, L, num_ctx, nkv, hd] contiguous.
# layer-major layout. Result: [2, L, num_ctx, nkv, hd] contiguous.
# Indexing dim-0 gives contiguous [L, num_ctx, nkv, hd] for K and V.
all_kv = (
all_kv_flat.view(num_ctx, num_layers, 2, num_kv_heads, head_dim)
.permute(2, 1, 0, 3, 4)
.contiguous()
)
all_kv = all_kv.permute(2, 1, 0, 3, 4).contiguous()
all_k = all_kv[0] # [L, num_ctx, nkv, hd], contiguous
all_v = all_kv[1] # [L, num_ctx, nkv, hd], contiguous
return all_k, all_v
@@ -724,6 +774,11 @@ class DFlashQwen3ForCausalLM(Qwen3ForCausalLM):
self.lm_head = ParallelLMHead(
self.config.draft_vocab_size,
self.config.hidden_size,
quant_config=(
get_draft_quant_config(vllm_config)
if getattr(self.config, "has_own_lm_head", False)
else None
),
prefix=maybe_prefix(prefix, "lm_head"),
)
self.logits_processor = LogitsProcessor(
+3 -1
View File
@@ -176,6 +176,7 @@ class DFlashGroupedConv(nn.Module):
block_size: int,
params_dtype: torch.dtype,
prefix: str,
quant_config: QuantizationConfig | None = None,
) -> None:
super().__init__()
if hidden_size % group_size:
@@ -195,7 +196,7 @@ class DFlashGroupedConv(nn.Module):
2 * taps * self.num_groups,
bias=False,
params_dtype=params_dtype,
quant_config=None,
quant_config=quant_config,
prefix=maybe_prefix(prefix, "kernel_projection"),
return_bias=False,
)
@@ -254,6 +255,7 @@ class DFlash2Qwen3DecoderLayer(DFlashQwen3DecoderLayer):
# Query tokens per request: the bonus token plus the mask tokens.
block_size=1 + speculative_config.num_speculative_tokens,
params_dtype=vllm_config.model_config.dtype,
quant_config=quant_config,
)
self.attention_conv = DFlashGroupedConv(
**conv_args, prefix=maybe_prefix(prefix, "attention_conv")
+1
View File
@@ -633,6 +633,7 @@ _SPECULATIVE_DECODING_MODELS = {
"EagleLlama4ForCausalLM": ("llama4_eagle", "EagleLlama4ForCausalLM"),
"EagleMiniCPMForCausalLM": ("minicpm_eagle", "EagleMiniCPMForCausalLM"),
"DFlashDraftModel": ("qwen3_dflash", "DFlashQwen3ForCausalLM"),
"LiLiCorrDraftModel": ("lilicorr", "LiLiCorrForCausalLM"),
"DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM"),
# Muse Glimmer's DFlash draft head, reusing the generic qwen3_dflash
# implementation. EAGLEConfig rewrites a dflash draft's architecture to
+3 -1
View File
@@ -68,7 +68,9 @@ class EAGLEConfig(PreTrainedConfig):
)
kwargs["architectures"] = [
arch
if arch.startswith("DFlash") or arch.endswith("DFlash")
if arch.startswith("DFlash")
or arch.endswith("DFlash")
or arch == "LiLiCorrDraftModel"
else f"DFlash{arch}"
for arch in self.model.architectures
]
@@ -15,6 +15,12 @@ def init_speculator(vllm_config: VllmConfig, device: torch.device):
return ExtractHiddenStatesSpeculator(vllm_config, device)
elif speculative_config.method == "dflash":
if "LiLiCorrDraftModel" in speculative_config.draft_model_config.architectures:
from vllm.v1.worker.gpu.spec_decode.lilicorr.speculator import (
LiLiCorrSpeculator,
)
return LiLiCorrSpeculator(vllm_config, device)
if "DFlash2DraftModel" in speculative_config.draft_model_config.architectures:
from vllm.v1.worker.gpu.spec_decode.dflash2.speculator import (
DFlash2Speculator,
@@ -322,6 +322,11 @@ class DFlashSpeculator(DraftModelSpeculator):
context_slots,
)
def prepare_context_anchor(
self, input_batch: InputBatch, num_rejected: torch.Tensor
) -> None:
"""Publish context features required by a draft's candidate head."""
@torch.inference_mode()
def propose(
self,
@@ -369,6 +374,7 @@ class DFlashSpeculator(DraftModelSpeculator):
else:
hidden_states = last_hidden_states
self.hidden_states[:num_target_tokens].copy_(hidden_states[:num_target_tokens])
self.prepare_context_anchor(input_batch, num_rejected)
if dummy_run and skip_attn_for_dummy_run:
# Memory profiling path: block_tables / kv_cache_config are not initialized.
@@ -61,8 +61,12 @@ def load_dflash_model(target_model: nn.Module, vllm_config: VllmConfig) -> nn.Mo
target_lm_head = get_target_lm_head(target_model, target_language_model)
draft_lm_head = getattr(dflash_model, "lm_head", None)
if target_lm_head is not None and _should_share(
dflash_model, "has_own_lm_head", draft_lm_head, target_lm_head
if (
target_lm_head is not None
and not getattr(dflash_model.config, "has_own_lm_head", False)
and _should_share(
dflash_model, "has_own_lm_head", draft_lm_head, target_lm_head
)
):
if draft_lm_head is not None:
del dflash_model.lm_head
@@ -108,26 +108,93 @@ def _cache_draft_logits_kernel(
tl.store(cached_candidate_ptr + cache_base + offsets, token_ids, mask=mask)
class CandidateSampler:
"""The shared DFlash2/LiLiCorr candidate walk and realized proposal cache."""
def __init__(
self, max_num_reqs: int, num_steps: int, top_k: int, device: torch.device
):
self.num_steps = num_steps
self.top_k = top_k
self.scores = torch.empty(
max_num_reqs, num_steps, top_k, dtype=torch.float32, device=device
)
self.cached_candidate_ids = torch.zeros(
self.scores.shape, dtype=torch.int64, device=device
)
def sample(
self,
candidate_ids: torch.Tensor,
scores: torch.Tensor,
num_reqs: int,
sample_pos: torch.Tensor,
idx_mapping: torch.Tensor,
temperature: torch.Tensor,
seeds: torch.Tensor,
draft_tokens: torch.Tensor,
draft_logits: torch.Tensor | None,
use_fp64: bool,
) -> None:
candidate_ids = candidate_ids.contiguous()
sample_pos = sample_pos.contiguous()
idx_mapping = idx_mapping.contiguous()
block_k = triton.next_power_of_2(self.top_k)
_selector_walk_kernel[(num_reqs,)](
scores.contiguous(),
candidate_ids.contiguous(),
sample_pos,
idx_mapping,
temperature,
seeds,
draft_tokens,
self.scores,
num_steps=self.num_steps,
top_k=self.top_k,
BLOCK_K=block_k,
SAMPLE_PROBABILISTIC=draft_logits is not None,
USE_FP64=use_fp64,
num_warps=1,
)
if draft_logits is not None:
self._cache_draft_logits(
candidate_ids, num_reqs * self.num_steps, idx_mapping, draft_logits
)
def _cache_draft_logits(
self,
candidate_ids: torch.Tensor,
num_sample: int,
idx_mapping: torch.Tensor,
draft_logits: torch.Tensor,
) -> None:
block_k = triton.next_power_of_2(self.top_k)
_cache_draft_logits_kernel[(num_sample,)](
draft_logits,
self.cached_candidate_ids,
candidate_ids,
self.scores,
idx_mapping,
draft_logits.stride(0),
draft_logits.stride(1),
num_steps=self.num_steps,
top_k=self.top_k,
BLOCK_K=block_k,
num_warps=1,
)
class DFlash2Speculator(DFlashSpeculator):
_speculator_name = "DFlash2"
_candidate_top_k_key = "selector_top_k"
def __init__(self, vllm_config: VllmConfig, device: torch.device):
super().__init__(vllm_config, device)
draft_config = self.draft_model_config.hf_config.dflash_config
self.selector_top_k = int(draft_config["selector_top_k"])
self._anchor_indices = (
torch.arange(self.max_num_reqs, dtype=torch.int64, device=device)
* self.num_query_per_req
)
self._selector_scores = torch.empty(
self.max_num_reqs,
self.num_speculative_steps,
self.selector_top_k,
dtype=torch.float32,
device=device,
)
self._cached_candidate_ids = torch.zeros(
self._selector_scores.shape, dtype=torch.int64, device=device
self.top_k = int(draft_config[self._candidate_top_k_key])
self.candidate_sampler = CandidateSampler(
self.max_num_reqs, self.num_speculative_steps, self.top_k, device
)
def draft_logits_spec(self, vllm_config: VllmConfig) -> tuple[torch.dtype, float]:
@@ -136,48 +203,6 @@ class DFlash2Speculator(DFlashSpeculator):
# candidates.
return torch.float32, -float("inf")
def _sample_path(
self,
candidate_ids: torch.Tensor,
scores: torch.Tensor,
num_reqs: int,
) -> None:
block_k = triton.next_power_of_2(self.selector_top_k)
_selector_walk_kernel[(num_reqs,)](
scores.contiguous(),
candidate_ids.contiguous(),
self.sample_pos,
self.sample_idx_mapping,
self.temperature,
self.seeds,
self.draft_tokens,
self._selector_scores,
num_steps=self.num_speculative_steps,
top_k=self.selector_top_k,
BLOCK_K=block_k,
SAMPLE_PROBABILISTIC=self.draft_logits is not None,
USE_FP64=self.use_fp64_gumbel,
num_warps=1,
)
def _cache_draft_logits(self, candidate_ids: torch.Tensor, num_sample: int) -> None:
draft_logits = self.draft_logits
assert draft_logits is not None
block_k = triton.next_power_of_2(self.selector_top_k)
_cache_draft_logits_kernel[(num_sample,)](
draft_logits,
self._cached_candidate_ids,
candidate_ids,
self._selector_scores,
self.sample_idx_mapping,
draft_logits.stride(0),
draft_logits.stride(1),
num_steps=self.num_speculative_steps,
top_k=self.selector_top_k,
BLOCK_K=block_k,
num_warps=1,
)
def _generate_draft(
self,
num_reqs: int,
@@ -202,22 +227,39 @@ class DFlash2Speculator(DFlashSpeculator):
hidden_states.flatten(0, 1)
)
candidate_ids = candidate_ids.view(
num_reqs, self.num_speculative_steps, self.selector_top_k
num_reqs, self.num_speculative_steps, self.top_k
)
unary_logits = unary_logits.view_as(candidate_ids)
anchor_token_ids = self.input_buffers.input_ids[self._anchor_indices[:num_reqs]]
scores = self.model.model.candidate_selector(
scores = self._score_candidates(candidate_ids, unary_logits, hidden_states)
self.candidate_sampler.sample(
candidate_ids,
unary_logits,
hidden_states,
anchor_token_ids,
scores,
num_reqs,
self.sample_pos,
self.sample_idx_mapping,
self.temperature,
self.seeds,
self.draft_tokens,
self.draft_logits,
self.use_fp64_gumbel,
)
self._sample_path(candidate_ids, scores, num_reqs)
if self.enable_adaptive_verification:
self._maybe_predict_acceptance(
self._selector_scores[:num_reqs].flatten(0, 1),
self.candidate_sampler.scores[:num_reqs].flatten(0, 1),
self.sample_idx_mapping[:num_sample],
self.sample_col[:num_sample],
)
if self.draft_logits is not None:
self._cache_draft_logits(candidate_ids, num_sample)
def _score_candidates(
self,
candidate_ids: torch.Tensor,
unary_logits: torch.Tensor,
hidden_states: torch.Tensor,
) -> torch.Tensor:
num_reqs = candidate_ids.shape[0]
anchor_token_ids = self.input_buffers.input_ids[
: num_reqs * self.num_query_per_req : self.num_query_per_req
]
return self.model.model.candidate_selector(
candidate_ids, unary_logits, hidden_states, anchor_token_ids
)
@@ -0,0 +1,2 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
@@ -0,0 +1,97 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import torch
from torch import nn
from vllm.config import VllmConfig
from vllm.model_executor.layers.vocab_parallel_embedding import VocabParallelEmbedding
from vllm.v1.worker.gpu.input_batch import InputBatch
from vllm.v1.worker.gpu.spec_decode.dflash2.speculator import DFlash2Speculator
from vllm.v1.worker.gpu.spec_decode.eagle.utils import get_target_lm_head
class LiLiCorrSpeculator(DFlash2Speculator):
_speculator_name = "LiLiCorr"
_candidate_top_k_key = "lilicorr_candidate_topk"
def __init__(self, vllm_config: VllmConfig, device: torch.device):
super().__init__(vllm_config, device)
config = self.draft_model_config.hf_config
self.anchor_hidden = torch.zeros(
self.max_num_reqs, config.hidden_size, dtype=self.dtype, device=device
)
self.anchor_valid = torch.zeros(
self.max_num_reqs, dtype=torch.bool, device=device
)
def load_draft_model(
self, target_model: nn.Module, target_attn_layer_names: set[str]
) -> nn.Module:
model = super().load_draft_model(target_model, target_attn_layer_names)
language_model = (
target_model.get_language_model()
if hasattr(target_model, "get_language_model")
else target_model
)
inner = getattr(language_model, "model", language_model)
embedding = getattr(inner, "embed_tokens", None) or getattr(
inner, "embedding", None
)
lm_head = (
model.lm_head
if model.has_own_lm_head
else get_target_lm_head(target_model, language_model)
)
if not isinstance(embedding, VocabParallelEmbedding) or not isinstance(
lm_head, VocabParallelEmbedding
):
raise ValueError(
"LiLiCorr requires the target input embedding and its selected "
"LM head on the draft rank."
)
if embedding.embedding_dim != model.config.hidden_size:
raise ValueError(
"LiLiCorr target embedding width must match its trained "
"token projection input."
)
if (
embedding.shard_indices.num_added_elements
or lm_head.shard_indices.num_added_elements
):
raise ValueError("LiLiCorr does not support added vocabulary entries.")
# Candidate embeddings always come from the target; the checkpoint may
# supply its own candidate LM head.
self.target_embeddings = embedding
model.lm_head = lm_head
return model
def prepare_context_anchor(
self, input_batch: InputBatch, num_rejected: torch.Tensor
) -> None:
num_reqs = input_batch.num_reqs
starts = input_batch.query_start_loc[:num_reqs]
ends = input_batch.query_start_loc[1 : num_reqs + 1] - num_rejected[:num_reqs]
valid = ends > starts
indices = (ends - 1).clamp_min(0).long()
self.anchor_hidden.zero_()
self.anchor_valid.zero_()
self.anchor_hidden[:num_reqs].copy_(
self.model.model.hidden_norm(self.hidden_states[indices]) * valid[:, None]
)
self.anchor_valid[:num_reqs].copy_(valid)
def _score_candidates(
self,
candidate_ids: torch.Tensor,
unary_logits: torch.Tensor,
hidden_states: torch.Tensor,
) -> torch.Tensor:
num_reqs = candidate_ids.shape[0]
return self.model.model.lilicorr(
self.target_embeddings(candidate_ids),
unary_logits,
hidden_states,
self.anchor_hidden[:num_reqs],
self.anchor_valid[:num_reqs],
)