A speculative window whose accepted tokens end exactly on a recurrent block
boundary selected the following block-table column for the committed state
(commit plan) and recorded it as the running column (postprocess). That
column can still be unallocated, so the next step reads an uninitialized
page. Select the block that holds the last committed token, matching the
(n - 1) // block convention used elsewhere in align mode. Applies to
Kimi-K3 KDA and to the Qwen GDN/PLE RecoverSSM paths that share the plan.
Fix and test from local-inference-lab/vllm a02f585d (Martin Vit, root cause
reported by astra); the GDN/PLE align tests now expect the same convention.
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com>
Co-authored-by: Martin Vit <martin@voipmonitor.org>
Co-authored-by: Claude <noreply@anthropic.com>
On a hybrid model (full-attention + Mamba/GDN layers, e.g. Qwen3.6-35B-A3B),
enabling MTP/EAGLE speculative decoding together with prefix caching corrupts
the recurrent state, and the server serves wrong answers. #43559 reports
tool-eval accuracy dropping from ~90% to ~50%. Nothing raises an exception
and nothing shows up in the logs, so an operator has no signal that anything
is wrong, and prefix caching then spreads it, because the corrupted state is
hash-reachable and every later request sharing that prefix restores from it.
The root cause is the find_longest_cache_hit method on MambaManager,
at vllm/v1/core/single_type_kv_cache_manager.py:1049 on main, which takes
drop_eagle_block in its signature and never acts on it. Its body searches
right-to-left for the last cached block, builds [null]*i + [cached], and
breaks, referencing neither drop_eagle_block nor .pop(.
Signed-off-by: MOVZX <movzx@yahoo.com>
use_local_argmax_reduction hard-fails at worker startup when the draft
model lacks get_top_tokens(): the V2 speculator probes for the method
in _validate_local_argmax_reduction and raises, with no fallback to
the full-vocab logits path. Qwen4ExpMTP (nvidia and amd variants) has
compute_logits but no get_top_tokens, so any Qwen4Exp MTP drafter with
the flag enabled crashed both TP ranks during load_model.
Qwen4ExpMTP owns a flat self.lm_head and its compute_logits is exactly
self.logits_processor(self.lm_head, hidden_states), the same shape as
Qwen3_5MTP, so LocalArgmaxMixin applies unchanged: vocab-parallel
argmax with a (value, index) gather, O(2*tp_size) vs O(vocab_size)
communication per draft token. No D2T remapping needed since the
draft vocabulary equals the target vocabulary.
Signed-off-by: MOVZX <movzx@yahoo.com>
Adds an opt-in backend, --engram-config '{"host_file_gather": true}', that
serves Qwen4Exp PLE rows from the checkpoint files through the page cache
instead of allocating the 47.7 GiB table on GPU or in pinned host memory.
- Qwen4ExpPLEFileGatherEmbedding allocates no table rows; load_weights keeps
the default loader's zero-copy safetensors views of each shard. At bind
(post-load hook) every view must be file-backed per /proc/self/maps (HF
cache symlinks work; heap/pinned/anonymous memory is rejected) and the
shards must cover the whole table.
- Each step, Qwen4ExpModelState.prepare_inputs computes the PLE n-gram ids,
copies them to pinned host memory, syncs the stream once, issues
POSIX_FADV_WILLNEED over the coalesced page ranges of the step's rows,
gathers the rows on the host, and copies them into a fixed device staging
buffer. The forward reads only the staging buffer, so FULL CUDA graph
capture is unaffected.
- Rows stay in the checkpoint dtype (FP8 table stays FP8, dequantized on the
GPU); dummy loads serve zeros; weight reload is rejected; takes precedence
over cpu_offload; rejected on ROCm and with elastic EP.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Fixes#58807. A compressed-tensors pack-quantized checkpoint that ships
its MTP head dense (plain mtp.*.weight names in a separate bf16 shard)
crashes at weight load: Qwen3_5MultiTokenPredictor builds its Linears
from the target quant config, so the loader expects packed names
(weight_packed) the checkpoint never had, and the weight-map walk dies
with "no module or parameter named fc.weight".
Detect the dense MTP head from the checkpoint itself: read the
safetensors index via the same repo-file channel used for recipe.yaml
(local cache first, Hub download as fallback). If any mtp.* tensor
carries a quantization artifact suffix the head is properly quantized
and nothing changes; if every mtp.* entry is a plain .weight the whole
predictor builds unquantized (fc and layers), mirroring the existing
modelopt_fp4 and GPTQ dynamic carve-outs. Single-file checkpoints have
no index and keep current behavior; NVFP4 checkpoints are unaffected
(their mtp.layers.* carry artifacts).
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>