Commit Graph
22151 Commits
Author SHA1 Message Date
MOVZXandHinata ae0350c0de [Spec Decode][MRV2] RecoverSSM for Qwen GDN and PLE short conv (PR #58863)
Intake of open upstream PR vllm-project/vllm#58863. Active only with
VLLM_USE_V2_MODEL_RUNNER=1; inert otherwise.

Speculative decoding on Gated DeltaNet snapshotted the full recurrent
state after every draft position and reserved one Mamba block per draft
position. On a 24 GB card that is reserved VRAM we never use, plus
per-step copy traffic.

RecoverSSM, the protocol Kimi-K3 KDA already uses, replaces this:
verify runs from a read-only checkpoint and stores a small per-token
replay record [correction (V), normalized key (K), gate (1)] in fp32;
commit replays only the accepted tokens and writes the recurrent state
once, plus the block-boundary state in align mode. The PLE dilated
short conv compacts its accepted window with the same plan.

Conflict in vllm/models/qwen4_exp/nvidia/mtp.py: our get_top_tokens()
work put LocalArgmaxMixin on Qwen4ExpMTP, the PR adds SupportsReplaySSM
to the same base list. Kept both.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-29 13:39:09 +07:00
MOVZXandHinata a1d3c2cf1e [Spec Decode] Fused draft metadata updates for the QSA side cache (PR #58449)
Intake of open upstream PR vllm-project/vllm#58449.

With MTP num_speculative_tokens > 1, the V2 autoregressive speculator
reuses one attention-metadata build across draft steps only when every
group's builder sets supports_draft_decode_metadata_update. The QSA
side-cache builder (QWEN4_EXP_EXP_QSA_STATE) did not, so every draft
step rebuilt attention metadata for the whole model.

QSAMetadataBuilder now implements update_draft_decode_metadata: it
re-launches the kernel with in-place seq_lens updates, so one metadata
build serves every draft step.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-29 13:36:09 +07:00
MOVZXandHinata cb275fdf20 [Qwen4Exp] Fix MTP buffer placement, config parsing, Triton warmup (PR #56742)
Intake of open upstream PR vllm-project/vllm#56742. Every rig here runs
MTP on both Qwen3.8 models, so these three bugs sit on our hot path.

The MTP hidden buffer was allocated with torch.empty and no device, so
it followed the ambient default instead of device_config.device; on a
CPU-default process the draft buffer lands on the wrong device. Both
NVIDIA and AMD backends now call make_mtp_hidden_buffer() in
vllm/models/qwen4_exp/common/mtp.py, which pins it to the configured
device.

Config parsing now accepts configs carrying M-RoPE keys and the
Transformers spelling of layer_types; both previously failed at load.

Triton warmup: Qwen4Exp was missing from the Qwen warmup dispatch, so
its Triton kernels compiled during the first served requests instead of
at startup.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-29 13:36:05 +07:00
MOVZX a63dca4afb Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-29 12:54:18 +07:00
Bugen ZhaoandClaude Opus 5.5 28f6739576 [Rust Frontend] Relax schema-aware tool argument conversion (#59004)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 05:40:41 +00:00
Ren YuzhouandClaude ec5e0c352f [Bugfix][Engram] Keep THP tables private when resolving shared memory (#59068)
Signed-off-by: Ren Yuzhou <54501155+yuzhouo7@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-29 05:19:49 +00:00
Shinichi Hemmi 17cc3cd80e [Model] Engine based plamo3 parser (#56575)
Signed-off-by: Shinichi Hemmi <shemmi@preferred.jp>
2026-09-29 05:17:42 +00:00
shaohuaxiandCodex 36768d1bfd [Bugfix][Frontend] Preserve caller parameters in offline pooling (#58842)
Signed-off-by: 子华 <huaxi.shx@alibaba-inc.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-29 05:02:59 +00:00
Bowen WangandClaude Opus 5.5 1fde8ff7da [Bugfix] Detect the CUDA toolkit the way FlashInfer does in has_flashinfer() (#58360)
Signed-off-by: Bowen Wang <bowen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 04:14:49 +00:00
debanshd 13c8589b86 fix: return loaded parameters in Aria load_weights (#57476)
Signed-off-by: Debanshu Das <debanshu@google.com>
2026-09-29 04:05:07 +00:00
Daniel M. García-Ocaña Hernández 295a0b8d64 [Bugfix][Multimodal] Fix flat/scoped mm_processor_kwargs merge and resolution (#56372)
Signed-off-by: Daniel M. García-Ocaña Hernández <danielgarciaocana@gmail.com>
2026-09-29 03:58:39 +00:00
cjackalandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 491f44adfa [Model] Extend device-side mm normalization to Qwen3VL/Qwen3.5/Qwen4Next (#51289)
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-29 11:44:54 +08:00
Isotr0pyandcjackal 07f1ba2ec6 [Multimodal] Remove processsor fallback for fused input norm params resolve (#59073)
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com>
2026-09-29 03:33:43 +00:00
Kevin H. LuuandClaude Opus 5.5 aedaba8664 [CI/Build] Skip the snapshot runtime on CUDA 12.x images (#59118)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 20:17:59 -07:00
4e59d53ab5 [MRV2] Support stock torch.compile mode (#59079)
Signed-off-by: cheesecake <farzanaman99@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-09-28 20:17:48 -07:00
Tyler Michael SmithandCodex 4406874e7c [Bugfix][FT] Pass stateless process-group timeouts explicitly (#58611)
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-29 03:14:02 +00:00
Fangzhou AiandCursor cf971b419a [Bugfix] Revert "[ROCm][Perf] Replace torch.topk in DSA candidate block selection" (#59125)
Signed-off-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-28 20:11:48 -07:00
c6843d397c [Bugfix][Frontend] Validate reused prompt token ids; render messages for media and echo (#55771)
Signed-off-by: Shijie Lyu <sjlyu@amazon.com>
Co-authored-by: Shijie Lyu <sjlyu@amazon.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-29 10:36:22 +08:00
Muhammad Ahmed 0af34418e9 [ROCm][DSv4.1][Perf] Run the delayed mHC seams through aiter's fused Triton kernel (#58655)
Signed-off-by: Ahmed <Muhammad.Ahmed@amd.com>
2026-09-29 10:19:10 +08:00
Chaojun Zhang 7230dfea50 [XPU] Preserve non-contiguous strides when pinning CPU tensors for UVA view (#54874)
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
2026-09-29 09:43:07 +08:00
Mehmet Cagri 70ae0b7435 [ROCm][MLA] Enable sparse MLA Gluon kernel from Aiter (#53492)
Signed-off-by: Mehmet Cagri Kaymak <mehmet.kaymak@amd.com>
Signed-off-by: Mehmet Cagri <mehmet.kaymak@amd.com>
2026-09-29 09:08:18 +08:00
Andrey Talman 869278cbb0 [CI] Bound the CRCR report by build age, and stop gating it on a job (#59066) 2026-09-28 17:42:32 -07:00
Laith Sakka b39f760cc7 [torch.compile] Canonicalize functionalized split slices for fusion pa… (#57299)
Signed-off-by: Laith Sakka <lsakka@meta.com>
2026-09-28 20:21:59 -04:00
Almog Tavor 171d1b9ef6 [Elastic EP] Support Model Runner V2 (#53934)
Signed-off-by: almogtavor <almogtavor@gmail.com>
Signed-off-by: Almog Tavor <70065337+almogtavor@users.noreply.github.com>
2026-09-29 00:20:40 +00:00
Benedikt Falk c68eb98b69 [Bugfix][Frontend] Keep in-flight requests on the same DP engine (#59017)
Signed-off-by: Benedikt Falk <bfalk@nvidia.com>
2026-09-28 23:52:29 +00:00
Mehmet CagriandClaude Opus 5.5 75fad5bbef [Bugfix][ROCm] Drop -1 sentinels when building the ragged sparse-MLA indices (#58058)
Signed-off-by: Mehmet Cagri Kaymak <mehmet.kaymak@amd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 18:40:11 -05:00
9b29370524 [Bugfix] Fix moe_wna16 w13 zero-point shard split for 8-bit asym GPTQ MoE (#58950)
Signed-off-by: Spandan Tiwari <sptiwari@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-28 23:38:28 +00:00
rasmithmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Andreas Karatzas
8a497d7557 [CI/Build][BugFix][The Rock] Make supports_mm_prefix return False for ROCm attn and unified attn since Prefix-LM not implemented (#52395)
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-28 23:30:38 +00:00
e03875f4d2 [Minimax M3] Enable fp8 indexer cache on triton indexer for non-SM100 architectures. (#59081)
Signed-off-by: Jarrel Seah <jarrel@harrison.ai>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Signed-off-by: Rajas Mhaskar <rmhaskar@nvidia.com>
Co-authored-by: Jarrel Seah <jarrel@harrison.ai>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-28 19:21:13 -04:00
Fanli LinandKunshang Ji 6a6c2952fa [XPU][CI] Make test_mamba_prefix_cache block-size agnostic (#58269)
Signed-off-by: Lin, Fanli <fanli.lin@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-09-29 07:12:43 +08:00
Yan Ma 3bd7302396 [XPU] skip test_online_quantization_loads_real_weights (#58987)
Signed-off-by: Yan Ma <yan.ma@intel.com>
2026-09-29 07:10:52 +08:00
Yan MaandKunshang Ji 754aa29c00 [XPU] use rms_norm xpu kernel for context-key normalization (#58896)
Signed-off-by: Yan Ma <yan.ma@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-09-29 07:09:58 +08:00
Abhinandan 32cc3f1ea8 [CI] Drop test-group rules that no longer match the tree (#58895)
Signed-off-by: Abhinandan <abhinandanverma551@gmail.com>
2026-09-28 21:46:21 +00:00
Giancarlo DelfinandWoosuk Kwon 3d5f4d4cd5 [Perf][Spec Decode] Avoid triton recompiles in the acceptance estimator (#57107)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
2026-09-28 13:46:25 -07:00
0xsenseiandOpenAI Codex d28795f1a7 [Bugfix][Scheduler] Refresh max tokens for streaming continuations (#57676)
Signed-off-by: 0xsensei <prblmslvr.aditya@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-28 20:31:57 +00:00
beef814e1d [ROCm][CI] Add missing test coverage for upstream parity (#50519)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-09-28 13:12:07 -07:00
fedbc3b564 [Bugfix][MRV2][Spec Decode] Reject draft slots that were never proposed (#58784)
Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-28 13:07:15 -07:00
Nick HillandRobert Shaw 42dbe4578c [Core] Rework scheduler skipped_waiting queue (#58947)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
2026-09-28 20:03:16 +00:00
Ashraf BhuiyanandWentao Ye 2619b6b72b [Mypy] Fix mypy typing for Voxtral and vision models (#58251)
Signed-off-by: Ashraf Bhuiyan <mbhuiyan@redhat.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-09-28 15:52:50 -04:00
0xsensei 2840ca7a33 [Bugfix][Scheduler] Preserve logprobs across streaming continuations (#57447)
Signed-off-by: 0xsensei <prblmslvr.aditya@gmail.com>
2026-09-28 19:50:27 +00:00
Nick HillNicolò LucchesiClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Yongye Zhu
e7c903609f [Perf][MRV2] Allow FULL decode graphs for one-token prompt tails (#58400)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
2026-09-28 12:48:52 -07:00
Xiaoshuang Wangandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> ef42093a36 [Model Runner V2][Spec Decode] Support spec decode with draft model (#43091)
Signed-off-by: Icey <1790571317@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-28 19:38:05 +00:00
yzong-rh 2f4a975901 [Bugfix] Fix resumable request + async scheduling handoff race (#58259)
Signed-off-by: Yifan Zong <yzong@redhat.com>
2026-09-28 15:26:57 -04:00
764413559a [Bugfix] Don't sync-police or retry FlashInfer all-reduce workspace creation in eager mode (#58498)
Signed-off-by: khluu <khluu000@gmail.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-28 12:14:21 -07:00
25624f65cf [ROCm][MLA] Add an AITER ASM round-robin decode route for DCP multi-token verify (#56861)
Signed-off-by: Xiaohu Guo <Xiaohu.Guo@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: seungrokj <144636725+seungrokj@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-28 14:52:21 -04:00
djramicandAndreas Karatzas 5ba2e3090b [ROCm][CI] Increase timeout for Entrypoints Unit (#59076)
Signed-off-by: Djordje Ramic <djoramic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-28 17:56:10 +00:00
Rohan PotdarandClaude Opus 5.5 7a160c6fe5 [ROCm][CI] Pin OpenTelemetry to LMCache's cap in the ROCm images (#59051)
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 12:33:09 -05:00
stefankoncarevicandAndreas Karatzas 6c300ddb9e [CI][Bugfix] Relax packed_qk_rope_ correctness test to one ULP (#59008)
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-28 17:29:59 +00:00
HDCharlesClaude Opus 4.6HDCharles <{"message":"Not Found","documentation_url":"https://docs.github.com/rest/users/emails#list-email-addresses-for-the-authenticated-user","status":"404"}>OpenAI CodexOpenAI Codex
c01368fd64 [Quant] Use canonical N-first weight format for CT WNA16 MoE (#52798)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: HDCharles <{"message":"Not Found","documentation_url":"https://docs.github.com/rest/users/emails#list-email-addresses-for-the-authenticated-user","status":"404"}>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-09-28 11:22:44 -06:00
qizixi 6d3ea3c2c9 [KVConnector][NIXL] Support packed MLA KV layouts in pipeline-parallel push prefill (#50499)
Signed-off-by: zixi-qi <zixi@inferact.ai>
2026-09-28 18:50:46 +02:00