Commit Graph
23242 Commits
Author SHA1 Message Date
Shanshan Shen 7ba3df63cb [ROCm][Refactor] Move DeepSeek-V4/V4.1 multi-stream overlap gate to ROCm platform (#58983)
Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com>
2026-09-28 05:46:28 -07:00
Jiangyun ZhuandOpenAI Codex 20b52e9f5b [Perf][Qwen3.8] Reduce PLE metadata construction overhead (#58114)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-28 20:46:22 +08:00
BaoYunkai f6877f563c [Bugfix][Spec Decode] Implement get_top_tokens() on the ROCm DeepSeek V4 MTP drafter (#57568)
Signed-off-by: BaoYunkai <ybao@amd.com>
2026-09-28 20:16:38 +08:00
Yannick Schnider 8cc9aa5ad3 [Bugfix][Model] Gemma4: register aliased embedding scalars as buffers (#54213)
Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>
2026-09-28 10:56:31 +00:00
Thien Tran d6cce94fd4 [Perf][Qwen4Exp] Fuse HC down projection and SiLU on NVIDIA (#58957)
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-09-28 18:51:38 +08:00
Kevin H. LuuandClaude Opus 5.5 252279314b Revert "[CI] Shard (H200 MIG 35GB / MI355 DPX) Entrypoints Integration (Pooling) into named jobs (#58652)" (#59011)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 03:51:17 -07:00
Louie Tsai ccfd1cea75 [CPU] Include vLLM Recipes tooling in release image to deploy models using vLLM Recipes (#58796)
Signed-off-by: louie-tsai <louie.tsai@intel.com>
2026-09-28 16:46:53 +08:00
Gabriel WuClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
a2be4d3cc7 [Bugfix][Qwen4Exp] Release the profiling KV cache held by QSA key views (#58961)
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-28 08:11:46 +00:00
Thang NguyenandClaude Opus 5.5 66d2d4b2e7 [CI] Split Dynamic Shapes out of (H200 MIG 35GB) PyTorch Compilation + (MI300) mirror (#58451)
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 00:32:11 -07:00
413ae4b167 [CI] Shard (H200 MIG 35GB / MI355 DPX) Entrypoints Integration (Pooling) into named jobs (#58652)
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: khluu <khluu000@gmail.com>
2026-09-28 00:21:08 -07:00
Eugenio "Jay" ZuccarelliClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
004e37ece2 [Misc] Name each backend and its kernel block sizes in block-size errors (#58557)
Signed-off-by: Eugenio "Jay" Zuccarelli <11176606+jayzuccarelli@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-28 14:33:18 +08:00
Thien TranandKimi 4694145045 [Skills] Update kernel-microbenchmark to include ROCm (#58646)
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Kimi <noreply@moonshot.cn>
2026-09-28 14:18:38 +08:00
Rehan KhanandLi, Jiang <jiang1.li@intel.com> fd2f3fcfa5 [CPU] Add video inferencing via torchcodec on s390x (#58693)
Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
2026-09-28 14:16:59 +08:00
zhao, zhenhuiandLi, Jiang <jiang1.li@intel.com> 1668e5976d [CPU] Use accelerator memory API in DiffusionGemma (#58964)
Signed-off-by: zhejiangxiaomai <zhenhui.zhao@intel.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
2026-09-28 13:31:42 +08:00
Turner JabbourandClaude Sonnet 5 2407f405b5 [CI] Allowlist-shrink batch 1: wire 17 root-level tests + drop 5 stale watermarking entries into misc.yaml (#58055)
Signed-off-by: Turner Jabbour <doubleujabbour@gmail.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-27 22:23:46 -07:00
MOVZX 76af434852 Merge remote-tracking branch 'upstream/main' into patches 2026-09-28 12:06:52 +07:00
af7f9488c2 [Benchmark] Add Responses API backend to vllm bench serve (#54628)
Signed-off-by: QHarshil <harshil_c@hotmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-09-28 05:04:11 +00:00
Kevin H. LuuandClaude Opus 5.5 0da126676e [Bugfix] Fix the two multimodal root tests that fail on main (OpenPangu-VL embed merge, MiMo sink test fixture) (#58900)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 04:52:51 +00:00
dd3bc9c72f [Perf][MRV2] Reuse Mamba/GDN metadata across KV cache groups (#58762)
Signed-off-by: Luca Motz <luca.motz@icloud.com>
Co-authored-by: Xin Yang <xyangx@amazon.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
2026-09-28 04:36:58 +00:00
Thien TranandCodex 55de40a2fc [Minimax-M3] Add Encoder CUDA graph support (#58673)
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Codex <codex@openai.com>
2026-09-28 04:17:36 +00:00
Flora Feng 6f65fde419 [Bugfix][Frontend] Apply Harmony adjust_request in batched chat completions (#58958)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-09-28 04:12:04 +00:00
jz-yoloandjz-yolo 4a190dc518 [Model] Enable LoRA support for RobertaForSequenceClassification (#58884)
Signed-off-by: jz-yolo <jz-yolo@users.noreply.github.com>
Signed-off-by: Jane Zhu <jane.zhu@slack-corp.com>
Co-authored-by: jz-yolo <jz-yolo@users.noreply.github.com>
2026-09-28 03:50:12 +00:00
Andreas Karatzas 77fbd9e225 [Test][ROCm] Stabilize the mixed OLMoE LoRA test (#58945)
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
2026-09-28 11:47:41 +08:00
zhao, zhenhuiandjiang1.li 5eaa50f151 [CPU] Build CPU wheels on Ubuntu 22.04 with AMX-FP8 support (#58515)
Signed-off-by: zhejiangxiaomai <zhenhui.zhao@intel.com>
Signed-off-by: jiang1.li <jiang1.li@intel.com>
Co-authored-by: jiang1.li <jiang1.li@intel.com>
2026-09-28 11:46:02 +08:00
Tianyu Guo 53d2e16a97 [Bugfix][EPD] Skip sampling for encoder-only async steps (#58490)
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
2026-09-27 20:35:09 -07:00
Flora Feng 027b6f3a22 [Bugfix][Frontend] Use a fresh parser per choice in non-streaming chat completions (#58939)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-09-28 02:01:35 +00:00
Flora Feng 4d07e90653 [Bugfix][Frontend] Sample batched chat completions from the adjusted requests (#58929)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-09-28 01:46:55 +00:00
Ashraf Bhuiyan b9e2903b89 [Mypy] Fix mypy typing for Ultravox and Unlimited-OCR models (#58239)
Signed-off-by: Ashraf Bhuiyan <mbhuiyan@redhat.com>
2026-09-27 21:19:04 -04:00
Thien Tran 39d49539de [KimiViT][Perf] Fuse per-layer QK RoPE into one in-place kernel (#58651) 2026-09-27 18:18:06 -07:00
Flora Feng 3184226984 [Bugfix][Frontend] Count Responses reasoning tokens per tool round (#58927)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-09-28 00:34:38 +00:00
31afa4f1bb [ROCm] Bump AITER to v0.1.23 (#58867)
Signed-off-by: fai <fangzhouai@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-27 19:04:58 -05:00
Anton Peganovmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>OpenAI Codex
73859fec58 [Frontend] Switch Python Harmony dependency to oss-harmony (#55128)
Signed-off-by: Anton Peganov <apeganov@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-28 00:43:56 +04:00
Chao Lei 386ac2573e [Bugfix][KV Connector] Retry Mooncake bootstrap registration on timeout (reopens #55763) (#58919) 2026-09-27 19:47:36 +00:00
Fangzhou Ai 187c81b32d [ROCm][Bugfix] Fall back to default GEMM for CPU tensors on ROCm builds (#58923)
Signed-off-by: fai <fangzhouai@gmail.com>
2026-09-27 14:47:07 -05:00
Jiangyun Zhu 231fdb83cc [Perf][MoE] Use fused MiniMax2 routing with non-unit routed scaling (#58880)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
2026-09-27 19:37:33 +00:00
Xun SunandIsotr0py e7900156e1 [Fast Start] Add /health endpoint for the weight cache daemon (#58552)
Signed-off-by: Xun Sun <UNIDY2002@outlook.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
2026-09-27 17:48:54 +00:00
2b9b55c7f1 [Bugfix] Support repsonse_format + tool_choice=auto (#56086)
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Co-authored-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Co-authored-by: pablopupo <145598901+pablopupo@users.noreply.github.com>
Co-authored-by: hubunt <150658615+hubunt@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-27 21:17:34 +04:00
Frank WangMisha Goinmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Vadim Gimpelson
0376f81530 [Perf][Pooling] Avoid blocking seq_lens GPU-to-CPU copy for pooling in FlashInfer metadata builder (#57214)
Signed-off-by: frankwang28 <frank.wbb@hotmail.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com>
2026-09-27 16:06:38 +00:00
Shanshan Shen 44af287ebe [ROCm][Perf] Enable layer-aware CSA2 multi-stream overlap for DeepSeek-V4.1-Flash (#57407)
Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com>
2026-09-27 23:48:01 +08:00
924707f1bf [CI] Split (H200 MIG 35GB) Spec Decode Speculators + MTP into 4 named jobs (#57237)
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Thang Nguyen <thang.nguyen@inferact.ai>
Co-authored-by: Kimi Code <noreply@moonshot.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 05:01:31 -07:00
Nicolò Lucchesi c8d7a7dd13 [Perf][Attention] Remove D2H sync from FlashInfer SM90 sparse MLA plan under async scheduling (#58684)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-09-27 13:06:23 +02:00
lesj0610 521235d6df Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-09-27 18:25:21 +09:00
5567cc1b25 [Mamba] RecoverSSM align: keep a boundary-ending state in its block
A speculative window whose accepted tokens end exactly on a recurrent block
boundary selected the following block-table column for the committed state
(commit plan) and recorded it as the running column (postprocess). That
column can still be unallocated, so the next step reads an uninitialized
page. Select the block that holds the last committed token, matching the
(n - 1) // block convention used elsewhere in align mode. Applies to
Kimi-K3 KDA and to the Qwen GDN/PLE RecoverSSM paths that share the plan.

Fix and test from local-inference-lab/vllm a02f585d (Martin Vit, root cause
reported by astra); the GDN/PLE align tests now expect the same convention.

Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com>
Co-authored-by: Martin Vit <martin@voipmonitor.org>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 11:12:40 +02:00
Rakul-Chauhanmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Cursor
24c9772d19 [Attention][CPU] Use zentorch SDPA for CPU MLA prefill (#54967)
Signed-off-by: Rakul Chauhan <rakul.chauhan@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-27 08:30:55 +00:00
Jürgen Schmied ed2a9665ea Merge branch 'main' into gdn-recoverssm 2026-09-27 08:01:32 +02:00
Juntian LiuandClaude Opus 5.5 a9eafde59c [Perf][DSv4.1] Fuse small-batch WO-A with inverse RoPE and MXFP8 quant on SM100/SM103 (#58634)
Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 21:16:12 -07:00
MOVZX 5da84df6b7 [BugFix] Honor drop_eagle_block in MambaManager
On a hybrid model (full-attention + Mamba/GDN layers, e.g. Qwen3.6-35B-A3B),
enabling MTP/EAGLE speculative decoding together with prefix caching corrupts
the recurrent state, and the server serves wrong answers. #43559 reports
tool-eval accuracy dropping from ~90% to ~50%. Nothing raises an exception
and nothing shows up in the logs, so an operator has no signal that anything
is wrong, and prefix caching then spreads it, because the corrupted state is
hash-reachable and every later request sharing that prefix restores from it.

The root cause is the find_longest_cache_hit method on MambaManager,
at vllm/v1/core/single_type_kv_cache_manager.py:1049 on main, which takes
drop_eagle_block in its signature and never acts on it. Its body searches
right-to-left for the last cached block, builds [null]*i + [cached], and
breaks, referencing neither drop_eagle_block nor .pop(.

Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 10:45:10 +07:00
VarshithandClaude fba47397f3 [Bugfix] V1: clear stale allowed_token_ids mask in InputBatch.condense (#43931)
Signed-off-by: Varshith <kvarshithgowda@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 03:44:54 +00:00
MOVZX 5e6656e7a0 [Bugfix][Spec Decode] Add get_top_tokens() to Qwen4ExpMTP draft model
use_local_argmax_reduction hard-fails at worker startup when the draft
model lacks get_top_tokens(): the V2 speculator probes for the method
in _validate_local_argmax_reduction and raises, with no fallback to
the full-vocab logits path. Qwen4ExpMTP (nvidia and amd variants) has
compute_logits but no get_top_tokens, so any Qwen4Exp MTP drafter with
the flag enabled crashed both TP ranks during load_model.

Qwen4ExpMTP owns a flat self.lm_head and its compute_logits is exactly
self.logits_processor(self.lm_head, hidden_states), the same shape as
Qwen3_5MTP, so LocalArgmaxMixin applies unchanged: vocab-parallel
argmax with a (value, index) gather, O(2*tp_size) vs O(vocab_size)
communication per draft token. No D2T remapping needed since the
draft vocabulary equals the target vocabulary.

Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 10:31:47 +07:00
YannikHintereggerandxijiade.aihemaiti fff03267da [Bugfix][NIXL] Release a dead peer's NIXL state without waiting for TTL (#50047)
Signed-off-by: Yannik Hinteregger <37209495+YannikHinteregger@users.noreply.github.com>
Co-authored-by: xijiade.aihemaiti <3146335281@qq.com>
2026-09-26 20:30:09 -07:00