Commit Graph
22030 Commits
Author SHA1 Message Date
5567cc1b25 [Mamba] RecoverSSM align: keep a boundary-ending state in its block
A speculative window whose accepted tokens end exactly on a recurrent block
boundary selected the following block-table column for the committed state
(commit plan) and recorded it as the running column (postprocess). That
column can still be unallocated, so the next step reads an uninitialized
page. Select the block that holds the last committed token, matching the
(n - 1) // block convention used elsewhere in align mode. Applies to
Kimi-K3 KDA and to the Qwen GDN/PLE RecoverSSM paths that share the plan.

Fix and test from local-inference-lab/vllm a02f585d (Martin Vit, root cause
reported by astra); the GDN/PLE align tests now expect the same convention.

Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com>
Co-authored-by: Martin Vit <martin@voipmonitor.org>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 11:12:40 +02:00
Jürgen Schmied ed2a9665ea Merge branch 'main' into gdn-recoverssm 2026-09-27 08:01:32 +02:00
Juntian LiuandClaude Opus 5.5 a9eafde59c [Perf][DSv4.1] Fuse small-batch WO-A with inverse RoPE and MXFP8 quant on SM100/SM103 (#58634)
Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-26 21:16:12 -07:00
VarshithandClaude fba47397f3 [Bugfix] V1: clear stale allowed_token_ids mask in InputBatch.condense (#43931)
Signed-off-by: Varshith <kvarshithgowda@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 03:44:54 +00:00
YannikHintereggerandxijiade.aihemaiti fff03267da [Bugfix][NIXL] Release a dead peer's NIXL state without waiting for TTL (#50047)
Signed-off-by: Yannik Hinteregger <37209495+YannikHinteregger@users.noreply.github.com>
Co-authored-by: xijiade.aihemaiti <3146335281@qq.com>
2026-09-26 20:30:09 -07:00
Hongxia Yang 3137ff0773 [ROCm][Kimi-K3] Make VLLM_ROCM_USE_AITER_MOE_SITUV2 select a4w4/a8w4/a16w4 (#58201)
Signed-off-by: Hongxia Yang <hongxia.yang@amd.com>
2026-09-27 02:26:49 +00:00
Lio Einaudiandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 0c87a197b8 [Bugfix] Disable sequence parallelism / async TP under batch invariance and add a TP regression test (#56377)
Signed-off-by: LioEinaudi <zhao3024667639@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-26 22:19:08 -04:00
Thien Tran 7a877ae34d [Qwen3.8-Flash-Next] Avoid memory fragmentation in QSA indexer logits workspace (#57105)
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-09-27 10:11:52 +08:00
eb0f2ca37f [mooncake] support CUSTOM_MEM_POOL in vllm (#49300)
Signed-off-by: bruce.xu <bruce.xb@alibaba-inc.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: bruce.xu <bruce.xb@alibaba-inc.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-27 00:26:05 +00:00
mevince cd1daf58a9 [Metrics][KV Offload] Add Prometheus metrics for SimpleCPUOffloadConnector (#57251)
Signed-off-by: Vincent <vincexxchan@gmail.com>
2026-09-26 17:17:11 -07:00
Jürgen SchmiedandClaude 03cae7837f [Spec Decode][Qwen4Exp] RecoverSSM for Qwen GDN and the PLE short conv
With --use-replayssm and speculative decoding, Qwen4Exp's Gated DeltaNet layers verify the
draft window from a read-only checkpoint and keep a per-token replay record
[correction(V), normalized key(K), gate(1)] in fp32. After sampling, one commit replays the
accepted tokens and writes the recurrent state once (plus the block-boundary state in align
mode), instead of snapshotting the state after every draft position. The PLE dilated short
conv follows the same protocol (accepted-suffix compaction). num_speculative_blocks = 0,
so no per-draft Mamba blocks are reserved.

- GDN verify/commit kernels: model_executor/layers/mamba/gdn/recoverssm_gdn.py
- metadata/backends: v1/attention/backends/gdn_recoverssm.py, ple_recoverssm.py
- the verify reads strided q/k/v views of the conv output and writes in place
- builders declare AttentionCGSupport.NEVER: FULL_AND_PIECEWISE falls back to PIECEWISE;
  the fallback now also counts breakable CUDA graphs (empty splitting_ops, with or
  without torch.compile) as piecewise
  attention instead of dropping to NONE
- the generic Mamba metadata builder skips the Mamba2 ReplaySSM ring-buffer workspaces
  when --use-replayssm selects RecoverSSM (the PLE short conv uses that builder)
- tests: GDN verify/commit vs the native kernels (fp32 and bf16 state, align boundaries,
  mixed and ragged batches, extreme gates), PLE vs token-by-token decode, config selection

Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 01:17:24 +02:00
4be061c5af [Kernel] Bump FlashKDA to keep the recurrent state in fp32 (#58846)
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
2026-09-26 22:20:44 +00:00
927c87b347 [PCP][DCP] Support DCP target model with non-DCP Dspark (#56723)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-26 18:12:35 -04:00
Jürgen SchmiedandClaude a80e41d690 [Mamba] Share RecoverSSM commit kernels and helpers across models
RecoverSSM's commit-plan and conv-window compaction kernels are model independent.
Move them from models/kimi_k3 into model_executor/layers/mamba/ops/recoverssm.py
(Kimi-K3 keeps its private names as import aliases; kernel source unchanged), add
mamba/recoverssm_utils.py (activation predicate, argument validation) and the
generic spec-row/request mapping helpers to v1/attention/backends/recoverssm_metadata.py.

CacheConfig.use_kda_recoverssm becomes use_recoverssm: the switch now selects RecoverSSM
for any model whose recurrent layers implement it, not only KDA.

No functional change for Kimi-K3.

Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 00:07:04 +02:00
Yongye ZhuandClaude Opus 5.5 77871126f9 [Kernel][DSV4.1] Fuse MoE finalize into the TP all-reduce + mHC boundary (#58586)
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 03:08:36 +08:00
Cyrus Leung 379e9a1ea8 [Security] Harden message sanitization (#58832)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-09-26 16:28:32 +00:00
Kevin H. Luu 7d8c5fe9a9 [Bugfix][DSV4.1] Avoid host sync in ViT CUDA graph replay metadata (#58499) 2026-09-26 11:37:36 -04:00
Andreas Karatzas 0239bd2549 [CI] Stabilize batch submission in full CUDA graph tests (#58810) 2026-09-26 11:35:56 -04:00
Wentao Ye 3576691c42 [GLM5.3 Bug] Fix sparse indexer attn topk backend selection (#58594)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-09-26 11:35:28 -04:00
Misha Goin 79ac28c5bd [CI] Reduce CUDA graph mode test overhead (#58749) 2026-09-26 11:22:33 -04:00
Ashraf Bhuiyan 5113cf9d16 [Mypy] Fix mypy typing for Qwen and Qianfan models (#58046)
Signed-off-by: Ashraf Bhuiyan <mbhuiyan@redhat.com>
2026-09-26 14:31:51 +00:00
Robert ShawandRobert Shaw dfab504333 [Bugfix][Frontend] Detect Anthropic inline-system merge against the resolved chat template (#58754)
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
2026-09-26 14:00:56 +00:00
Robert ShawandRobert Shaw d64f277c54 [Bugfix] Fix Anthropic Thinking Disabled with P/D (#58786)
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
2026-09-26 13:50:57 +00:00
Willian Zandmgoin ad6817b68d [Perf][MoE] Index expert mapping lookups in RoutedExperts.load_weights (#58720)
Signed-off-by: Willian <willian@willian.email>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
2026-09-26 06:48:24 -07:00
Wentao Ye bf93e1f92c [Refactor] Remove dead tests utils (#58803)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-09-26 09:16:53 -04:00
Itay AlroyandTyler Michael Smith 2bb7605ae1 [Elastic EP] Fix EPLB load statistics during scaling (#58473)
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
2026-09-26 12:05:37 +00:00
Juan Pérez de AlgabaandCursor 31f2e70cd3 [Security] Gate per-request multimodal processor kwargs (#58830)
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-26 11:10:35 +00:00
2f41e00235 [CI] Split (B200) Miscellaneous Kernels into mHC, FLA Ops and Misc named jobs (#58609)
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-09-26 04:10:14 -07:00
a4eb3f25d6 [Spec Decode] Enable Gemma4 DSpark adaptive verification with FlashInfer (#57263)
Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-26 17:01:17 +08:00
Aarushi JainandAndreas Karatzas 3b4566c5cf [CI] Only isolate the registry tests that need a fresh process (#58764)
Signed-off-by: aarushjain29 <Aarushi.Jain2@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-26 14:03:45 +08:00
ddd6fbca14 [Bugfix][Frontend][Rust Frontend] Update DeepSeek V4.1 Flash reasoning effort mappings (#58316)
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: zhec <chengyunfei@ruc.edu.cn>
2026-09-26 02:18:58 +00:00
xiao-llmandCursor 5840d95284 [Bugfix][ROCm] AMD-Quark mixed-precision DeepSeek-V4.1 support (#57071)
Signed-off-by: Xiao Yu <xiao.yu.dc@outlook.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-25 20:51:18 -05:00
Ke Wenmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>aoshen02
b6d2b4fc25 [RL] Add sharding-aware NCCL M2N weight transfer (#51520)
Signed-off-by: Ke Wen <kwen@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-09-26 01:38:53 +00:00
Shuolei Wang 4bb804cc5b [Perf][DSv4.1] Shard the Engram wkv projection across TP ranks (#58678)
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com>
2026-09-26 01:15:57 +00:00
Chandra MouryaandVS Chandra Mourya 8a2364605c [Core] Bound draft-token RPC waits by the execute-model timeout (#58779)
Signed-off-by: VS Chandra Mourya <219748331+vschandramourya@users.noreply.github.com>
Co-authored-by: VS Chandra Mourya <219748331+vschandramourya@users.noreply.github.com>
2026-09-26 00:40:22 +00:00
Baljinder HothiandFlora Feng fdcce47e9b [Bugfix][Frontend] Fix Inkling tool name leaking into content after reasoning (#58792)
Signed-off-by: Baljinder Hothi <baljinder.hothi@cohere.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-09-26 00:38:52 +00:00
da1aaec31f [CI] Split (H200 MIG/MI300) Basic Correctness into named jobs (#57054)
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-25 17:05:33 -07:00
Elvir CrnčevićandFlora Feng 9bf44c4013 [Bugfix][Reasoning] Count Kimi K3 reasoning tokens (#58372)
Signed-off-by: Elvir Crncevic <elvircrn@gmail.com>
Signed-off-by: Flora Feng <4florafeng@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-09-25 23:29:56 +00:00
Rui ZhuandClaude 974cb65155 [Bugfix] V1: fix allowed_token_ids_mask aliasing in InputBatch.swap_states (#48419)
Signed-off-by: Rui Zhu <rui.zhu.rz399@yale.edu>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-25 23:23:11 +00:00
Ricardo-M-LandClaude Opus 4.6 b040321b33 fix: perf: use startswith(x, i) instead of string slicing to avoid O(N^2) (#52580)
Signed-off-by: Ricardo-M-L <ricardoporsche001@icloud.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-09-25 23:22:13 +00:00
Taneem IbrahimandWentao Ye 6ec2f84e1a [Pooling] Preserve BERT-family heads for raw logits (#57664)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-09-25 19:10:14 -04:00
Linkun 614b55f0b6 [AuxOutput] Only require Model Runner V2 on GPU platform (#58205)
Signed-off-by: Linkun Chen <github@lkchen.net>
2026-09-25 15:55:21 -07:00
Oxana Korzh e55d076f89 [Bugfix] Don't drop the rest of the allocator config when toggling expandable segments (#57982)
Signed-off-by: Oxana Korzh <okorzh@amd.com>
2026-09-25 17:34:19 -05:00
Raphaël RiallandSimon VeitnerOpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
4ccfe12398 [watermarking] golden tests for backwards compatibility (#56809)
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-25 22:01:48 +00:00
yzong-rhandBen Browning b6761e8ded [Bugfix] Default missing detail for Responses API input images (#57241)
Signed-off-by: Yifan Zong <yzong@redhat.com>
Co-authored-by: Ben Browning <56071+bbrowning@users.noreply.github.com>
2026-09-25 14:08:55 -07:00
8b47e8b22c [Perf][Frontend] Defer reasoning usage recounts for non-continuous chat streams (#56067)
Signed-off-by: Cheng Rui <286040359@qq.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-09-25 14:08:23 -07:00
Nick HillandKimi Code de2d478d9a [Bugfix][Frontend] Document 404 response for /generative_scoring (#58788)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
2026-09-25 14:08:00 -07:00
Denis ErmakovDu BinClaude Opus 5.5Flora Fengmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
7871963fcc [Bugfix][Frontend] Respect max_output_tokens in the Harmony tool-call loop (#58551)
Signed-off-by: errmakov <ide404@gmail.com>
Co-authored-by: Du Bin <8174807+dubin555@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-25 14:00:39 -07:00
Yousafandsfeng33 83db8c8390 [Feature][Frontend] Add granite_thinking_parser reasoning parser for Granite 4.2 (#55957)
Signed-off-by: Yousaf shah <yousaf.shah@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
2026-09-25 20:43:08 +00:00
e440b75bb6 [ROCm] Bump torch 2.13, triton 3.8, torchaudio, torchvision (#50605)
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Signed-off-by: jpvillam <juan.villamizar@amd.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: jpvillam <juan.villamizar@amd.com>
2026-09-25 14:55:52 -05:00