5567cc1b25
[Mamba] RecoverSSM align: keep a boundary-ending state in its block
...
A speculative window whose accepted tokens end exactly on a recurrent block
boundary selected the following block-table column for the committed state
(commit plan) and recorded it as the running column (postprocess). That
column can still be unallocated, so the next step reads an uninitialized
page. Select the block that holds the last committed token, matching the
(n - 1) // block convention used elsewhere in align mode. Applies to
Kimi-K3 KDA and to the Qwen GDN/PLE RecoverSSM paths that share the plan.
Fix and test from local-inference-lab/vllm a02f585d (Martin Vit, root cause
reported by astra); the GDN/PLE align tests now expect the same convention.
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Martin Vit <martin@voipmonitor.org >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 11:12:40 +02:00
Jürgen Schmied
ed2a9665ea
Merge branch 'main' into gdn-recoverssm
2026-09-27 08:01:32 +02:00
Juntian Liu and Claude Opus 5.5
a9eafde59c
[Perf][DSv4.1] Fuse small-batch WO-A with inverse RoPE and MXFP8 quant on SM100/SM103 ( #58634 )
...
Signed-off-by: Juntian Liu <Juntianl777@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-09-26 21:16:12 -07:00
Varshith and Claude
fba47397f3
[Bugfix] V1: clear stale allowed_token_ids mask in InputBatch.condense ( #43931 )
...
Signed-off-by: Varshith <kvarshithgowda@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 03:44:54 +00:00
YannikHinteregger and xijiade.aihemaiti
fff03267da
[Bugfix][NIXL] Release a dead peer's NIXL state without waiting for TTL ( #50047 )
...
Signed-off-by: Yannik Hinteregger <37209495+YannikHinteregger@users.noreply.github.com >
Co-authored-by: xijiade.aihemaiti <3146335281@qq.com >
2026-09-26 20:30:09 -07:00
Hongxia Yang
3137ff0773
[ROCm][Kimi-K3] Make VLLM_ROCM_USE_AITER_MOE_SITUV2 select a4w4/a8w4/a16w4 ( #58201 )
...
Signed-off-by: Hongxia Yang <hongxia.yang@amd.com >
2026-09-27 02:26:49 +00:00
Lio Einaudi and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
0c87a197b8
[Bugfix] Disable sequence parallelism / async TP under batch invariance and add a TP regression test ( #56377 )
...
Signed-off-by: LioEinaudi <zhao3024667639@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-26 22:19:08 -04:00
Thien Tran
7a877ae34d
[Qwen3.8-Flash-Next] Avoid memory fragmentation in QSA indexer logits workspace ( #57105 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-09-27 10:11:52 +08:00
eb0f2ca37f
[mooncake] support CUSTOM_MEM_POOL in vllm ( #49300 )
...
Signed-off-by: bruce.xu <bruce.xb@alibaba-inc.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: bruce.xu <bruce.xb@alibaba-inc.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-27 00:26:05 +00:00
mevince
cd1daf58a9
[Metrics][KV Offload] Add Prometheus metrics for SimpleCPUOffloadConnector ( #57251 )
...
Signed-off-by: Vincent <vincexxchan@gmail.com >
2026-09-26 17:17:11 -07:00
Jürgen Schmied and Claude
03cae7837f
[Spec Decode][Qwen4Exp] RecoverSSM for Qwen GDN and the PLE short conv
...
With --use-replayssm and speculative decoding, Qwen4Exp's Gated DeltaNet layers verify the
draft window from a read-only checkpoint and keep a per-token replay record
[correction(V), normalized key(K), gate(1)] in fp32. After sampling, one commit replays the
accepted tokens and writes the recurrent state once (plus the block-boundary state in align
mode), instead of snapshotting the state after every draft position. The PLE dilated short
conv follows the same protocol (accepted-suffix compaction). num_speculative_blocks = 0,
so no per-draft Mamba blocks are reserved.
- GDN verify/commit kernels: model_executor/layers/mamba/gdn/recoverssm_gdn.py
- metadata/backends: v1/attention/backends/gdn_recoverssm.py, ple_recoverssm.py
- the verify reads strided q/k/v views of the conv output and writes in place
- builders declare AttentionCGSupport.NEVER: FULL_AND_PIECEWISE falls back to PIECEWISE;
the fallback now also counts breakable CUDA graphs (empty splitting_ops, with or
without torch.compile) as piecewise
attention instead of dropping to NONE
- the generic Mamba metadata builder skips the Mamba2 ReplaySSM ring-buffer workspaces
when --use-replayssm selects RecoverSSM (the PLE short conv uses that builder)
- tests: GDN verify/commit vs the native kernels (fp32 and bf16 state, align boundaries,
mixed and ragged batches, extreme gates), PLE vs token-by-token decode, config selection
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 01:17:24 +02:00
4be061c5af
[Kernel] Bump FlashKDA to keep the recurrent state in fp32 ( #58846 )
...
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-09-26 22:20:44 +00:00
927c87b347
[PCP][DCP] Support DCP target model with non-DCP Dspark ( #56723 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-26 18:12:35 -04:00
Jürgen Schmied and Claude
a80e41d690
[Mamba] Share RecoverSSM commit kernels and helpers across models
...
RecoverSSM's commit-plan and conv-window compaction kernels are model independent.
Move them from models/kimi_k3 into model_executor/layers/mamba/ops/recoverssm.py
(Kimi-K3 keeps its private names as import aliases; kernel source unchanged), add
mamba/recoverssm_utils.py (activation predicate, argument validation) and the
generic spec-row/request mapping helpers to v1/attention/backends/recoverssm_metadata.py.
CacheConfig.use_kda_recoverssm becomes use_recoverssm: the switch now selects RecoverSSM
for any model whose recurrent layers implement it, not only KDA.
No functional change for Kimi-K3.
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 00:07:04 +02:00
Yongye Zhu and Claude Opus 5.5
77871126f9
[Kernel][DSV4.1] Fuse MoE finalize into the TP all-reduce + mHC boundary ( #58586 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-27 03:08:36 +08:00
Cyrus Leung
379e9a1ea8
[Security] Harden message sanitization ( #58832 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-09-26 16:28:32 +00:00
Kevin H. Luu
7d8c5fe9a9
[Bugfix][DSV4.1] Avoid host sync in ViT CUDA graph replay metadata ( #58499 )
2026-09-26 11:37:36 -04:00
Andreas Karatzas
0239bd2549
[CI] Stabilize batch submission in full CUDA graph tests ( #58810 )
2026-09-26 11:35:56 -04:00
Wentao Ye
3576691c42
[GLM5.3 Bug] Fix sparse indexer attn topk backend selection ( #58594 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-26 11:35:28 -04:00
Misha Goin
79ac28c5bd
[CI] Reduce CUDA graph mode test overhead ( #58749 )
2026-09-26 11:22:33 -04:00
Ashraf Bhuiyan
5113cf9d16
[Mypy] Fix mypy typing for Qwen and Qianfan models ( #58046 )
...
Signed-off-by: Ashraf Bhuiyan <mbhuiyan@redhat.com >
2026-09-26 14:31:51 +00:00
Robert Shaw and Robert Shaw
dfab504333
[Bugfix][Frontend] Detect Anthropic inline-system merge against the resolved chat template ( #58754 )
...
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Robert Shaw <robshaw@redhat.com >
2026-09-26 14:00:56 +00:00
Robert Shaw and Robert Shaw
d64f277c54
[Bugfix] Fix Anthropic Thinking Disabled with P/D ( #58786 )
...
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Robert Shaw <robshaw@redhat.com >
2026-09-26 13:50:57 +00:00
Willian Z and mgoin
ad6817b68d
[Perf][MoE] Index expert mapping lookups in RoutedExperts.load_weights ( #58720 )
...
Signed-off-by: Willian <willian@willian.email >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
2026-09-26 06:48:24 -07:00
Wentao Ye
bf93e1f92c
[Refactor] Remove dead tests utils ( #58803 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-26 09:16:53 -04:00
Itay Alroy and Tyler Michael Smith
2bb7605ae1
[Elastic EP] Fix EPLB load statistics during scaling ( #58473 )
...
Signed-off-by: Itay Alroy <ialroy@nvidia.com >
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
2026-09-26 12:05:37 +00:00
Juan Pérez de Algaba and Cursor
31f2e70cd3
[Security] Gate per-request multimodal processor kwargs ( #58830 )
...
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-26 11:10:35 +00:00
2f41e00235
[CI] Split (B200) Miscellaneous Kernels into mHC, FLA Ops and Misc named jobs ( #58609 )
...
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-09-26 04:10:14 -07:00
a4eb3f25d6
[Spec Decode] Enable Gemma4 DSpark adaptive verification with FlashInfer ( #57263 )
...
Signed-off-by: zixi-qi <zixi@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-26 17:01:17 +08:00
Aarushi Jain and Andreas Karatzas
3b4566c5cf
[CI] Only isolate the registry tests that need a fresh process ( #58764 )
...
Signed-off-by: aarushjain29 <Aarushi.Jain2@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-26 14:03:45 +08:00
ddd6fbca14
[Bugfix][Frontend][Rust Frontend] Update DeepSeek V4.1 Flash reasoning effort mappings ( #58316 )
...
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: zhec <chengyunfei@ruc.edu.cn >
2026-09-26 02:18:58 +00:00
xiao-llm and Cursor
5840d95284
[Bugfix][ROCm] AMD-Quark mixed-precision DeepSeek-V4.1 support ( #57071 )
...
Signed-off-by: Xiao Yu <xiao.yu.dc@outlook.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-25 20:51:18 -05:00
b6d2b4fc25
[RL] Add sharding-aware NCCL M2N weight transfer ( #51520 )
...
Signed-off-by: Ke Wen <kwen@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-26 01:38:53 +00:00
Shuolei Wang
4bb804cc5b
[Perf][DSv4.1] Shard the Engram wkv projection across TP ranks ( #58678 )
...
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com >
2026-09-26 01:15:57 +00:00
Chandra Mourya and VS Chandra Mourya
8a2364605c
[Core] Bound draft-token RPC waits by the execute-model timeout ( #58779 )
...
Signed-off-by: VS Chandra Mourya <219748331+vschandramourya@users.noreply.github.com >
Co-authored-by: VS Chandra Mourya <219748331+vschandramourya@users.noreply.github.com >
2026-09-26 00:40:22 +00:00
Baljinder Hothi and Flora Feng
fdcce47e9b
[Bugfix][Frontend] Fix Inkling tool name leaking into content after reasoning ( #58792 )
...
Signed-off-by: Baljinder Hothi <baljinder.hothi@cohere.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-09-26 00:38:52 +00:00
da1aaec31f
[CI] Split (H200 MIG/MI300) Basic Correctness into named jobs ( #57054 )
...
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com >
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-25 17:05:33 -07:00
Elvir Crnčević and Flora Feng
9bf44c4013
[Bugfix][Reasoning] Count Kimi K3 reasoning tokens ( #58372 )
...
Signed-off-by: Elvir Crncevic <elvircrn@gmail.com >
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-09-25 23:29:56 +00:00
Rui Zhu and Claude
974cb65155
[Bugfix] V1: fix allowed_token_ids_mask aliasing in InputBatch.swap_states ( #48419 )
...
Signed-off-by: Rui Zhu <rui.zhu.rz399@yale.edu >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-25 23:23:11 +00:00
Ricardo-M-L and Claude Opus 4.6
b040321b33
fix: perf: use startswith(x, i) instead of string slicing to avoid O(N^2) ( #52580 )
...
Signed-off-by: Ricardo-M-L <ricardoporsche001@icloud.com >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
2026-09-25 23:22:13 +00:00
Taneem Ibrahim and Wentao Ye
6ec2f84e1a
[Pooling] Preserve BERT-family heads for raw logits ( #57664 )
...
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-09-25 19:10:14 -04:00
Linkun
614b55f0b6
[AuxOutput] Only require Model Runner V2 on GPU platform ( #58205 )
...
Signed-off-by: Linkun Chen <github@lkchen.net >
2026-09-25 15:55:21 -07:00
Oxana Korzh
e55d076f89
[Bugfix] Don't drop the rest of the allocator config when toggling expandable segments ( #57982 )
...
Signed-off-by: Oxana Korzh <okorzh@amd.com >
2026-09-25 17:34:19 -05:00
4ccfe12398
[watermarking] golden tests for backwards compatibility ( #56809 )
...
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai >
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-25 22:01:48 +00:00
yzong-rh and Ben Browning
b6761e8ded
[Bugfix] Default missing detail for Responses API input images ( #57241 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
Co-authored-by: Ben Browning <56071+bbrowning@users.noreply.github.com >
2026-09-25 14:08:55 -07:00
8b47e8b22c
[Perf][Frontend] Defer reasoning usage recounts for non-continuous chat streams ( #56067 )
...
Signed-off-by: Cheng Rui <286040359@qq.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-09-25 14:08:23 -07:00
Nick Hill and Kimi Code
de2d478d9a
[Bugfix][Frontend] Document 404 response for /generative_scoring ( #58788 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Kimi Code <noreply@moonshot.cn >
2026-09-25 14:08:00 -07:00
7871963fcc
[Bugfix][Frontend] Respect max_output_tokens in the Harmony tool-call loop ( #58551 )
...
Signed-off-by: errmakov <ide404@gmail.com >
Co-authored-by: Du Bin <8174807+dubin555@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-25 14:00:39 -07:00
Yousaf and sfeng33
83db8c8390
[Feature][Frontend] Add granite_thinking_parser reasoning parser for Granite 4.2 ( #55957 )
...
Signed-off-by: Yousaf shah <yousaf.shah@gmail.com >
Co-authored-by: sfeng33 <4florafeng@gmail.com >
2026-09-25 20:43:08 +00:00
e440b75bb6
[ROCm] Bump torch 2.13, triton 3.8, torchaudio, torchvision ( #50605 )
...
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com >
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com >
Signed-off-by: jpvillam <juan.villamizar@amd.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: jpvillam <juan.villamizar@amd.com >
2026-09-25 14:55:52 -05:00