Jürgen Schmied
ed2a9665ea
Merge branch 'main' into gdn-recoverssm
2026-09-27 08:01:32 +02:00
Juntian Liu and Claude Opus 5.5
a9eafde59c
[Perf][DSv4.1] Fuse small-batch WO-A with inverse RoPE and MXFP8 quant on SM100/SM103 ( #58634 )
...
Signed-off-by: Juntian Liu <Juntianl777@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-09-26 21:16:12 -07:00
Varshith and Claude
fba47397f3
[Bugfix] V1: clear stale allowed_token_ids mask in InputBatch.condense ( #43931 )
...
Signed-off-by: Varshith <kvarshithgowda@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 03:44:54 +00:00
YannikHinteregger and xijiade.aihemaiti
fff03267da
[Bugfix][NIXL] Release a dead peer's NIXL state without waiting for TTL ( #50047 )
...
Signed-off-by: Yannik Hinteregger <37209495+YannikHinteregger@users.noreply.github.com >
Co-authored-by: xijiade.aihemaiti <3146335281@qq.com >
2026-09-26 20:30:09 -07:00
Hongxia Yang
3137ff0773
[ROCm][Kimi-K3] Make VLLM_ROCM_USE_AITER_MOE_SITUV2 select a4w4/a8w4/a16w4 ( #58201 )
...
Signed-off-by: Hongxia Yang <hongxia.yang@amd.com >
2026-09-27 02:26:49 +00:00
Lio Einaudi and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
0c87a197b8
[Bugfix] Disable sequence parallelism / async TP under batch invariance and add a TP regression test ( #56377 )
...
Signed-off-by: LioEinaudi <zhao3024667639@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-26 22:19:08 -04:00
Thien Tran
7a877ae34d
[Qwen3.8-Flash-Next] Avoid memory fragmentation in QSA indexer logits workspace ( #57105 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-09-27 10:11:52 +08:00
eb0f2ca37f
[mooncake] support CUSTOM_MEM_POOL in vllm ( #49300 )
...
Signed-off-by: bruce.xu <bruce.xb@alibaba-inc.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: bruce.xu <bruce.xb@alibaba-inc.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-27 00:26:05 +00:00
mevince
cd1daf58a9
[Metrics][KV Offload] Add Prometheus metrics for SimpleCPUOffloadConnector ( #57251 )
...
Signed-off-by: Vincent <vincexxchan@gmail.com >
2026-09-26 17:17:11 -07:00
Jürgen Schmied and Claude
03cae7837f
[Spec Decode][Qwen4Exp] RecoverSSM for Qwen GDN and the PLE short conv
...
With --use-replayssm and speculative decoding, Qwen4Exp's Gated DeltaNet layers verify the
draft window from a read-only checkpoint and keep a per-token replay record
[correction(V), normalized key(K), gate(1)] in fp32. After sampling, one commit replays the
accepted tokens and writes the recurrent state once (plus the block-boundary state in align
mode), instead of snapshotting the state after every draft position. The PLE dilated short
conv follows the same protocol (accepted-suffix compaction). num_speculative_blocks = 0,
so no per-draft Mamba blocks are reserved.
- GDN verify/commit kernels: model_executor/layers/mamba/gdn/recoverssm_gdn.py
- metadata/backends: v1/attention/backends/gdn_recoverssm.py, ple_recoverssm.py
- the verify reads strided q/k/v views of the conv output and writes in place
- builders declare AttentionCGSupport.NEVER: FULL_AND_PIECEWISE falls back to PIECEWISE;
the fallback now also counts breakable CUDA graphs (empty splitting_ops, with or
without torch.compile) as piecewise
attention instead of dropping to NONE
- the generic Mamba metadata builder skips the Mamba2 ReplaySSM ring-buffer workspaces
when --use-replayssm selects RecoverSSM (the PLE short conv uses that builder)
- tests: GDN verify/commit vs the native kernels (fp32 and bf16 state, align boundaries,
mixed and ragged batches, extreme gates), PLE vs token-by-token decode, config selection
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 01:17:24 +02:00
4be061c5af
[Kernel] Bump FlashKDA to keep the recurrent state in fp32 ( #58846 )
...
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-09-26 22:20:44 +00:00
927c87b347
[PCP][DCP] Support DCP target model with non-DCP Dspark ( #56723 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-26 18:12:35 -04:00
Jürgen Schmied and Claude
a80e41d690
[Mamba] Share RecoverSSM commit kernels and helpers across models
...
RecoverSSM's commit-plan and conv-window compaction kernels are model independent.
Move them from models/kimi_k3 into model_executor/layers/mamba/ops/recoverssm.py
(Kimi-K3 keeps its private names as import aliases; kernel source unchanged), add
mamba/recoverssm_utils.py (activation predicate, argument validation) and the
generic spec-row/request mapping helpers to v1/attention/backends/recoverssm_metadata.py.
CacheConfig.use_kda_recoverssm becomes use_recoverssm: the switch now selects RecoverSSM
for any model whose recurrent layers implement it, not only KDA.
No functional change for Kimi-K3.
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 00:07:04 +02:00
Yongye Zhu and Claude Opus 5.5
77871126f9
[Kernel][DSV4.1] Fuse MoE finalize into the TP all-reduce + mHC boundary ( #58586 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-27 03:08:36 +08:00
Cyrus Leung
379e9a1ea8
[Security] Harden message sanitization ( #58832 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-09-26 16:28:32 +00:00
Kevin H. Luu
7d8c5fe9a9
[Bugfix][DSV4.1] Avoid host sync in ViT CUDA graph replay metadata ( #58499 )
2026-09-26 11:37:36 -04:00
Andreas Karatzas
0239bd2549
[CI] Stabilize batch submission in full CUDA graph tests ( #58810 )
2026-09-26 11:35:56 -04:00
Wentao Ye
3576691c42
[GLM5.3 Bug] Fix sparse indexer attn topk backend selection ( #58594 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-26 11:35:28 -04:00
Misha Goin
79ac28c5bd
[CI] Reduce CUDA graph mode test overhead ( #58749 )
2026-09-26 11:22:33 -04:00
Ashraf Bhuiyan
5113cf9d16
[Mypy] Fix mypy typing for Qwen and Qianfan models ( #58046 )
...
Signed-off-by: Ashraf Bhuiyan <mbhuiyan@redhat.com >
2026-09-26 14:31:51 +00:00
Robert Shaw and Robert Shaw
dfab504333
[Bugfix][Frontend] Detect Anthropic inline-system merge against the resolved chat template ( #58754 )
...
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Robert Shaw <robshaw@redhat.com >
2026-09-26 14:00:56 +00:00
Robert Shaw and Robert Shaw
d64f277c54
[Bugfix] Fix Anthropic Thinking Disabled with P/D ( #58786 )
...
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Robert Shaw <robshaw@redhat.com >
2026-09-26 13:50:57 +00:00
Willian Z and mgoin
ad6817b68d
[Perf][MoE] Index expert mapping lookups in RoutedExperts.load_weights ( #58720 )
...
Signed-off-by: Willian <willian@willian.email >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
2026-09-26 06:48:24 -07:00
Wentao Ye
bf93e1f92c
[Refactor] Remove dead tests utils ( #58803 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-26 09:16:53 -04:00
Itay Alroy and Tyler Michael Smith
2bb7605ae1
[Elastic EP] Fix EPLB load statistics during scaling ( #58473 )
...
Signed-off-by: Itay Alroy <ialroy@nvidia.com >
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
2026-09-26 12:05:37 +00:00
Juan Pérez de Algaba and Cursor
31f2e70cd3
[Security] Gate per-request multimodal processor kwargs ( #58830 )
...
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-26 11:10:35 +00:00
2f41e00235
[CI] Split (B200) Miscellaneous Kernels into mHC, FLA Ops and Misc named jobs ( #58609 )
...
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-09-26 04:10:14 -07:00
a4eb3f25d6
[Spec Decode] Enable Gemma4 DSpark adaptive verification with FlashInfer ( #57263 )
...
Signed-off-by: zixi-qi <zixi@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-26 17:01:17 +08:00
Aarushi Jain and Andreas Karatzas
3b4566c5cf
[CI] Only isolate the registry tests that need a fresh process ( #58764 )
...
Signed-off-by: aarushjain29 <Aarushi.Jain2@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-26 14:03:45 +08:00
ddd6fbca14
[Bugfix][Frontend][Rust Frontend] Update DeepSeek V4.1 Flash reasoning effort mappings ( #58316 )
...
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: zhec <chengyunfei@ruc.edu.cn >
2026-09-26 02:18:58 +00:00
xiao-llm and Cursor
5840d95284
[Bugfix][ROCm] AMD-Quark mixed-precision DeepSeek-V4.1 support ( #57071 )
...
Signed-off-by: Xiao Yu <xiao.yu.dc@outlook.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-25 20:51:18 -05:00
b6d2b4fc25
[RL] Add sharding-aware NCCL M2N weight transfer ( #51520 )
...
Signed-off-by: Ke Wen <kwen@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-26 01:38:53 +00:00
Shuolei Wang
4bb804cc5b
[Perf][DSv4.1] Shard the Engram wkv projection across TP ranks ( #58678 )
...
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com >
2026-09-26 01:15:57 +00:00
Chandra Mourya and VS Chandra Mourya
8a2364605c
[Core] Bound draft-token RPC waits by the execute-model timeout ( #58779 )
...
Signed-off-by: VS Chandra Mourya <219748331+vschandramourya@users.noreply.github.com >
Co-authored-by: VS Chandra Mourya <219748331+vschandramourya@users.noreply.github.com >
2026-09-26 00:40:22 +00:00
Baljinder Hothi and Flora Feng
fdcce47e9b
[Bugfix][Frontend] Fix Inkling tool name leaking into content after reasoning ( #58792 )
...
Signed-off-by: Baljinder Hothi <baljinder.hothi@cohere.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-09-26 00:38:52 +00:00
da1aaec31f
[CI] Split (H200 MIG/MI300) Basic Correctness into named jobs ( #57054 )
...
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com >
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-25 17:05:33 -07:00
Elvir Crnčević and Flora Feng
9bf44c4013
[Bugfix][Reasoning] Count Kimi K3 reasoning tokens ( #58372 )
...
Signed-off-by: Elvir Crncevic <elvircrn@gmail.com >
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-09-25 23:29:56 +00:00
Rui Zhu and Claude
974cb65155
[Bugfix] V1: fix allowed_token_ids_mask aliasing in InputBatch.swap_states ( #48419 )
...
Signed-off-by: Rui Zhu <rui.zhu.rz399@yale.edu >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-25 23:23:11 +00:00
Ricardo-M-L and Claude Opus 4.6
b040321b33
fix: perf: use startswith(x, i) instead of string slicing to avoid O(N^2) ( #52580 )
...
Signed-off-by: Ricardo-M-L <ricardoporsche001@icloud.com >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
2026-09-25 23:22:13 +00:00
Taneem Ibrahim and Wentao Ye
6ec2f84e1a
[Pooling] Preserve BERT-family heads for raw logits ( #57664 )
...
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-09-25 19:10:14 -04:00
Linkun
614b55f0b6
[AuxOutput] Only require Model Runner V2 on GPU platform ( #58205 )
...
Signed-off-by: Linkun Chen <github@lkchen.net >
2026-09-25 15:55:21 -07:00
Oxana Korzh
e55d076f89
[Bugfix] Don't drop the rest of the allocator config when toggling expandable segments ( #57982 )
...
Signed-off-by: Oxana Korzh <okorzh@amd.com >
2026-09-25 17:34:19 -05:00
4ccfe12398
[watermarking] golden tests for backwards compatibility ( #56809 )
...
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai >
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-25 22:01:48 +00:00
yzong-rh and Ben Browning
b6761e8ded
[Bugfix] Default missing detail for Responses API input images ( #57241 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
Co-authored-by: Ben Browning <56071+bbrowning@users.noreply.github.com >
2026-09-25 14:08:55 -07:00
8b47e8b22c
[Perf][Frontend] Defer reasoning usage recounts for non-continuous chat streams ( #56067 )
...
Signed-off-by: Cheng Rui <286040359@qq.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-09-25 14:08:23 -07:00
Nick Hill and Kimi Code
de2d478d9a
[Bugfix][Frontend] Document 404 response for /generative_scoring ( #58788 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Kimi Code <noreply@moonshot.cn >
2026-09-25 14:08:00 -07:00
7871963fcc
[Bugfix][Frontend] Respect max_output_tokens in the Harmony tool-call loop ( #58551 )
...
Signed-off-by: errmakov <ide404@gmail.com >
Co-authored-by: Du Bin <8174807+dubin555@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-25 14:00:39 -07:00
Yousaf and sfeng33
83db8c8390
[Feature][Frontend] Add granite_thinking_parser reasoning parser for Granite 4.2 ( #55957 )
...
Signed-off-by: Yousaf shah <yousaf.shah@gmail.com >
Co-authored-by: sfeng33 <4florafeng@gmail.com >
2026-09-25 20:43:08 +00:00
e440b75bb6
[ROCm] Bump torch 2.13, triton 3.8, torchaudio, torchvision ( #50605 )
...
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com >
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com >
Signed-off-by: jpvillam <juan.villamizar@amd.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: jpvillam <juan.villamizar@amd.com >
2026-09-25 14:55:52 -05:00
31cc226401
[ROCm][CI] Run the MLA attention+quant fusion test on ROCm ( #58717 )
...
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com >
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-25 14:45:32 -05:00