2edbb32e58
[Bugfix][GLM-5.3-Flash] SM90 sparse MLA: index_kpool mismatch leads to corruption via unread query token ( #58704 )
...
Signed-off-by: Matt Mastracci <matthew@mastracci.com >
Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-09-25 12:36:36 -04:00
6491f481a7
[Bugfix] Stop allocator fragmentation from shrinking the KV cache during memory profiling ( #58430 )
...
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com >
Signed-off-by: Robert Shaw <robertgshaw2-redhat@h100-01.nemg-001.lab.rdu2.dc.redhat.com >
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: Robert Shaw <robertgshaw2-redhat@h100-01.nemg-001.lab.rdu2.dc.redhat.com >
Co-authored-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-25 12:16:27 -04:00
Nick Hill
234df712ed
[MRV2] Minor model_runner.py code cleanup ( #58610 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-09-25 07:59:27 -07:00
73a78e6f1f
[SpecDecode] Add LiLiCorr drafter ( #57934 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Signed-off-by: Andrii Skliar <andreyws96@gmail.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
2026-09-25 10:08:28 -04:00
Mark McLoughlin and AI Assistant
45d56c0baf
[Core] Model console logging as CLI configuration ( #57205 )
...
Add `--logging-config` CLI argument which can be supplied as
JSON or using dotted arguments. The `--log-level` argument
is provided for convenience, and `--log-config-file` is deprecated
in favor of `--logging-config.pylogging_config_file`.
Signed-off-by: Mark McLoughlin <markmc@redhat.com >
Co-authored-by: AI Assistant <noreply@openai.com >
2026-09-25 11:36:59 +01:00
29468dde8b
[Bugfix][KV Cache][MLA] Align packed block strides for V3.2 sparse MLA ( #55528 )
...
Signed-off-by: lz <145014769+200lz@users.noreply.github.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-25 05:33:36 +00:00
1f0bc49ee6
[Structured Outputs] Parse Lark grammars natively in the xgrammar backend ( #58321 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-09-25 04:35:10 +00:00
4013f3ad6c
[ROCm] Fix CI runtime and tests for MI355 DPX ( #58244 )
...
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: Mahesh Kunreddi <mahesh.kunreddi@amd.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-24 23:15:11 -05:00
김재억
32fc82343f
[Bugfix][LogitsProcessor] Validate ':' separator in custom logits processor FQCN ( #56020 )
...
Signed-off-by: 100milliongold <gadian88@gmail.com >
2026-09-25 01:49:20 +00:00
Nick Hill and Claude Opus 5.5
7dbd0a8d22
[Bugfix][MRV2] Treat padded prompt tails as spec-decode rows for hybrid models ( #58434 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-09-24 18:29:53 -07:00
Samuel Kim and Claude
34ee85d5ca
[Perf] Batch Mamba2 prefill SSM state saves, removing GPU<->CPU syncs ( #49371 )
...
Signed-off-by: samuelkim7 <samuelmwkim@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-25 01:11:19 +00:00
b3a87b2d04
[Bugfix][KVConnector] Finalize saves on steps without a forward ( #57775 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Kimi Code <noreply@moonshot.ai >
2026-09-25 00:06:36 +00:00
Khushali Desai
7833ff25dd
Release prompt_embeds tensor when its InputBatch slot is freed ( #57988 )
...
Signed-off-by: khushali9 <khushali.desai9@gmail.com >
2026-09-25 00:00:44 +00:00
7a9406690e
[KV Connector] Fix DecodeBench fp8 fill values and add a startup fill mode ( #58472 )
...
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-24 23:55:47 +00:00
d5051abaf1
[Bugfix][Mamba] Restore prompt-tail prefix-cache hits with MTP ( #58368 )
...
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
2026-09-24 23:46:42 +00:00
stefankoncarevic and Andreas Karatzas
9ac3b7c69f
[ROCm][CI] Mirror the DSv4-Flash disaggregated DP EP group on MI355 ( #58558 )
...
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-24 18:38:25 -05:00
f6aa2919bc
[Bugfix][Core] Keep every multimodal feature in the partial-block KV event ( #58288 )
...
Signed-off-by: haosenwang1018 <haosenwang1018@users.noreply.github.com >
Co-authored-by: haosenwang1018 <haosenwang1018@users.noreply.github.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-09-24 23:25:01 +00:00
snadampal and Nicolò Lucchesi
6a86bf626e
[PD][PushConnector] Record last activity of remotes on the D side ( #52245 )
...
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com >
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
2026-09-24 23:15:32 +00:00
Fanli Lin and Kunshang Ji
7edb27f68e
[XPU][CI] enable prompt embeds tests on XPU ( #58283 )
...
Signed-off-by: Lin, Fanli <fanli.lin@intel.com >
Signed-off-by: Fanli Lin <fanli.lin@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-09-25 06:56:59 +08:00
Jellow
e6c07ea576
[Bugfix][KV Cache] Fix incremental multimodal block hashing ( #51694 )
...
Signed-off-by: Jellow <49915976+CZT0@users.noreply.github.com >
Signed-off-by: Jellow <dvdx@foxmail.com >
2026-09-24 15:38:03 -07:00
Nick Hill
36f94d5fe8
[Bugfix][Outlines] Fix EOS termination and unconstrained masks after rejected drafts ( #58612 )
2026-09-24 21:55:56 +00:00
ab3de6edf2
[Perf] DiffusionGemma: constrained reads over the request's logprob_token_ids ( #58216 )
...
Signed-off-by: Matt Mastracci <matthew@mastracci.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-09-24 13:15:08 -07:00
92e9c6bf13
[ROCm] Fix misrouting race-condition in multi-decode P/D disagg with mori-io ( #51681 )
...
Signed-off-by: Vincent Cave <vincent.cave@amd.com >
Signed-off-by: Shiksha Patel <shikpate@amd.com >
Co-authored-by: Shiksha Patel <shikpate@amd.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-24 19:52:43 +00:00
90a9515006
[Cleanup] Remove online quantization support in fp8.py in favor of online shorthands ( #53585 )
...
Signed-off-by: Felix Marty <Felix.Marty@amd.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-24 12:48:14 -07:00
Nick Hill and Kimi Code
e30559b58f
[Core] Skip JIT monitor when JIT warmup is disabled ( #58590 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Kimi Code <noreply@moonshot.cn >
2026-09-24 12:08:16 -07:00
Nick Hill
09fe178dba
Revert "[DSpark] Support pipeline-parallel targets in aggregated serving ( #56956 )" ( #58484 )
2026-09-24 08:48:36 -07:00
Yueh-Ting (eop) Chen
26f49e336a
[Perf][Attention] Bound FlashInfer prefill dequantization scratch ( #57918 )
...
Signed-off-by: Yueh-Ting Chen <yueh.ting.chen@gmail.com >
2026-09-24 10:42:08 -04:00
Wentao Ye
3c42385d7d
[Refactor] Remove dead or duplicate tests ( #58446 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-24 10:29:56 -04:00
Harry Mellor and Claude Opus 5.5
d881812500
[Chore] Use Transformers v5 names and drop redundant processor use_fast ( #58550 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-09-24 14:14:10 +00:00
c0c35ddd8e
[Bugfix][CI] Fix the flaky sharded-sampling tests, and the engine teardown need ( #58342 )
...
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com >
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-24 02:05:11 -07:00
Andreas Karatzas and OpenAI Codex
23110a0c9e
[Bugfix] Capture prefill kernels for mixed FULL graphs ( #58275 )
...
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-24 02:05:07 -07:00
jackLei and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
bbc4ddeca9
[Bugfix][V1] Reject encoder-cache hits with mismatched embedding counts ( #57696 )
...
Signed-off-by: jackLei0901 <42642542+jackLei0901@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-24 08:40:47 +00:00
44bf011f44
[Dependency] Upgrade FlashInfer version to 0.7.0 ( #58069 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-09-23 22:45:23 -07:00
Kapil Arya and Or Ozeri
70fc359d25
[Bugfix][KV Offload] Retain offload event metadata through batch translation ( #57453 )
...
Signed-off-by: Kapil Arya <kapila@nvidia.com >
Signed-off-by: Kapil Arya <kapil.arya.17@gmail.com >
Co-authored-by: Or Ozeri <or@ozery.com >
2026-09-24 07:09:12 +03:00
Jhao-Ting Chen
4fb767ff0a
[Kernel] Resubmit PR 48666 - Gemma4 FP8 KV FA4 head dim 512 backend selection ( #53175 )
...
Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com >
2026-09-23 20:29:06 -07:00
e78e367c6e
[ROCm] Give turboquant boundary layers a layout-compatible backend ( #54988 )
...
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com >
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-23 22:00:07 -05:00
e6dc16cebd
[PCP] Support prefill context parallelism with data parallelism ( #57075 )
...
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
Co-authored-by: QiuChunshuo <qiuchunshuo@huawei.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-23 22:32:36 -04:00
Robert Shaw
6697a7dd30
[Scheduler] Tune --long-prefill-token-threshold adaptiveness ( #58459 )
...
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com >
2026-09-23 22:23:18 -04:00
e9f167129c
[CI][ROCM] Add the Fusion E2E TP2 Quick group on MI355, and the AITER MLA fix it needs ( #58369 )
...
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com >
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: Andreas Karatzas <andreas.karatzas@protonmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-23 21:01:58 -05:00
Bob Ok and Wentao Ye
bcdacfc1ff
[Test][Determinism] Cover chunked prefill in the batch-invariance suite ( #55612 )
...
Signed-off-by: Bob Ok <49168652+blipbyte@users.noreply.github.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-09-23 21:21:56 -04:00
Kevin H. Luu and Claude Opus 5.5
7ef0d69ea0
[5/12][ci-selector][CI] Skip the Proton GPU test when another CUPTI tool is injected ( #58455 )
...
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-23 17:25:41 -07:00
Divy and Claude Opus 5
e26547adfb
[Bugfix] Pick a KV block size supported by every attention backend ( #49845 )
...
Signed-off-by: Divy <divy@coralbricks.ai >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-23 16:36:35 -04:00
Dao007forever and Codex
c4f6ce4874
[Bugfix][NIXL] Restore successful push completion reporting ( #58188 )
...
Signed-off-by: Dao Le <Dao007forever@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-23 09:39:04 -07:00
Andreas Karatzas and OpenAI Codex
8033bd08c1
[ROCm][CI] Validate Mooncake and NIXL prefill/decode accuracy ( #58095 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-23 11:16:31 -05:00
157bcb7c48
[DSpark] Support pipeline-parallel targets in aggregated serving ( #56956 )
...
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
Co-authored-by: Kimi Code <noreply@moonshot.cn >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
2026-09-23 04:52:28 -07:00
Nick Hill
afec265bca
[MRV2] Miscellaneous code cleanup ( #57980 )
2026-09-23 01:51:52 -07:00
Nicolò Lucchesi and Yifan Qiao
9488318c6a
[Mooncake] Address review nits from #56855 ( #57174 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
Co-authored-by: Yifan Qiao <17067717+ivanium@users.noreply.github.com >
2026-09-23 10:15:45 +02:00
Andreas Karatzas and OpenAI Codex
0f2a15c927
[Tests] Select V2 for diffusion scheduler unit tests ( #58272 )
...
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-23 02:04:22 -05:00
88aa0d287d
[Spec decode] Support variable-length decode for Kimi-K3 adaptive ver ( #52988 )
...
Signed-off-by: Albert Cheng <albecheng@nvidia.com >
Signed-off-by: Albert Cheng (Engrg-Hardware 1) <albecheng@login-bia01.bia.clusters.nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
2026-09-22 20:43:32 -07:00
94f4170df3
[Bugfix] Set worker runtime threads before profiling and compilation ( #55891 )
...
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-22 21:12:20 -05:00
