87a4bf664f
[Perf][GDN] Slice pure spec-decode rows instead of a host-mask gather ( #58763 )
...
Signed-off-by: Luca Motz <luca.motz@icloud.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com >
2026-10-01 06:53:06 +00:00
4a9d6f9572
[Bugfix] Preserve sampling masks in DELTA and TITO streaming outputs ( #55935 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: fh3 <a@b>
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: fh3 <a@b>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-01 14:43:38 +08:00
70be6aae19
[Feature][Spec Decode] Support sampling mask replay for MRV2 MTP ( #59359 )
...
Signed-off-by: Cuiping Cheng <chengcuipingswu@163.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Cuiping Cheng <chengcuipingswu@163.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 13:51:39 +08:00
edaf3e7b61
[Attention][SM120] Occupancy-adaptive Triton split-K segment count ( #58482 )
...
Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: vadiklyutiy <156319763+vadiklyutiy@users.noreply.github.com >
2026-10-01 02:20:33 +00:00
7314360c9e
[Perf][PP] Skip sampled-token broadcasts whose requests leave the engine ( #58542 )
...
Signed-off-by: LostFox11 <wangziyue17@huawei.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: LostFox11 <wangziyue17@huawei.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-30 19:18:55 -07:00
34c458d27f
[Bugfix] Don't let a structured-output request sample from an unmasked row ( #54442 )
...
Signed-off-by: ArcheyChen <45146080+ArcheyChen@users.noreply.github.com >
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
Co-authored-by: GPT-5.6 <noreply@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-10-01 02:11:44 +00:00
df8fd42116
[CI][Spec Decode] Add async scheduling accuracy tests ( #55840 )
...
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-10-01 01:06:24 +00:00
Guy Stone and Claude Fable 5
37d6174096
[Metrics] Add --custom-histogram-buckets to override histogram bucket families ( #48867 )
...
The core engine histograms ship with default bucket boundaries tuned for
typical serving workloads. The `--custom-histogram-buckets` option replaces
the boundaries of one or more bucket families. Use it, for example, to track
sub-300ms latency SLOs with the request-phase histograms, whose smallest
default boundary is 0.3s.
Signed-off-by: Guy Stone <guys@spotify.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-10-01 01:03:39 +00:00
Nick Hill and Claude Opus 5.5
c055c1e075
[Core] Include the LoRA path in prefix-cache block hashes ( #59335 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 08:37:09 +08:00
Venky and Claude Opus 5.5
d7c55116ac
[CI] Fix MiniMax-M3 PP aux-state test mock after #58648 ( #59508 )
...
Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-09-30 17:34:52 -07:00
Kevin H. Luu and Claude Opus 5.5
8bc31bd210
[CI] Deflake the multi-API-server metrics test and Mooncake PD ports ( #59507 )
...
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 17:22:58 -07:00
Andrii Skliar and Andrii Skliar
deb148e730
[Mamba] Add FlashInfer ReplaySSM support for MTP ( #52928 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
2026-09-30 21:41:17 +00:00
kliuae and Nick Hill
7e583e615c
[Bugfix][Core] Fix mamba prefill checkpoint block reservation and prompt-end eviction in align mode ( #59175 )
...
Signed-off-by: kliuae <kuanfu.liu@embeddedllm.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-30 21:26:54 +00:00
Venky and Claude Opus 5.5
42f0c17ea7
[Bugfix][PP][Spec Decode] MiniMax-M3 EAGLE3 aux-state relay at PP > 1 and per-stage FlashInfer autotune ( #57197 )
...
Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-01 01:14:24 +04:00
3eb6cec22a
[Bugfix][HiSparse] Adopt GPU prefix copies after the hit's allocation ( #59282 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-30 15:43:09 -04:00
0f8b398158
[watermarking] add context deduplication support to speculative decoding ( #56807 )
...
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai >
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-09-30 12:22:10 -04:00
Yueh-Ting (eop) Chen and Matthew Bonanni
ff1b87cca2
[Bugfix][HiSparse] Preserve host prefix publication after request completion ( #59007 )
...
Signed-off-by: Yueh-Ting Chen <yueh.ting.chen@gmail.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-09-30 16:14:51 +00:00
Harry Mellor and Claude Opus 5.5
fc2c801ad9
Disable mypy arg-type and assignment checks in tests ( #59428 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 15:18:50 +00:00
Simon Danielsson and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2df122e65e
[ROCm][Perf][GLM-5.3-Flash] "Fit kpool top-k indices to AITER" with a single Triton kernel ( #58008 )
...
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-30 09:45:00 -05:00
Qiu Chunshuo and Lucas Wilkinson
98b45c1a94
[Feat] Support PP with PCP in GPU Model Runner V2 ( #59139 )
...
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-09-30 10:21:56 -04:00
Giulio De Pasquale and Isotr0py
2e52a558f0
[Bugfix][Core] Schedule encoder-only prompts larger than one step ( #59029 )
...
Signed-off-by: Giulio De Pasquale <git@depasquale.giugl.io >
Co-authored-by: Isotr0py <2037008807@qq.com >
2026-09-30 22:04:28 +08:00
Varun Sundar Rabindranath
5463fe4962
[KV-Offloading][TP] : Expand replicated_layout detection to multi-group MLA ( #57652 )
2026-09-30 09:11:03 -04:00
5225f1bacb
[MyPy][3/N] Fix MyPy errors in test groups (part 3) ( #55939 )
...
Signed-off-by: Taimys <ffy2281693949@gmail.com >
Signed-off-by: Luyi Xiao <81631911+luyixiao95@users.noreply.github.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Luyi Xiao <81631911+luyixiao95@users.noreply.github.com >
Co-authored-by: Taimys <ffy2281693949@gmail.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 11:06:07 +00:00
e5f3d08d72
[Frontend][RL] Align sleep-mode API responses and operation metrics ( #52864 )
...
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-29 23:16:38 -07:00
Andreas Karatzas
866fa130fa
[Core] Bound UniProc EngineCore startup threads to available CPUs ( #58946 )
...
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com >
2026-09-29 23:02:55 -07:00
Wentao Ye and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
834ad456d3
[Refactor] Remove dead tests code ( #58916 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-30 04:39:03 +00:00
90e13fc757
[Bugfix][HiSparse] Never allocate GPU pages without host backing ( #59036 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 04:13:29 +00:00
Mia Ojeda
e006d761a5
[Bugfix][CPU][DiffusionGemma] Support narrower canvas w/ sync scheduling ( #59107 )
...
Signed-off-by: ojeda-e <estefania.b.ojeda@gmail.com >
2026-09-29 23:28:50 -04:00
42d9f8ccdb
support rl feature : weight checker ( #51350 )
...
Signed-off-by: shiyuan680 <yangcheng104@huawei.com >
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: shiyuan680 <yangcheng104@huawei.com >
Co-authored-by: Ronald1995 <ronaldautomobile@163.com >
Co-authored-by: DeepSeek Harness <noreply@deepseek.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-30 03:13:43 +00:00
a8e069f81b
[Bugfix][MiniMax M3] Share target embeddings with MTP under PP ( #58648 )
...
Signed-off-by: Peiyuan Zhou <peiyuanzhou1994@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
2026-09-29 20:04:10 -07:00
Matthew Bonanni and Claude Opus 5.5
2798f66860
[Bugfix][HiSparse] Resolve MTP verification rows with a union residency kernel ( #59235 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 03:01:45 +00:00
765872e7ed
[Core][BugFix] Tag prefix-cache extra keys by source ( #51899 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com >
Co-authored-by: Tai An <antai12232931@outlook.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-29 18:24:59 -07:00
Harshil and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
be255076d0
[Bugfix] Reject a prefix_match_unit that a single KV cache group cannot honor ( #58021 )
...
Signed-off-by: QHarshil <harshil_c@hotmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-29 23:12:15 +00:00
c37f86e572
[Bugfix] GLM-5.3-Flash: fp8 plan dtype on SM90 sparse MLA, and right-size the indexer prefill workspace ( #55222 )
...
Signed-off-by: Mikhail Kostryukov <mike@triptrack.net >
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
Co-authored-by: Mikhail Kostryukov <drakosha81@proton.me >
Co-authored-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-09-29 22:36:39 +00:00
d882bddbea
[Bugfix][Mamba] Keep the prompt-end prefill checkpoint under sparse retention ( #59146 )
...
Signed-off-by: Jared Wen <jaredwen@inferact.ai >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com >
2026-09-29 21:00:16 +00:00
faacc13565
[Perf][KV Connector][Mooncake] Pack hybrid/MLA KV into coalesced transfer regions ( #57952 )
...
Signed-off-by: staryxchen <staryxchen@tencent.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-09-29 13:29:04 -07:00
5dd693122c
[MyPy][2/N] Fix mypy errors in small tests/ dirs ( #51043 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-29 21:27:45 +01:00
sashko-zakharchuk
a96ee5935d
[MRV2] Add DRY as a custom logits processor example ( #50584 )
...
Signed-off-by: Oleksandr Zakharchuk <oleksandr.zakharchuk@gmail.com >
2026-09-29 19:46:00 +00:00
YukioZzz and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
30ae8248b6
[KVConnector][MoRIIO] Support K3 DSpark hybrid READ ( #57700 )
...
Signed-off-by: Yichao Zhu <Yichao.Zhu@amd.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-29 19:14:24 +00:00
Chanbin Lim
91de237333
[Bugfix][DP] add_dp_placement_groups does not require ray[default] ( #57648 )
...
Signed-off-by: beenpow <57864041+beenpow@users.noreply.github.com >
2026-09-29 18:15:13 +00:00
f5ed3d9c03
[Fast Start] Charge daemon-held weights against gpu_memory_utilization ( #57298 )
...
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Xun Sun <UNIDY2002@outlook.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Xun Sun <UNIDY2002@outlook.com >
2026-09-29 17:58:35 +00:00
cd947f5c62
[MM] Fix compiled ViT attention output layouts ( #58182 )
...
Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
2026-09-29 06:27:03 -07:00
aoshen02 and OpenAI Codex
4861833ae2
[Bugfix][Core] Allow AuxOutput reset after abort without running requests ( #59060 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-29 09:23:32 +00:00
Chauncey
3e2a7e74a5
[Feat][Model] Enable KDA prefill checkpoints for GLM-5.3-Flash ( #56960 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
2026-09-29 09:14:30 +00:00
Andreas Karatzas and OpenAI Codex
3acddf01e7
[Bugfix][ROCm] Keep Mooncake bootstrap ports bound during startup ( #58967 )
...
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-29 04:14:26 -05:00
63b9931b73
[ROCm][DSv4.1] Paged MXFP4 sparse indexer on aiter's MQA-logits kernel ( #58671 )
...
Signed-off-by: Mehmet Cagri Kaymak <mehmet.kaymak@amd.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Shanshan Shen <467638484@qq.com >
2026-09-29 14:37:57 +08:00
Bowen Wang and Claude Opus 5.5
35d6fb3187
[Perf][Spec Decode] Enable fused multi-step draft decode for FlashInfer trtllm-gen ( #58371 )
...
Signed-off-by: Bowen Wang <bowen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-29 06:06:47 +00:00
Shinichi Hemmi
17cc3cd80e
[Model] Engine based plamo3 parser ( #56575 )
...
Signed-off-by: Shinichi Hemmi <shemmi@preferred.jp >
2026-09-29 05:17:42 +00:00
4e59d53ab5
[MRV2] Support stock torch.compile mode ( #59079 )
...
Signed-off-by: cheesecake <farzanaman99@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-09-28 20:17:48 -07:00
Mehmet Cagri
70ae0b7435
[ROCm][MLA] Enable sparse MLA Gluon kernel from Aiter ( #53492 )
...
Signed-off-by: Mehmet Cagri Kaymak <mehmet.kaymak@amd.com >
Signed-off-by: Mehmet Cagri <mehmet.kaymak@amd.com >
2026-09-29 09:08:18 +08:00
