c4fc3c7d32
[Bugfix][Spec Decode] Per-module LM heads for multi-layer MTP on Model Runner V2 ( #58921 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: chanh <channguyen@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 01:09:14 -07:00
e5e38ba9b7
[Model Runner V2] Support randomized dummy inputs ( #58411 )
...
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-10-01 00:52:26 -07:00
5addde2a0a
[CPU][Zen] Pass f32 weight scales to the zentorch INT8 MoE ( #59434 )
...
Signed-off-by: Ganesh R <Ganesh.R@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
2026-10-01 15:51:53 +08:00
e12291d733
[CPU][Zen] Add DA8W4 (W4A8) int4 support for dense and MoE layers ( #54024 )
...
Signed-off-by: R <Ganesh.R@amd.com >
Signed-off-by: Ganesh R <Ganesh.R@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-01 15:18:25 +08:00
Wentao Ye
4d6874d137
[GLM 5.3 Perf] Skip qlnorm calculation for MHA, 4.4~7.7% E2E TTFT Improvement ( #58845 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-10-01 15:13:57 +08:00
aee8fe202a
[Bugfix][ROCm] Fix race on AITER MLA FP8 prefill scheduling metadata under async scheduling ( #58887 )
...
Signed-off-by: Mehdi Ghanimifard <mehdi.ghanimifard@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-01 15:02:52 +08:00
87a4bf664f
[Perf][GDN] Slice pure spec-decode rows instead of a host-mask gather ( #58763 )
...
Signed-off-by: Luca Motz <luca.motz@icloud.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com >
2026-10-01 06:53:06 +00:00
4a9d6f9572
[Bugfix] Preserve sampling masks in DELTA and TITO streaming outputs ( #55935 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: fh3 <a@b>
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: fh3 <a@b>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-01 14:43:38 +08:00
Shuolei Wang and Thien
c9578f10ab
[Qwen3.8-Flash-Next] Fuse main QK-norm/RoPE/gate and KV-cache write into the QSA pre-indexer launch ( #57097 )
...
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com >
Signed-off-by: Thien <thien@inferact.ai >
Co-authored-by: Thien <thien@inferact.ai >
2026-10-01 14:21:06 +08:00
70be6aae19
[Feature][Spec Decode] Support sampling mask replay for MRV2 MTP ( #59359 )
...
Signed-off-by: Cuiping Cheng <chengcuipingswu@163.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Cuiping Cheng <chengcuipingswu@163.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 13:51:39 +08:00
Shijie Lyu
37cecc8141
[Bugfix][Tokenizer] Fix off-by-one max_token_id from vocab_size ( #59491 )
...
Signed-off-by: Shijie Lyu <lshjhf@gmail.com >
2026-10-01 05:43:30 +00:00
Juntian Liu and Claude Opus 5.5
4c2d277643
[Perf][DSv4.1] Faster Engram host lookups: sorted rows, inline big lookups ( #59327 )
...
Signed-off-by: Juntian Liu <Juntianl777@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-01 11:54:44 +08:00
namgyu-youn and Thien Tran
4ed438b79c
[Perf][Qwen4Exp] Add SM100 low-latency decode GEMM plans ( #59214 )
...
Signed-off-by: namgyu-youn <namgyu.dev@gmail.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-10-01 03:40:55 +00:00
Yoni Gozlan
2eaa3bc5ac
[Frontend] Enrich chat_parsing streaming events ( #58603 )
...
Give serving adapters the delimiter spans and opener captures they need,
and keep streaming after a region that fails to parse.
Signed-off-by: Yoni Gozlan <yonigozlan@users.noreply.github.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Fix mypy index types in the event-stream test.
2026-10-01 11:13:13 +08:00
Ning Xie
a37c402f32
[weight loader] add dtype equality validation between parameter and weight ( #51792 )
...
Signed-off-by: Andy Xie <andy.xning@gmail.com >
2026-10-01 03:07:31 +00:00
Ning Xie
bfbcde26d5
[Observability] add model initializing duration log ( #52141 )
...
Signed-off-by: Andy Xie <andy.xning@gmail.com >
2026-10-01 02:54:33 +00:00
Nils Matteson
9e6550bd05
[Bugfix] Register IR providers before hashing configuration ( #58756 )
...
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com >
2026-10-01 02:34:28 +00:00
bc505fc823
[Bugfix] Support CuTe DSL 4.8.0 block-scale API ( #59480 )
...
Signed-off-by: hjjq <50634613+hjjq@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Shang Wang <samshang.wang@mail.utoronto.ca >
2026-10-01 02:30:06 +00:00
766b4ae54c
[Kimi-K3] Add FlashInfer speculative KDA backend ( #54255 )
...
Signed-off-by: Duncan Moss <dmoss@nvidia.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-10-01 02:28:50 +00:00
edaf3e7b61
[Attention][SM120] Occupancy-adaptive Triton split-K segment count ( #58482 )
...
Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: vadiklyutiy <156319763+vadiklyutiy@users.noreply.github.com >
2026-10-01 02:20:33 +00:00
7314360c9e
[Perf][PP] Skip sampled-token broadcasts whose requests leave the engine ( #58542 )
...
Signed-off-by: LostFox11 <wangziyue17@huawei.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: LostFox11 <wangziyue17@huawei.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-30 19:18:55 -07:00
34c458d27f
[Bugfix] Don't let a structured-output request sample from an unmasked row ( #54442 )
...
Signed-off-by: ArcheyChen <45146080+ArcheyChen@users.noreply.github.com >
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
Co-authored-by: GPT-5.6 <noreply@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-10-01 02:11:44 +00:00
Guy Stone and Claude Fable 5
37d6174096
[Metrics] Add --custom-histogram-buckets to override histogram bucket families ( #48867 )
...
The core engine histograms ship with default bucket boundaries tuned for
typical serving workloads. The `--custom-histogram-buckets` option replaces
the boundaries of one or more bucket families. Use it, for example, to track
sub-300ms latency SLOs with the request-phase histograms, whose smallest
default boundary is 0.3s.
Signed-off-by: Guy Stone <guys@spotify.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-10-01 01:03:39 +00:00
Nick Hill and Claude Opus 5.5
c055c1e075
[Core] Include the LoRA path in prefix-cache block hashes ( #59335 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 08:37:09 +08:00
Andrii Skliar and Andrii Skliar
deb148e730
[Mamba] Add FlashInfer ReplaySSM support for MTP ( #52928 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
2026-09-30 21:41:17 +00:00
kliuae and Nick Hill
7e583e615c
[Bugfix][Core] Fix mamba prefill checkpoint block reservation and prompt-end eviction in align mode ( #59175 )
...
Signed-off-by: kliuae <kuanfu.liu@embeddedllm.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-30 21:26:54 +00:00
Venky and Claude Opus 5.5
42f0c17ea7
[Bugfix][PP][Spec Decode] MiniMax-M3 EAGLE3 aux-state relay at PP > 1 and per-stage FlashInfer autotune ( #57197 )
...
Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-01 01:14:24 +04:00
73c7cae4d7
[Bugfix][HiSparse] Stop the host pool feeding device KV cache residency metrics ( #58725 )
...
Signed-off-by: Yueh-Ting Chen <yueh.ting.chen@gmail.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 20:50:01 +00:00
3eb6cec22a
[Bugfix][HiSparse] Adopt GPU prefix copies after the hit's allocation ( #59282 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-30 15:43:09 -04:00
84f738f187
[Perf][MiniMax-M3] Triton indexer: decode grid retune + SM12.0 split-K ( #56151 )
...
Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com >
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com >
Co-authored-by: Kyle Liang <199154931+kyleliang-nv@users.noreply.github.com >
2026-09-30 23:33:23 +04:00
Aarushi Jain and Andreas Karatzas
b33e1cb172
[ROCm]Transpose compressed-tensors MoE weights on device ( #59253 )
...
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-30 12:44:40 -05:00
8f24ab30a3
[ROCm][Kimi-K3][Perf] Fuse MLA decode KV-cache write and Q-prep via AITER ( #57640 )
...
Signed-off-by: Rita Brugarolas Brufau <rita.brugarolasbrufau@amd.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-30 12:20:05 -05:00
0f8b398158
[watermarking] add context deduplication support to speculative decoding ( #56807 )
...
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai >
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-09-30 12:22:10 -04:00
73a5831127
[Frontend] Port chat_parsing core from Transformers ( #58602 )
...
Signed-off-by: Yoni Gozlan <yonigozlan@users.noreply.github.com >
Co-authored-by: Yoni Gozlan <yonigozlan@users.noreply.github.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-01 00:21:02 +08:00
Yueh-Ting (eop) Chen and Matthew Bonanni
ff1b87cca2
[Bugfix][HiSparse] Preserve host prefix publication after request completion ( #59007 )
...
Signed-off-by: Yueh-Ting Chen <yueh.ting.chen@gmail.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-09-30 16:14:51 +00:00
Eldar Kurtić
c32513b99d
Add support for unquantized ngram in CT format ( #59431 )
2026-09-30 16:08:31 +00:00
Juan Pérez de Algaba and Cursor
cff08b461e
[Security] Fix chat template resource-exhaustion DoS (GHSA-4hhp-h66f-… ( #50300 )
...
Signed-off-by: jperezde <jperezde@redhat.com >
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-30 08:32:00 -07:00
Yida Weng and Grok
17e9295dd5
[BugFix][Multimodal] Pick worst-case DeepSeek-V4 VL dummy image size ( #59271 ) ( #59373 )
...
Signed-off-by: Yida Weng <84164537+YidaWeng@users.noreply.github.com >
Co-authored-by: Grok <grok@x.ai >
2026-09-30 15:09:11 +00:00
Sergey Shlyapnikov
4e1182a3a6
[ROCm][MoE] Pad the AITER MoE intermediate size at allocation time, and round the expert-group count to a kernel that exists ( #55368 )
...
Signed-off-by: Sergei Shliapnikov <sergei.shliapnikov@amd.com >
2026-09-30 23:08:21 +08:00
Simon Danielsson and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2df122e65e
[ROCm][Perf][GLM-5.3-Flash] "Fit kpool top-k indices to AITER" with a single Triton kernel ( #58008 )
...
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-30 09:45:00 -05:00
aoshen02 and Claude Opus 5.5
643bbef55c
[Refactor] Share workspace and model runner init between GPU and XPU workers ( #59200 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 14:29:12 +00:00
Qiu Chunshuo and Lucas Wilkinson
98b45c1a94
[Feat] Support PP with PCP in GPU Model Runner V2 ( #59139 )
...
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-09-30 10:21:56 -04:00
Giulio De Pasquale and Isotr0py
2e52a558f0
[Bugfix][Core] Schedule encoder-only prompts larger than one step ( #59029 )
...
Signed-off-by: Giulio De Pasquale <git@depasquale.giugl.io >
Co-authored-by: Isotr0py <2037008807@qq.com >
2026-09-30 22:04:28 +08:00
Varun Sundar Rabindranath
5463fe4962
[KV-Offloading][TP] : Expand replicated_layout detection to multi-group MLA ( #57652 )
2026-09-30 09:11:03 -04:00
effdea9a53
[MM][Mistral] Add compile support for Pixtral vision encoders ( #57168 )
...
Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com >
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-30 20:27:12 +08:00
almayne and Li, Jiang <jiang1.li@intel.com>
93dd80f636
[CPU] Conv1d optimised kernel for aarch64 ( #54093 )
...
Signed-off-by: Anna Mayne <anna.mayne@arm.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-09-30 20:23:42 +08:00
df5668e5c9
[XPU] Dispatch nn.LayerNorm to fused SYCL kernel via CustomOp ( #57172 )
...
Signed-off-by: Michal Ganczarenko <michal.ganczarenko@intel.com >
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-09-30 12:19:33 +00:00
3d5625cc3c
[Security] Authenticate shared-memory multimodal cache handles ( #59357 )
...
Signed-off-by: Clinton Thomas <1033162+KernelClint@users.noreply.github.com >
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-09-30 05:16:26 -07:00
Rukhaiya2004 and Li, Jiang <jiang1.li@intel.com>
72e7874fa6
[Hardware][Power] Enable W4A16 (AWQ & GPTQ) quantization on POWER10 using VSX ( #59149 )
...
Signed-off-by: Rukhaiya <bibirukhaiya123@gmail.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-09-30 19:44:48 +08:00
Jakub Zakrzewski and Codex
5a4351ab09
[MM] Keep device input normalization fused when encoder compilation is enabled ( #59195 )
...
Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com >
Co-authored-by: Codex <codex@openai.com >
2026-09-30 11:08:24 +00:00
