d1f3d8b870
[Quantization] Prefer Humming before Marlin backends on SM90 ( #56997 )
...
Signed-off-by: jinzhen.ljz <jinzhen.ljz@antgroup.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-06 12:31:35 -07:00
e37e51dd24
[Metrics] Expose cached prompt tokens by cache tier ( #56318 )
...
Signed-off-by: Cam Quilici <cjquilici@gmail.com >
Signed-off-by: Cam Quilici <cameron@semianalysis.com >
Co-authored-by: Cam Quilici <cameron@semianalysis.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-10-06 14:52:05 -04:00
Nishant Revur and Claude
ad0f67a7ec
[Bugfix][Spec Decode] Keep heterogeneous-vocab draft models on Model Runner V1 ( #59541 )
...
Signed-off-by: Nishant Revur <kumrfni@amazon.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-06 18:03:44 +00:00
3403e0f176
[Bugfix] Persist FlashInfer autotune cache per rank to fix the rank-0-only cache-hit deadlock ( #57635 )
...
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com >
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
Co-authored-by: NickLucche <nicolo.lucchesi@mistral.ai >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-10-06 17:51:01 +00:00
Luciano Martins and Luciano Martins
02b83919aa
[Model] Support EmbeddingGemma2 multimodal pooling architecture ( #60254 )
...
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com >
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com >
2026-10-06 13:43:38 -04:00
7b665f7a58
[Frontend] Integer token IDs for generate output logprobs (GenerateLogProbs) ( #58181 )
...
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com >
Co-authored-by: Yin Li <kevinli.ai.work@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-06 17:22:47 +00:00
f8d92cae41
[Quantization] Layer re-quantization for linear layers through online quantization API (MXFP8 -> FP8 PTPC showcase) ( #55684 )
...
Signed-off-by: Felix Marty <Felix.Marty@amd.com >
Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com >
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com >
Co-authored-by: Tan Pin Siang <tanpinsiang@gmail.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-10-06 11:38:18 -05:00
eky-amd and Claude Fable 5
29f955cce0
[ROCm] Fuse MLA dual RMSNorm + FP8 group quant for DeepSeek-R1 ( #54857 )
...
Signed-off-by: Ethan Ky <ethan.ky@amd.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-10-06 11:26:38 -05:00
stefankoncarevic
4516896626
[CI/Build] Warm the DP engines before measuring load balance in test_load ( #60241 )
...
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com >
2026-10-06 11:23:37 -05:00
Giancarlo Delfin
fcf53d240c
[Spec Decode] Remove eager metadata rebuild during MTP fused multi-step decode ( #58463 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-10-06 12:17:30 -04:00
Samuel Nordmann and Misha Goin
4a99ca40d6
[Perf][MoE] Eliminate staging from NCCL symmetric reduce-scatter ( #49194 )
...
Signed-off-by: samnordmann <snordmann@nvidia.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-10-06 07:37:01 -07:00
Matthias Gehre
b267fe4881
[ROCm][Perf] W4A16: pad gfx11 weight and activation strides ( #56301 )
...
Signed-off-by: Matthias Gehre <matthias.gehre@amd.com >
2026-10-06 08:22:33 -05:00
Artem Perevedentsev and Shashank Varma
049507aa76
[Bugfix][Structured Output] Flag multi-branch allOf as unsupported for xgrammar ( #59061 )
...
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com >
Co-authored-by: Shashank Varma <324153016+shashankvarma499@users.noreply.github.com >
2026-10-06 17:01:04 +04:00
Sherif Waly
25385102fe
[Bugfix][KV Offload] Honor speculative cacheability in SimpleCPU ( #60071 )
...
Signed-off-by: Sherif Waly <sherif.waly@mistral.ai >
2026-10-06 14:25:08 +02:00
Mia Ojeda
3313825983
[Bugfix][DiffusionGemma] Read the causal mask as int32 and compile the sample step ( #59992 )
...
Signed-off-by: ojeda-e <eojeda@league.com >
2026-10-06 19:25:51 +08:00
09e0ce3d82
[Bugfix][Frontend] Apply model-default reasoning parser in GPU-less render server ( #54835 )
...
Signed-off-by: Shimi Bandiel <shimib@google.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-06 18:46:24 +08:00
roikoren755 and zjy0516
e11962fc1c
[Feat][Mamba2] Enable internal prefill checkpoints ( #57329 )
...
Signed-off-by: Roi Koren <roik@nvidia.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
2026-10-06 07:53:51 +00:00
edfccc9469
[ROCM][Perf] Add ROCM_SEGMENTED_ATTN attention-backend works across RDNA3/3.5/4 ( #59132 )
...
Signed-off-by: root <jeffaw99@hotmail.com >
Signed-off-by: big-yellow-duck <jeffaw99@hotmail.com >
Co-authored-by: Tan Pin Siang <1716735+tanpinsiang@users.noreply.github.com >
Co-authored-by: vllmellm <190700713+vllmellm@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
2026-10-06 07:53:47 +00:00
15c1cc12ac
[Bugfix][EPLB] Use group-local ranks for torch P2P transfers ( #55804 )
...
Signed-off-by: LostFox11 <wangziyue17@huawei.com >
Co-authored-by: LostFox11 <wangziyue17@huawei.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: YangShuai52 <178856220+YangShuai52@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-06 07:44:57 +00:00
sungbin1015 and Claude
1e5d0ea888
[CI/Build] Fix incorrect CLI arg validation tests in test_cli_args.py ( #46779 )
...
Signed-off-by: sungbin1015 <sbin@solbox.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-06 05:58:47 +00:00
40c86ae1ce
[Kernel] Enable OAI Triton MXFP4 MoE on SM12x ( #58877 )
...
Signed-off-by: lazymio <mio@lazym.io >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Shengqi Chen <harry-chen@outlook.com >
2026-10-06 01:49:13 -04:00
Bugen Zhao and Claude Opus 5.5
b3082d3044
[Rust Frontend] Remove the PyO3 tool-parser bridge ( #59744 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-06 05:46:27 +00:00
Bugen Zhao and Claude Opus 5.5
5145e35e50
[Frontend] Port the MiniMax M3 tool parser to the parser engine ( #59743 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-06 05:46:27 +00:00
czhu-cohere and aoshen02
6fddc9b21e
[Profiler] Add per-session profiling controls to Python and Rust frontends ( #57875 )
...
Signed-off-by: czhu <conway.zhu@cohere.com >
Signed-off-by: Conway Zhu <conway.zhu@cohere.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-06 05:21:19 +00:00
aoshen02 and Claude Opus 5.5
9a1f6fd4cb
[Model][DeepSeek-V4.1] Permute mega-attention wq_b / wo_a while loading ( #60064 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-06 05:20:09 +00:00
Chinmay-Kulkarni-AMD and Cursor Agent
533d81b4b0
[ZenCPU][Test] Add zentorch/torch version pin consistency test ( #52888 )
...
Signed-off-by: Chinmay Kulkarni <Chinmay.Kulkarni@amd.com >
Co-authored-by: Cursor Agent <cursoragent@cursor.com >
2026-10-06 13:12:19 +08:00
eb2901b735
[Perf][DSv4.1] Run decoder replay layers in CUDA graphs and trim in PIECEWISE graphs ( #59532 )
...
Signed-off-by: Juntian Liu <Juntianl777@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-05 22:11:45 -07:00
21d93d0d8c
[Attention][Model] FP8 KV cache for Triton DiffKV and MiMo-V2.6-Flash ( #58128 )
...
Signed-off-by: Luca Motz <luca.motz@icloud.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-10-05 21:18:29 -07:00
Tianyu Guo
30ab428235
[Bugfix][Attention] Restrict FlashInfer sparse MLA FULL graphs to decode ( #59751 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
2026-10-06 04:13:23 +00:00
lzhan011 and Flora Feng
a3e0243b1c
[Bugfix][Parser] Mistral pre-v11: don't fail requests on unexpected tool call JSON ( #54844 )
...
Signed-off-by: lzhan011 <35493221+lzhan011@users.noreply.github.com >
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-06 03:37:30 +00:00
Tianyu Guo and OpenAI Codex
8c417a2dae
[Bugfix][Model] Fix Qwen3-Omni interleaved M-RoPE boundary positions ( #59842 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-10-06 03:35:53 +00:00
Tianyu Guo
48eb44b719
[Bugfix][KVConnector][NIXL] Fix TP handshake test after per-region FA desc flags ( #60157 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
2026-10-06 03:19:41 +00:00
6fa67afed9
[Bugfix][Core] Retire stale Mamba spec blocks in prefill checkpoint steps ( #59759 )
...
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-10-06 11:01:19 +08:00
2b32d9ac6a
[Bugfix][Core] Drop stale block hashes when a session truncates tokens ( #59103 )
...
Signed-off-by: David Cheung <d7cheung@gmail.com >
Co-authored-by: Brian Woods <me@brian.biz >
Co-authored-by: Change72 <changg@nvidia.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-10-06 02:51:03 +00:00
9465e1c783
[Bugfix][NIXL] Recover locally invalidated pull peer metadata ( #55471 )
...
Signed-off-by: xijiade.aihemaiti <3146335281@qq.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Yannik Hinteregger <37209495+YannikHinteregger@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-05 18:46:53 -07:00
lzhan011 and Flora Feng
9bf4e51030
[Bugfix][Frontend] Malformed $defs in a tool schema should be a 400, not a 500 ( #54850 )
...
Signed-off-by: lzhan011 <35493221+lzhan011@users.noreply.github.com >
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-06 01:37:20 +00:00
0e8f8ecba8
[Bugfix][KVConnector][NIXL] Return NumPy descriptors from mixed-memory local registration ( #60108 )
...
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-10-06 01:23:20 +00:00
4e198cef83
[TEST][XPU][CI] Fix ci uva kernel test instability ( #60041 )
...
Signed-off-by: Your Name <you@example.com >
Co-authored-by: Your Name <you@example.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-10-06 09:19:01 +08:00
97bd8c74eb
[ROCm][DSv4.1][Perf] One-pass dequantize and a gather-sized grid for the compressed K cache ( #56720 )
...
Signed-off-by: JohnQinAMD <yanyuan.qin@amd.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-05 17:57:46 -07:00
b2863c0de9
[Kimi-K3] Use the fused K/V pack kernel on the decode-context-parallel prefill path ( #56881 )
...
Signed-off-by: Brandon Zhang <31413216+brandonfzhang@users.noreply.github.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-10-06 00:13:55 +00:00
Mai Hoàng Đức and Flora Feng
5cd8a2626b
[Bugfix][Structured Output] Cap JSON schema nesting to prevent 500s and API-server stalls ( #60036 )
...
Signed-off-by: Mazukiri <maihoangduc04@gmail.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-06 00:06:59 +00:00
Mason and Yongye Zhu
6dd813e0b7
[Bugfix][MoE] Allow MiniMax2 routing in TRTLLM BF16 monolithic backend ( #59571 )
...
Signed-off-by: Mason Pitt <bradley.b.pitt@gmail.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
2026-10-05 16:24:14 -07:00
0eac152707
[Bugfix] Route every speculation-capable row through the speculative path so recurrent state stays consistent ( #56531 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
Co-authored-by: Cursor Agent <cursoragent@cursor.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-06 03:03:44 +04:00
Ting SUN and Flora Feng
4a705c7b54
[Bugfix][Frontend] Preserve logprobs when top_logprobs is null ( #47838 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-05 22:43:31 +00:00
Kevin H. Luu and Claude Opus 5.5
89a00995c9
[CI] Read each profiler round's trace as it stops in test_gpu_profiler ( #59769 )
...
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-05 15:25:11 -07:00
RomeJay and Flora Feng
2e06395fd3
[Bugfix][Frontend] Build tool-call grammars from the tools the prompt renders ( #59879 )
...
Signed-off-by: Rongjie Jiang <j1870831863@outlook.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-05 22:01:03 +00:00
2e3154aa18
[HiSparse] Log steady-state max concurrency and expose host-tier utilization gauges ( #58949 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-10-05 14:20:32 -07:00
Matthew Bonanni and Claude Opus 5.5
8542c3eff5
[CI] Fix test_mixed_warmup_gate after #57053 ( #60116 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-05 21:17:00 +00:00
Yida Weng and Flora Feng
18c8a65edd
[BugFix][Frontend] Pass reasoning_ended through render → generate ( #60059 ) ( #60062 )
...
Signed-off-by: Yida Weng <84164537+YidaWeng@users.noreply.github.com >
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-05 20:39:25 +00:00
Wentao Ye
74c5cbcd7b
[Bugfix][MRV2][Spec Decode] Honor dynamic K in autoregressive speculators, 1.2~1.3x kernel perf improvement ( #57053 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-10-05 14:54:11 -04:00
