Commit Graph
22018 Commits
Author SHA1 Message Date
MOVZXandHinata 835c44af15 [Qwen4Exp] Reinterpret FP8 E4M3 PLE as int8 for SM86 pinned-host lookup (PR #58856)
The pinned-host PLE lookup Triton kernel only gathers and stores bytes,
but passing an fp8e4nv tensor to it fails on architectures without
native FP8 pointer support (e.g. SM86) with an error saying type
fp8e4nv is not supported (supported fp8 dtypes are fp8e4b15, fp8e5).

Since FP8 E4M3 and int8 both occupy one byte per element, reinterpret
the PLE weight view and the output tensor as int8 when the weight dtype
is float8_e4m3fn; the non-FP8 path is unchanged.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 09:54:05 +07:00
MOVZXandHinata 758cc4f9e1 [Qwen3.5] Pass quant_config and prefix to embed_tokens in Qwen3_5Model
Apply the qwen3_5_quantized_embedding patch: construct the embedding
through the model's quant config so quantized (e.g. fp8/int8)
embeddings are supported, and give it a stable prefix for weight
loading.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 09:53:21 +07:00
Yongye ZhuandClaude Opus 5.5 77871126f9 [Kernel][DSV4.1] Fuse MoE finalize into the TP all-reduce + mHC boundary (#58586)
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-27 03:08:36 +08:00
Cyrus Leung 379e9a1ea8 [Security] Harden message sanitization (#58832)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-09-26 16:28:32 +00:00
Kevin H. Luu 7d8c5fe9a9 [Bugfix][DSV4.1] Avoid host sync in ViT CUDA graph replay metadata (#58499) 2026-09-26 11:37:36 -04:00
Andreas Karatzas 0239bd2549 [CI] Stabilize batch submission in full CUDA graph tests (#58810) 2026-09-26 11:35:56 -04:00
Wentao Ye 3576691c42 [GLM5.3 Bug] Fix sparse indexer attn topk backend selection (#58594)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-09-26 11:35:28 -04:00
Misha Goin 79ac28c5bd [CI] Reduce CUDA graph mode test overhead (#58749) 2026-09-26 11:22:33 -04:00
Ashraf Bhuiyan 5113cf9d16 [Mypy] Fix mypy typing for Qwen and Qianfan models (#58046)
Signed-off-by: Ashraf Bhuiyan <mbhuiyan@redhat.com>
2026-09-26 14:31:51 +00:00
Robert ShawandRobert Shaw dfab504333 [Bugfix][Frontend] Detect Anthropic inline-system merge against the resolved chat template (#58754)
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
2026-09-26 14:00:56 +00:00
Robert ShawandRobert Shaw d64f277c54 [Bugfix] Fix Anthropic Thinking Disabled with P/D (#58786)
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
2026-09-26 13:50:57 +00:00
Willian Zandmgoin ad6817b68d [Perf][MoE] Index expert mapping lookups in RoutedExperts.load_weights (#58720)
Signed-off-by: Willian <willian@willian.email>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
2026-09-26 06:48:24 -07:00
Wentao Ye bf93e1f92c [Refactor] Remove dead tests utils (#58803)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-09-26 09:16:53 -04:00
Itay AlroyandTyler Michael Smith 2bb7605ae1 [Elastic EP] Fix EPLB load statistics during scaling (#58473)
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
2026-09-26 12:05:37 +00:00
Juan Pérez de AlgabaandCursor 31f2e70cd3 [Security] Gate per-request multimodal processor kwargs (#58830)
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-26 11:10:35 +00:00
2f41e00235 [CI] Split (B200) Miscellaneous Kernels into mHC, FLA Ops and Misc named jobs (#58609)
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-09-26 04:10:14 -07:00
a4eb3f25d6 [Spec Decode] Enable Gemma4 DSpark adaptive verification with FlashInfer (#57263)
Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-26 17:01:17 +08:00
Aarushi JainandAndreas Karatzas 3b4566c5cf [CI] Only isolate the registry tests that need a fresh process (#58764)
Signed-off-by: aarushjain29 <Aarushi.Jain2@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-26 14:03:45 +08:00
ddd6fbca14 [Bugfix][Frontend][Rust Frontend] Update DeepSeek V4.1 Flash reasoning effort mappings (#58316)
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: zhec <chengyunfei@ruc.edu.cn>
2026-09-26 02:18:58 +00:00
xiao-llmandCursor 5840d95284 [Bugfix][ROCm] AMD-Quark mixed-precision DeepSeek-V4.1 support (#57071)
Signed-off-by: Xiao Yu <xiao.yu.dc@outlook.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-25 20:51:18 -05:00
Ke Wenmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>aoshen02
b6d2b4fc25 [RL] Add sharding-aware NCCL M2N weight transfer (#51520)
Signed-off-by: Ke Wen <kwen@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-09-26 01:38:53 +00:00
Shuolei Wang 4bb804cc5b [Perf][DSv4.1] Shard the Engram wkv projection across TP ranks (#58678)
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com>
2026-09-26 01:15:57 +00:00
Chandra MouryaandVS Chandra Mourya 8a2364605c [Core] Bound draft-token RPC waits by the execute-model timeout (#58779)
Signed-off-by: VS Chandra Mourya <219748331+vschandramourya@users.noreply.github.com>
Co-authored-by: VS Chandra Mourya <219748331+vschandramourya@users.noreply.github.com>
2026-09-26 00:40:22 +00:00
Baljinder HothiandFlora Feng fdcce47e9b [Bugfix][Frontend] Fix Inkling tool name leaking into content after reasoning (#58792)
Signed-off-by: Baljinder Hothi <baljinder.hothi@cohere.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-09-26 00:38:52 +00:00
da1aaec31f [CI] Split (H200 MIG/MI300) Basic Correctness into named jobs (#57054)
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-25 17:05:33 -07:00
Elvir CrnčevićandFlora Feng 9bf44c4013 [Bugfix][Reasoning] Count Kimi K3 reasoning tokens (#58372)
Signed-off-by: Elvir Crncevic <elvircrn@gmail.com>
Signed-off-by: Flora Feng <4florafeng@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-09-25 23:29:56 +00:00
Rui ZhuandClaude 974cb65155 [Bugfix] V1: fix allowed_token_ids_mask aliasing in InputBatch.swap_states (#48419)
Signed-off-by: Rui Zhu <rui.zhu.rz399@yale.edu>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-25 23:23:11 +00:00
Ricardo-M-LandClaude Opus 4.6 b040321b33 fix: perf: use startswith(x, i) instead of string slicing to avoid O(N^2) (#52580)
Signed-off-by: Ricardo-M-L <ricardoporsche001@icloud.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-09-25 23:22:13 +00:00
Taneem IbrahimandWentao Ye 6ec2f84e1a [Pooling] Preserve BERT-family heads for raw logits (#57664)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-09-25 19:10:14 -04:00
Linkun 614b55f0b6 [AuxOutput] Only require Model Runner V2 on GPU platform (#58205)
Signed-off-by: Linkun Chen <github@lkchen.net>
2026-09-25 15:55:21 -07:00
Oxana Korzh e55d076f89 [Bugfix] Don't drop the rest of the allocator config when toggling expandable segments (#57982)
Signed-off-by: Oxana Korzh <okorzh@amd.com>
2026-09-25 17:34:19 -05:00
Raphaël RiallandSimon VeitnerOpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
4ccfe12398 [watermarking] golden tests for backwards compatibility (#56809)
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-25 22:01:48 +00:00
yzong-rhandBen Browning b6761e8ded [Bugfix] Default missing detail for Responses API input images (#57241)
Signed-off-by: Yifan Zong <yzong@redhat.com>
Co-authored-by: Ben Browning <56071+bbrowning@users.noreply.github.com>
2026-09-25 14:08:55 -07:00
8b47e8b22c [Perf][Frontend] Defer reasoning usage recounts for non-continuous chat streams (#56067)
Signed-off-by: Cheng Rui <286040359@qq.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-09-25 14:08:23 -07:00
Nick HillandKimi Code de2d478d9a [Bugfix][Frontend] Document 404 response for /generative_scoring (#58788)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
2026-09-25 14:08:00 -07:00
Denis ErmakovDu BinClaude Opus 5.5Flora Fengmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
7871963fcc [Bugfix][Frontend] Respect max_output_tokens in the Harmony tool-call loop (#58551)
Signed-off-by: errmakov <ide404@gmail.com>
Co-authored-by: Du Bin <8174807+dubin555@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-25 14:00:39 -07:00
Yousafandsfeng33 83db8c8390 [Feature][Frontend] Add granite_thinking_parser reasoning parser for Granite 4.2 (#55957)
Signed-off-by: Yousaf shah <yousaf.shah@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
2026-09-25 20:43:08 +00:00
e440b75bb6 [ROCm] Bump torch 2.13, triton 3.8, torchaudio, torchvision (#50605)
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Signed-off-by: jpvillam <juan.villamizar@amd.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: jpvillam <juan.villamizar@amd.com>
2026-09-25 14:55:52 -05:00
31cc226401 [ROCm][CI] Run the MLA attention+quant fusion test on ROCm (#58717)
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-25 14:45:32 -05:00
stefankoncarevicandAndreas Karatzas 2d7dd0e5c1 [Bugfix][CI] Report subprocess test skips as skips, not passes (#58701)
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-25 14:44:49 -05:00
Taneem Ibrahim 2bc902eb0f [CI] [MRV2] Restore MRV2 pp dp coverage (#57735)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
2026-09-25 19:13:59 +00:00
22bbe3f102 [ROCm][CI] Expand single-GPU coverage on MI355 DPX (#57599)
Signed-off-by: Sheral Kumar <shekumar@amd.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
2026-09-25 13:27:53 -05:00
Matt MastracciandClaude Fable 5.1 353e491efd [Perf] DiffusionGemma: one-pass sampler statistics kernel (#58226)
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-25 14:14:55 -04:00
e3dd5b5a75 [ROCm][CI] Test AMD DeepSeek V4 MoE routing against a PyTorch reference (#58740)
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-25 18:05:54 +00:00
1b922ffcbe [ROCm][CI] Add quantized MoE serving test for gfx950 (#58748)
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-25 18:02:15 +00:00
b21757555a [ROCm][CI] Cover the AITER MQA logits dispatch on gfx950 (#58724)
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-25 17:43:40 +00:00
1417022c3d [ROCm][Kimi-K3] Optimize low-concurrency speculative KDA (#58045)
Signed-off-by: jiacao-amd <jiahui.cao@amd.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-25 12:33:30 -05:00
GokayAIandGokayAI c89b5b4176 [Bugfix][KV Connector] Reap expired NIXL leases behind a heartbeated head (#58292)
Signed-off-by: GokayAI <60583610+gokay-ai@users.noreply.github.com>
Co-authored-by: GokayAI <gokay-ai@users.noreply.github.com>
2026-09-25 17:23:44 +00:00
Wentao Ye c88f4824ff [GLM5.3 Perf] Optimize glm 5.3 metadata op, 1.6~4.8x kernel level performance improvement (#58450)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-09-25 13:15:18 -04:00
Matt MastracciYifan QiaoClaude Opus 5.5Robert Shawmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2617fe9383 [Bugfix][GLM-5.3-Flash] kpool corruption with speculative decoding (#58454)
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-25 12:51:34 -04:00