181 Commits
Author SHA1 Message Date
Jack HuandDouglas Lehr ef38e8d006 [Model][Hardware][AMD][Kernel] Part 2/2 -> Enable e2e QK Norm + RoPE + KV Cache runtime fusion for Qwen3-30B-A3B on ROCM_ATTN (#42755)
Signed-off-by: Jack Hu <Jack.Hu@amd.com>
Signed-off-by: Jack Hu <jingxujaster@hotmail.com>
Signed-off-by: Jack Hu <jack.hu@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
2026-10-15 17:10:14 -07:00
Yongye ZhuandClaude Opus 5.5 a6009ea85b [Kernel][Build] Bump DeepGEMM to the 26/09/30 release and adapt the sparse indexer (#61149)
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-12 23:24:43 -07:00
Sriram KumarDouglas Lehrmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2ce8012fa0 [ROCm][Perf] Skip cleaning sparse prefill MQA logits (#51314)
Signed-off-by: Sriram Kumar <sriramkumar.kishorekumar@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-13 00:21:52 -05:00
Hexiang WangOpenAI CodexClaude CodeFangzhou Aimergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Simon Danielsson
c6566bb9f6 [ROCm][MLA] Keep DCP prefill context in FP8 through AllGather (#59070)
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Simon Danielsson <70206058+simondanielsson@users.noreply.github.com>
2026-10-13 09:20:38 +08:00
VenkyOpenAI GPT-6 AstraDevin <158243242+devin-ai-integration[bot]@users.noreply.github.com>Claude Opus 5.5
98f7211162 [Attention][Spec Decode] Multi-query Triton split-K for spec-decode verify, with empty-segment/padding hardening (#56148)
Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com>
Signed-off-by: Venky <23023424+venkywonka@users.noreply.github.com>
Co-authored-by: OpenAI GPT-6 Astra <noreply@openai.com>
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 04:37:56 +04:00
Giulio De Pasquale f1a3c60558 [Feature][Attention] Cover a whole spec-verify in one DiffKV split-KV program (#59085)
Signed-off-by: Giulio De Pasquale <git@depasquale.giugl.io>
2026-10-10 11:55:45 +08:00
Sriram Kumar 560fae64cd [ROCm][Bugfix] Pass per-sequence context lengths to the AITER paged MQA-logits kernels (#59420)
Signed-off-by: Sriram Kumar <sriramkumar.kishorekumar@amd.com>
2026-10-09 08:16:20 +00:00
Andy Lo ba77c4c130 [Bugfix][Attention] Preserve small FP8 softmax weights in Triton attention (#60156)
Signed-off-by: Andy Lo <andy@mistral.ai>
2026-10-08 12:35:22 +02:00
4ac0179f8a [ROCm][Perf] Use native merge_attn_states kernel on gfx942/gfx950 (#60159)
Signed-off-by: mghanimi <mghanimi@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
2026-10-08 04:49:49 +00:00
Simon DanielssonandDouglas Lehr 87d9996abe [ROCm][Perf][GLM-5.3-Flash] BF16 splitk kernel for sparse MLA decode (#58584)
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
2026-10-07 21:37:51 -05:00
aditi-amd 385d86a6cf [Attention] Add UltraQuant 4-bit KV cache backend (FlyDSL D=256) (#57057)
Signed-off-by: Aditi Rana <aditi.rana@amd.com>
2026-10-06 20:42:52 +00:00
Luciano MartinsandLuciano Martins 02b83919aa [Model] Support EmbeddingGemma2 multimodal pooling architecture (#60254)
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
2026-10-06 13:43:38 -04:00
Giancarlo Delfin fcf53d240c [Spec Decode] Remove eager metadata rebuild during MTP fused multi-step decode (#58463)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
2026-10-06 12:17:30 -04:00
edfccc9469 [ROCM][Perf] Add ROCM_SEGMENTED_ATTN attention-backend works across RDNA3/3.5/4 (#59132)
Signed-off-by: root <jeffaw99@hotmail.com>
Signed-off-by: big-yellow-duck <jeffaw99@hotmail.com>
Co-authored-by: Tan Pin Siang <1716735+tanpinsiang@users.noreply.github.com>
Co-authored-by: vllmellm <190700713+vllmellm@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-06 07:53:47 +00:00
21d93d0d8c [Attention][Model] FP8 KV cache for Triton DiffKV and MiMo-V2.6-Flash (#58128)
Signed-off-by: Luca Motz <luca.motz@icloud.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
2026-10-05 21:18:29 -07:00
Gawr Gura 0721Gawr Gura 0721mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Wentao Ye
e8a53a55e6 [Bugfix][Attention] Support int4_per_token_head on non-power-of-two head sizes (#56198)
Signed-off-by: Gawr Gura 0721 <LiRunGuo@users.noreply.github.com>
Co-authored-by: Gawr Gura 0721 <LiRunGuo@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-10-05 04:52:39 +00:00
Simon Danielssonandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 98b29aa99a [ROCm][Perf][GLM-5.3-Flash] Add AITER topk backend for decodes (#58167)
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-02 11:47:41 +01:00
Simon Danielssonandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 751fa96a49 [ROCm][Perf][GLM-5.3-Flash] Remove redundant copy after ragged sparse MLA (#58569)
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-02 11:45:51 +01:00
Qiu ChunshuoandLucas Wilkinson 8486f1ebfe [Perf][PCP] Shard decode requests across PCP ranks (#52162)
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
2026-10-01 23:50:13 -04:00
Fabian Joswigandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 1117140edb [Bugfix][Model] Fix DiffusionGemma silently freezing attention mask under CUDA graph replay (#51994)
Signed-off-by: Fabian Joswig <fjosw@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-30 05:34:26 +00:00
cd947f5c62 [MM] Fix compiled ViT attention output layouts (#58182)
Signed-off-by: Jakub Zakrzewski <jzakrzewski@nvidia.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
2026-09-29 06:27:03 -07:00
63b9931b73 [ROCm][DSv4.1] Paged MXFP4 sparse indexer on aiter's MQA-logits kernel (#58671)
Signed-off-by: Mehmet Cagri Kaymak <mehmet.kaymak@amd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>
2026-09-29 14:37:57 +08:00
Mehmet Cagri 70ae0b7435 [ROCm][MLA] Enable sparse MLA Gluon kernel from Aiter (#53492)
Signed-off-by: Mehmet Cagri Kaymak <mehmet.kaymak@amd.com>
Signed-off-by: Mehmet Cagri <mehmet.kaymak@amd.com>
2026-09-29 09:08:18 +08:00
Mehmet CagriandClaude Opus 5.5 75fad5bbef [Bugfix][ROCm] Drop -1 sentinels when building the ragged sparse-MLA indices (#58058)
Signed-off-by: Mehmet Cagri Kaymak <mehmet.kaymak@amd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 18:40:11 -05:00
Isotr0py 522101f6ed [Multimodal] Avoid extra d2d for encoder cudagraph with fused input norm (#56711)
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
2026-09-28 21:04:01 +08:00
a4eb3f25d6 [Spec Decode] Enable Gemma4 DSpark adaptive verification with FlashInfer (#57263)
Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-26 17:01:17 +08:00
Fangzhou AiandClaude Opus 5.5 3059155b47 [ROCm][Perf] MXFP8 GEMM on native 32x32 block scales for gfx950 (#58510)
Signed-off-by: fai <fangzhouai@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 21:26:30 -05:00
Fangzhou AiandClaude Opus 5.5 721d0e5c11 [ROCm][DSv4.1][Perf] Emit MXFP8 from the sparse decode reduce and run wo_a as a grouped FP8 GEMM (#58456)
Signed-off-by: fai <fangzhouai@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 11:58:14 +00:00
5963795ec7 [Attention][CPU] Run Zen CPU encoder attention on zentorch SDPA (#54508)
Signed-off-by: priyansh jain <priyansh.jain2@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-24 02:05:44 -07:00
Lucas WilkinsonQiuChunshuoClaude Fable 5.1CodexRobert ShawNick Hillmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
e6dc16cebd [PCP] Support prefill context parallelism with data parallelism (#57075)
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: QiuChunshuo <qiuchunshuo@huawei.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-23 22:32:36 -04:00
1904ed9402 [Perf][ROCm][Attention] Narrow the Triton prefill-attention KV tile on RDNA3/RDNA4 (#58225)
Signed-off-by: Jipeng Li <jipengli@amd.com>
Co-authored-by: GitHub Copilot CLI <noreply@github.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-23 08:39:27 +00:00
Fangzhou AiCursormergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
ca831d1c55 [ROCm][DSv4.1][Perf] Fuse the inverse RoPE into the sparse decode reduce (#57435)
Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-22 18:05:11 +00:00
066ee91197 [Bugfix][Attention] Avoid NaN in the Triton softcap for large attention logits (#56579)
Signed-off-by: Kushal Dabbe <72650064+kushaldabbe@users.noreply.github.com>
Co-authored-by: opencode <noreply@opencode.ai>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-09-22 12:11:05 -04:00
frida-andersson 71fc70d3ae [ROCm][DSv4] Fix sparse-indexer logits collapse on gfx950/gfx942 (#50455)
Signed-off-by: Frida Andersson <fanderss@amd.com>
Signed-off-by: frida-andersson <fanderss@amd.com>
2026-09-18 16:57:44 +00:00
Sriram KumarandDouglas Lehr ff6b5808c4 [ROCm] Resolve the indexer fp8 cache dtype once at import (#53792)
Signed-off-by: Sriram Kumar <sriramkumar.kishorekumar@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
2026-09-17 17:51:35 +00:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
Wentao Yeandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 18f8aa0465 [Refactor] Remove dead kernel code (#56845)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-15 13:17:53 -07:00
Fangzhou Ai dffbb714e4 [ROCm][Perf] Optimize DSV4.1 K=512 decode top-k on gfx950 (#56743)
Signed-off-by: fai <fangzhouai@gmail.com>
2026-09-15 18:34:24 +00:00
241e9391ed [Bugfix][Attention] Stabilize sparse-MLA DCP for GLM PCP evals (#55879)
Signed-off-by: khluu <khluu000@gmail.com>
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Jason Yao <wsyjh8@gmail.com>
2026-09-15 09:58:30 -07:00
Fangzhou AiandCursor dabc4362b4 [ROCm][DSV4.1][Perf] Stride the DSA decode candidate mask over the live context (#56628)
Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-14 20:47:32 +00:00
Woosuk Kwon 13e221f830 [Perf] Fuse DSV4.1 input metadata preparation with Triton (#56562) 2026-09-12 07:20:04 -07:00
Roberto L. CastroandCodex 120ec4ebd2 [5/N][warmup][DSv4] Migrate NVIDIA CuTeDSL attention kernels (#53566)
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com>
Co-authored-by: Codex <codex@openai.com>
2026-09-12 00:05:59 -04:00
amd-xavierwang 2d75e586fc [ROCm][Kernel][DSV4] Remove tl.constexpr to avoid cold-compile churn in indexer gather kernel (#56153)
Signed-off-by: xawang <xawang@amd.com>
2026-09-11 21:34:17 +00:00
e77daef89e [Model] Support DeepSeek-V4.1-Flash (#56214)
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Signed-off-by: andyluo7 <andy.luo@amd.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 02:11:19 -07:00
Rita Brugarolasandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 6ee5bb0a0b [DCP][Kernel][Perf] Fuse the empty-shard LSE mask into the A2A pack kernel (#54889)
Signed-off-by: Rita Brugarolas Brufau <rita.brugarolasbrufau@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-10 11:43:00 -04:00
jiacao-amdOpenAI CodexClaudemergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
7e91760650 [ROCm][Perf] Route large DSV4 sparse prefill to AITER OPUS (#54855)
Signed-off-by: jiacao-amd <jiahui.cao@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-10 14:39:12 +08:00
Shanshan Shen 25047604fe [ROCm][Perf] Remove AITER paged-MQA outputs guard for DeepSeek-V4 (#55808)
Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com>
2026-09-08 06:58:26 -05:00
Roberto L. CastroCodexOpenAI CodexLucas Wilkinsonmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
1713b9866a [2/N][warmup][DSv4] Migrate sequence and DCP kernels (#53564)
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-07 09:35:54 -04:00
Andreas KaratzasandOpenAI Codex e476556189 [ROCm][CI] Add attention-sink support to ROCm AITER sparse MLA (#54404)
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-07 09:26:47 +00:00