cc2e4fecf2
[ROCm] [fusion] support ROCm aiter fusion of per tensor quant + rmsnorm ( #52831 )
...
Signed-off-by: Andy Friedrich <afriedri@amd.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-16 15:19:07 -05:00
Jack Hu and Douglas Lehr
ef38e8d006
[Model][Hardware][AMD][Kernel] Part 2/2 -> Enable e2e QK Norm + RoPE + KV Cache runtime fusion for Qwen3-30B-A3B on ROCM_ATTN ( #42755 )
...
Signed-off-by: Jack Hu <Jack.Hu@amd.com >
Signed-off-by: Jack Hu <jingxujaster@hotmail.com >
Signed-off-by: Jack Hu <jack.hu@amd.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
2026-10-15 17:10:14 -07:00
Mikko Tukiainen
abad32a718
[ROCm][Perf] Fix Gemma AR+RMS prefill fallback ( #58547 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Signed-off-by: Mikko Tukiainen <mikko.tukiainen@amd.com >
2026-10-12 19:53:08 +00:00
Laith Sakka and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
12e6a585ae
[Fix][Compile] Declare group_shape in the static FP8 quant fusion … ( #61108 )
...
Signed-off-by: Laith Sakka <lsakka@meta.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-13 01:09:57 +08:00
Woosuk Kwon
194bd72be7
[Compile] Remove torch.compile-based sequence parallelism and AsyncTP ( #60517 )
2026-10-08 08:28:23 -07:00
eky-amd and Claude Fable 5
29f955cce0
[ROCm] Fuse MLA dual RMSNorm + FP8 group quant for DeepSeek-R1 ( #54857 )
...
Signed-off-by: Ethan Ky <ethan.ky@amd.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-10-06 11:26:38 -05:00
Laith Sakka
b39f760cc7
[torch.compile] Canonicalize functionalized split slices for fusion pa… ( #57299 )
...
Signed-off-by: Laith Sakka <lsakka@meta.com >
2026-09-28 20:21:59 -04:00
b715699df4
[ROCm][Perf] Extend QK-norm/RoPE/KV-cache fusion to MRoPE ( #50212 )
...
Signed-off-by: Vorapol Assavasangthong <Vorapol.Assavasangthong@amd.com >
Co-authored-by: Santosh Hiremath <Santosh.Hiremath@amd.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
2026-09-23 11:11:57 -05:00
Andreas Karatzas and OpenAI Codex
26879f3260
[ROCm][Compile] Fuse AITER static FP8 attention output ( #58099 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-09-23 03:16:52 -05:00
Andreas Karatzas
0eb42dbbfc
[ROCm][Compile] Support BF16 AsyncTP fusion ( #58098 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-09-22 20:36:38 -05:00
Mikko Tukiainen and Cursor
68363d64ec
[ROCm][Perf] Avoid extra reshape kernel in Qwen GDN output norm ( #47842 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-22 14:38:16 +08:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
d43bb2f37f
[3/N] HiSparse: host-resident sparse-MLA decode hot-buffering ( #53781 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Signed-off-by: Matthew Bonanni <matthew@neuralmagic.com >
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-09-12 00:36:47 -04:00
Qidong Su and OpenAI Codex
9e137d8d23
[Kernel] Enable optimized FlashInfer add-RMSNorm NVFP4 fusion ( #51925 )
...
Signed-off-by: Qidong Su <qidongs@nvidia.com >
Signed-off-by: Qidong Su <soodoshll@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-08 17:20:32 -07:00
Wentao Ye
6b15bea080
[Refactor] Remove utils dead code ( #53941 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-08 11:22:09 -04:00
Jee Jee Li
199cb9b964
[Kernel] Remove unused fake implementation ( #55535 )
...
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
2026-09-06 17:54:32 -07:00
Jee Jee Li
88e1b11313
[Perf] Tune FlashInfer all-reduce thresholds for single-node TP8 on SM103 ( #53606 )
...
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
2026-08-26 01:46:56 +00:00
Khushali Desai
23ab0cfdbc
speculative decoding under tensor parallelism (TP>1) , workspace creation select max hidden dim of target and draft model ( #52193 )
...
Signed-off-by: khushali9 <khushali.desai9@gmail.com >
2026-08-24 21:33:36 +00:00
Lucas Wilkinson
8bdc70ec7b
[6/N][KV-Cache Layout Refactor] Standardize KV cache layout ( #51718 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
2026-08-21 17:47:49 -07:00
shanjiaz
e078a2238a
[Bugfix] Widen flashinfer.comm import guard so a failed import doesn't abort engine startup ( #52241 )
...
Signed-off-by: shanjiaz <zsjwpianpian@gmail.com >
2026-08-14 16:27:24 +00:00
Andreas Karatzas
7e724dca89
[ROCm] Fix AITER all-reduce fusion coverage ( #50802 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-08-06 14:52:23 +08:00
skysnow2001 and Douglas Lehr
f43e1d26e3
[ROCm] Enable AITER and FP8 inference on GFX120x ( #43615 )
...
Signed-off-by: skysnow2001 <skysnow9285@gmail.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
2026-08-03 16:45:37 -05:00
Wentao Ye
0cf49a5d15
[Refactor] Remove multiple dead codes ( #50285 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-08-03 10:59:08 -04:00
Richard Zou and Codex
c27b080d26
[compile] Fix fake kernel return dtype ( #50444 )
...
Signed-off-by: Richard Zou <zou3519@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-07-30 19:50:02 +00:00
BadrBasowid
59e831c09a
[Compilation]Fuse Transformers Residual Add + RMSNorm ( #48757 )
...
Signed-off-by: BadrBasowid <badr.basowid@gmail.com >
2026-07-30 13:46:36 +00:00
5fa0154448
[Rubin] Enable NVLink all-reduce paths on SM107 ( #49647 )
...
Signed-off-by: Zachary Aristei <zaristei@nvidia.com >
Co-authored-by: Sahithi Chigurupati <chigurupati.sahithi@gmail.com >
Co-authored-by: hlu1 <14827759+hlu1@users.noreply.github.com >
2026-07-29 20:53:25 +00:00
afriedri
f472ab0a4c
Remove triton per group quant [ROCm] [Bugfix] ( #49621 )
...
Signed-off-by: Andy Friedrich <afriedri@amd.com >
2026-07-28 05:53:46 +00:00
Chaojun Zhang and Kunshang Ji
bf4f633b4c
[XPU] Enable QK Norm + RoPE fusion pass on XPU ( #49394 )
...
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-07-27 03:22:29 +00:00
Schwinn Saereesitthipitak
b5b61c622c
[Core][Distributed] Add process-checkpoint lifecycle hooks for communicators (starting with Flashinfer) ( #46877 )
...
Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com >
2026-07-26 14:47:50 -04:00
Woosuk Kwon
213f681f81
Revert "[Perf][GLM-5.2] Blackwell decode optimizations" ( #49768 )
2026-07-24 16:04:59 -07:00
dd72658e7d
[Perf][GLM-5.2] Blackwell decode optimizations ( #48597 )
...
Signed-off-by: Peiyuan Zhou <peiyuanzhou1994@gmail.com >
Signed-off-by: zhou <zhou@zhoudeMacBook-Pro.local >
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: zhou <zhou@zhoudeMacBook-Pro.local >
Co-authored-by: stefankoncarevic <Stefan.Koncarevic@amd.com >
2026-07-23 21:36:27 -07:00
stefankoncarevic and Andreas Karatzas
387189c429
[ROCm] Remove redundant AITER fused_qk_rmsnorm probe (avoids config-time HIP init) ( #47992 )
...
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-07-21 22:58:14 -05:00
Jack Hu and Douglas Lehr
2cab53ddee
[Model][Hardware][AMD]: Part 1/2 -> Enable e2e QK Norm + RoPE + KV Cache runtime fusion for Qwen3-30B-A3B on ROCM_AITER_FA, and ROCM_AITER_UNIFIED_ATTN ( #42749 )
...
Signed-off-by: Jack Hu <Jack.Hu@amd.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
2026-07-16 17:39:04 -05:00
Wentao Ye
626c90b2d5
[Refactor] Move fla to third party ( #48500 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-07-16 19:22:36 +01:00
aoshen02 and OpenAI Codex
3b39fd284a
[Bugfix][Spec Decode] Support heterogeneous QK fusion geometry ( #48671 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-07-14 22:37:10 -07:00
Mohammad Miadh Angkad
75fe92a316
[Distributed][Perf] Enable FlashInfer MNNVL allreduce RMS quant fusion ( #48064 )
...
Signed-off-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com >
2026-07-13 15:02:59 +08:00
Hugo Centeno
5f8e73cb8b
[Bugfix] Guard mixed-dtype allreduce RMSNorm quant fusions ( #48330 )
...
Signed-off-by: hcenteno <hugo.centeno@estudiantat.upc.edu >
2026-07-12 09:39:27 +00:00
429f405748
[Bugfix] Guard CUDA-only rms_norm_per_block_quant in FUSED_OPS for non-CUDA builds ( #47296 )
...
Signed-off-by: Tsvika Shapira <tsvika@moonmath.ai >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-09 10:10:53 -04:00
jesco and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
598d51153a
[Bugfix][Distributed] Delegate MNNVL allreduce one-shot selection ( #47589 )
...
Signed-off-by: jesco-absolut <team@srswti.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-06 07:47:06 -07:00
BadrBasowid
740f379fae
[ROCm][AITER] Directly Implement AITER Custom All-reduce in CudaCommunicator ( #46065 )
...
Signed-off-by: BadrBasowid <badr.basowid@gmail.com >
2026-07-06 12:16:32 +00:00
xaguilar-amd
09663abde0
[ROCm][MLA] Fuse MLA q/kv RMSNorm + FP8 per-token quant in the FP8 attention path ( #44977 )
...
Signed-off-by: Xavier Aguilar <xavier.aguilarfruto@amd.com >
Signed-off-by: Xavier Aguilar <Xavier.AguilarFruto@amd.com >
2026-07-02 13:00:41 +08:00
Yejing Lai and Kunshang Ji
f651a8a9a4
[XPU][UT]Enable ut qk_norm_rope_fusion ( #42486 )
...
Signed-off-by: Lai, Yejing <yejing.lai@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-07-01 07:38:03 +00:00
Jeff (Junze) Ma and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
fc61c6fc26
[Perf] Enable + tune FlashInfer fused allreduce at world_size=16 on SM 10.3 (GB300) ( #46392 )
...
Signed-off-by: Jeff Ma <jeffjma@umich.edu >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-24 23:04:17 -07:00
Mohammad Miadh Angkad
d1a38c2762
[Kernel][Performance] Add FlashInfer cutedsl NVFP4 GEMM backend ( #42235 )
...
Signed-off-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com >
2026-06-22 16:17:18 -04:00
Ace Eldeib
b4092176b9
[Bugfix] Complete one-shot fused all-reduce PDL at end to avoid NaN ( #45448 )
2026-06-18 00:54:39 +00:00
Thomas Parnell
ebf3a6d705
[Bugfix] Fix trtllm fused allreduce+rms_norm for transformers backend ( #45307 )
...
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com >
2026-06-16 08:34:27 +00:00
Chaojun Zhang
2725c84aae
[XPU] Enable sequence parallel support for XPU ( #38608 )
...
Signed-off-by: chaojun-zhang <chaojun.zhang@intel.com >
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com >
Signed-off-by: Chaojun,Zhang <chaojun.zhang@intel.com >
2026-06-14 19:26:46 -07:00
Markus Hartikainen and Frida Andersson
80e2c4462d
[ROCm][Compile] Fuse AR + RMSNorm + per-group FP8 quant (+ DSv3.2 indexer fan-out) ( #42864 )
...
Signed-off-by: Markus Hartikainen <markus.hartikainen@amd.com >
Co-authored-by: Frida Andersson <fanderss@amd.com >
2026-06-09 12:06:56 +00:00
Vadim Gimpelson
67d3792d99
[Bugfix] Fix Qwen3.5-FP8 nightly fail. Guard fused_add_rms_norm input/weight dtype mismatch in RMSNorm + quant fusion ( #44694 )
2026-06-06 08:46:14 -04:00