Commit Graph
2337 Commits
Author SHA1 Message Date
Nick Hillandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> bbee532988 [Perf][1/n] Eliminate various GPU<->CPU syncs (#41429)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-11 20:36:03 +00:00
Jee Jee LiandYongye Zhu 53181384e0 [Bugfix] Fix DSV4 swiglu_limit on marlin backend (#42287)
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
2026-05-11 13:03:56 -07:00
Vensenandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 6fdb49392e [Bugfix] Fix int32 overflow in DeepGEMM SiLU/mul FP8 Triton kernel (#42201)
Signed-off-by: vensen <vensenmu@gmail.com>
Signed-off-by: Vensen <vensenmu@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-05-11 14:52:31 -04:00
Hexiang Wang 7863fff6e5 [ROCm][DSv4] implement flash sparse mla with triton kernels (#41812)
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
2026-05-11 09:27:11 -07:00
Wentao Ye 0d453e2336 [Perf] Batch invariance with Cutlass fp8 support, 28.9% E2E latency improvement (#40408)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-05-11 12:20:58 -04:00
Vadim Gimpelson a2e776d716 [Bugfix] Accept canonicalized modelopt_* quant_method in _extract_modelopt_quant_algo (#42181)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
2026-05-11 11:10:57 -04:00
Wentao Ye 4b64fc2cbf [Refactor] Cleanup batch invariant dead code (#41993)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-05-11 10:48:39 -04:00
Rohan PotdarandElizaWszola a51376b3f0 [Performance][DSR1]: Fused RoPE+KVCache+q_concat for MLA (#40392)
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Signed-off-by: Rohan Potdar <66227218+Rohan138@users.noreply.github.com>
Co-authored-by: ElizaWszola <ewszola@redhat.com>
2026-05-11 14:10:50 +00:00
wangxiyuan 5536fc0c01 [Misc] Replace mamba_type string literals with MambaAttentionBackendEnum (#41188)
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
2026-05-11 03:59:36 +00:00
gnovackandgeorge 171019ab19 add fused mhc_post_pre kernel (#41536)
Signed-off-by: george <george@inferact.ai>
Co-authored-by: george <george@inferact.ai>
2026-05-10 19:56:52 -07:00
1b57eb41f2 [MoE] Move various experts classes to fused_moe/experts/ (#41979)
Signed-off-by: Jackmin801 <ongjackm@gmail.com>
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com>
Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com>
Signed-off-by: Bill Nell <bnell@redhat.com>
Co-authored-by: Jackmin801 <ongjackm@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Robert Shaw <robertgshaw2@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com>
2026-05-11 07:54:33 +08:00
Vensenandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 215e2f7990 [Bugfix][Mamba] IMA in causal_conv1d kernel for long sequences (#41617)
Signed-off-by: vensen <vensenmu@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-10 12:38:28 +00:00
Isotr0py 48698b1b9b [Bugfix] Fuse Qwen3.5 in_qkvz_proj forwarding with LoRA enabled (#37912)
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
2026-05-10 10:59:02 +00:00
Juhi Mittal 7a2b596982 [Quantization] Add ModelOpt NVFP4 W4A16 (4-bit weights, fp16/bf16 activations) support (#41769)
Signed-off-by: Juhi Mittal <juhim@nvidia.com>
2026-05-09 21:15:50 +00:00
Thomas ParnellandJiangyun Zhu 3dda9aeb54 [Bugfix] Remove nested torch.compile in GDN rearrange_mixed_qkv causing CUDA graph capture failure (#42070)
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
2026-05-09 08:30:55 -07:00
Kermit adb6d96516 [Bugfix] Fix GDN KKT precision loss on Hopper GPUs by aligning tl.dot operand layout with WGMMA (#42076)
Signed-off-by: kermit <ckeming@outlook.com>
2026-05-09 13:08:46 +00:00
Jee Jee Li ecd0b60aad [LoRA] Initial EP support for LoRA (#40867)
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
2026-05-09 00:31:23 -07:00
Terrence ZhaoandCursor a2812becd6 [Models] Cohere Eagle + fix to Cohere MoE (#42078)
Signed-off-by: Terrencezzj <terrence@cohere.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-08 21:46:26 -07:00
Markus HartikainenandTJian e8f9038ebd [ROCm][Bugfix] Re-tag AITER MoE weights as preshuffled after replace_parameter (#42061)
Signed-off-by: Markus Hartikainen <markus.hartikainen@amd.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
2026-05-08 21:42:07 -07:00
Michael Goin 8bcd8a260c [Bugfix] Fix FlashInfer CUTLASS MXFP4-MXFP8 MoE by restoring swizzled scale (#42089)
Signed-off-by: mgoin <mgoin64@gmail.com>
2026-05-08 15:59:06 -07:00
2c6b59b807 [ROCm][Perf] Add Fused Shared Expert (FSE) support for Qwen3-Next (#39280)
Signed-off-by: nholmber <nholmber@users.noreply.github.com>
Signed-off-by: Tres Popp <tres.popp@amd.com>
Signed-off-by: Doug Lehr <douglehr@amd.com>
Co-authored-by: nholmber <nholmber@users.noreply.github.com>
Co-authored-by: Tres <tpopp@users.noreply.github.com>
Co-authored-by: Tres Popp <tres.popp@amd.com>
Co-authored-by: Doug Lehr <douglehr@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
2026-05-08 15:38:00 -04:00
Hiroaki Mikami 90f145aaf7 [Models][Gemma3/Gemma4] Support hidden_act variants in gated MLP (#40588)
Signed-off-by: Hiroaki Mikami <hiroaki8270+github@gmail.com>
2026-05-08 11:29:11 -07:00
ed582b6a4c [Aiter][ROCm] gdn_linear_attn kernel fusion (#40711)
Signed-off-by: Tres Popp <tres.popp@amd.com>
Signed-off-by: Chuan Li <chuali@amd.com>
Co-authored-by: hellozhuo <zhuo.su@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-07 23:11:37 -07:00
Stefano CastagnettaandYongye Zhu 54f548e9e5 [Bugfix] Restore moe_forward output shape invariant on TRTLLM MXFP4 path (#41646)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
2026-05-07 15:26:06 -07:00
Kyle Sayers c1819ca283 [Compressed Tensors] Allow configs with non-explicit ignores (#41965)
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
2026-05-07 14:03:45 -07:00
TomerBN-Nvidia 8189a15914 [Core] Replace routing replay with device cache and async D2H pipeline (#39917)
Signed-off-by: Tomer Barnatan <tbarnatan@nvidia.com>
2026-05-07 11:24:56 -07:00
Li, Jiang b3945cc316 [CPU] Bump up to the latest CPU kernels (#41924)
Signed-off-by: jiang1.li <jiang1.li@intel.com>
2026-05-07 05:45:59 -07:00
Mehdi Ghanimifardandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> d4b0048404 Eliminate redundant MoE buffer copies in AITER fused experts (without dependency on AITER changes) (#41713)
Signed-off-by: Mehdi Ghanimifard <mehdi.ghanimifard@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-07 03:46:40 -07:00
Yuwen Zhou 713b28bd0b [CPU] Add FP8 W8A16 MoE support (#41314)
Signed-off-by: yuwenzho <yuwen.zhou@intel.com>
2026-05-06 23:17:07 -07:00
Fadi ArafehandVadim Gimpelson 51f22dcfd0 [Feat][CPU] Enable Gated DeltaNet Attention (Qwen 3.5 / 3.6) (#41025)
Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com>
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com>
2026-05-07 12:57:09 +08:00
Yuankai Chenandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 2e777d21a8 [Bugfix][Rocm]Aiter MoE re-uses existing tensor addresses after weight update. (#40390)
Signed-off-by: Yuankai Chen <yuankach@amd.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-05-06 10:32:26 +00:00
Matthew Bonanni 01b9b5af67 [Attention] Minor refactor: layer takes ownership of the MLA prefill backend (#41744)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
2026-05-05 23:22:41 +00:00
Matej SirovatkaandTyler Michael Smith 48954de237 Fix DeepGEMM ep_scatter output address overflow (#39213)
Signed-off-by: S1ro1 <matej.sirovatka@gmail.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
2026-05-05 18:56:56 +00:00
628c436301 [New Model][ROCm] Add AMD support for DeepSeek V4 (#40871)
Signed-off-by: ganyi <ygan@amd.com>
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Signed-off-by: tjtanaavllm <tunjian.tan@amd.com>
Co-authored-by: ganyi <ygan@amd.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: tjtanaavllm <tunjian.tan@amd.com>
2026-05-05 08:55:37 -07:00
Harry Mellor 84bd8a3c1e Remove unnecessary runtime asserts from linear layers (#41729)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-05-05 14:42:56 +00:00
Luciano MartinsandLuciano Martins 6bb924bbf3 [Model] Fix Gemma4 MoE activation mismatch (#41574)
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
2026-05-05 04:34:11 +00:00
Bowen Bao 1e9500410a [ROCm][Quantization][2/N] Refactor quark_moe w4a8 w/ oracle (#39136)
Signed-off-by: Bowen Bao <bowenbao@amd.com>
2026-05-04 19:50:38 -07:00
Nick Hill 416f9cdede [Perf][2/n] Eliminate GPU<->CPU syncs in pooling code (#41433)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-05-05 02:43:25 +00:00
Chendi.XueandKunshang Ji 685bf811d6 [XPU] enable is_act_and_mul for xpu (#37481)
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-05-05 01:07:39 +00:00
4f2af1a7c0 [Feature] TurboQuant: support hybrid models and uniform quantization (#39931)
Signed-off-by: JartX <sagformas@epdcenter.es>
Signed-off-by: Jim Smith <jhsmith0@me.com>
Co-authored-by: Jim Smith <jhsmith0@me.com>
Co-authored-by: Sandermage <sandermage@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-05-04 20:14:01 -04:00
Andreas Karatzas 1cb0838721 [ROCm][CI] Fix MLA prefill scale for DeepSeek GSM8K (#41569)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
2026-05-04 16:32:55 -07:00
fxmarty-amd 9c07342fdc [NVFP4][fix] Fix layer.weight -> w13 typo in NVFP4 MOE emulation kernel preparation (#41630)
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
2026-05-04 20:13:37 +00:00
Jinzhen Lin 08834cc3ce [Quantization] add humming mxfp4 moe backend (#41083)
Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com>
2026-05-02 18:36:03 -07:00
Luka Govedič d58c42e19c [vLLM IR] 2/N fused_add_rms_norm and maybe_inplace overload (#36823)
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
2026-05-01 23:41:15 -04:00
Yongye Zhu bcf5cac9fb [DSV4] Add knob to enable pre-attn gemm (#41443)
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
2026-05-01 12:23:17 -07:00
Isotr0pyandRoger Wang a9484dac7b [Perf] Intergrate Tile Kernels head_compute_mix_kernel for Deepseek-V4 (#41255)
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
2026-05-01 15:01:17 -04:00
Matthew BonanniMichael GoinLucas Wilkinsonmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f3fef12350 [Attention] Abstract the MLA prefill backends and eliminate cuDNN (#32623)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-01 13:36:20 -04:00
sychen52 947138b6c2 Add nvfp4 kv cache support (#40177)
Signed-off-by: Shiyang Chen <shiychen@nvidia.com>
2026-05-01 04:55:16 +00:00
Juhi Mittalandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 6b6ac6c3c7 [Kernel][MoE] Support GELU on TRT-LLM NvFP4 fused MoE for Gemma4 (#41050)
Signed-off-by: Juhi Mittal <juhim@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-01 03:37:43 +00:00
Stefano Castagnetta b542bdf7fb [Bugfix] Disable FlashInfer CUTLASS MoE on SM110 (Jetson Thor AGX) (#40808)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
2026-04-30 20:08:49 -07:00