106 Commits
Author SHA1 Message Date
siyuandXun Sun c61a1e73c6 [Feature] Skip MLA weight absorption when weights are pre-processed (#57320)
Co-authored-by: Xun Sun <UNIDY2002@outlook.com>
2026-10-17 23:10:04 +08:00
642c18d64c [Perf] Kimi K3 Specialized Decode: CuTeDSL Optimized GB300 (1.5-2x) (#61867)
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-15 22:50:41 -04:00
Anerudhan GopalandOpenAI Codex af28c171f5 [Dependency] Upgrade FlashInfer to 0.7.2 and restore KDA auto backend (#61913)
Signed-off-by: Anerudhan Gopal <agopal@nvidia.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-10-15 17:04:24 -07:00
4d58fa254a [MoE] Unify DeepGEMM MegaMoE experts behind one feature-matrix layer; add NVFP4 (#61538)
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 19:02:01 +00:00
Wei ZhaoandJee Jee Li 4975fda882 [Dependency] Upgrade FlashInfer to 0.7.1 (#60948)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
2026-10-14 12:31:08 -07:00
c5bda658d2 [Model][MoE] Load MegaMoE experts directly into the DeepGEMM layout (#60096)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Aaron Hao <hao-aaron@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Aaron Hao <hao-aaron@users.noreply.github.com>
2026-10-13 11:52:08 -07:00
253e02b95e [Perf] Warm up DeepGEMM MegaMoE kernels before serving (#60468)
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com>
2026-10-13 20:17:48 +08:00
Misha GoinandClaude d79b52b182 [Misc] Consolidate csrc shared headers and prune dead code (#60721)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-12 09:46:22 -07:00
Nick Riasanovsky a686c8e1b7 [Triton] Migrate Triton kernels off tl.make_block_ptr for triton-main compat (#60502)
Signed-off-by: Nick Riasanovsky <njriasan@meta.com>
2026-10-12 11:38:43 -04:00
dingdangmaoandCodex 124b89d5b7 [Bugfix][RecoverSSM] Correct aligned state indices at exact block boundaries (#59962)
Signed-off-by: apex-mochen <2756823972@qq.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-12 02:08:14 +00:00
Warren Dengandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> fc39317392 [Bugfix][Kernel] Remove invalid NC from Kimi-K3 KDA autotune keys (#60446)
Signed-off-by: Warren Deng <warrdeng@meta.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-09 14:25:41 +00:00
Harry MellorandClaude Opus 5.5 2c99ee9333 [Chore] Remove vendored code made redundant by Transformers 5.16.1 (#60623)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:29:03 +00:00
Elvir CrnčevićandMisha Goin 8674a794c7 [Perf] Enable Kimi K3 shared expert sharding with DeepEP v2 (#56875)
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-10-08 16:42:15 +00:00
81198e97ba [Refactor] Abstract DeepGemm MegaMOE backend (#58053)
Signed-off-by: Hui Shi <kalinshi@tencent.com>
Co-authored-by: LuneExplorer <liyihan23@mails.tsinghua.edu.cn>
Co-authored-by: ArmageddonKnight <bojian@cs.toronto.edu>
2026-10-08 05:05:39 +00:00
Hyukjoon LeeClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Summer Yang
1fbbfa1657 [Bugfix][Kimi-K3] Declare max_tp_shards on the DSpark MLA KV cache spec (#59733)
Signed-off-by: hyukjlee <hyukjlee@amd.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Summer Yang <girasoleyang@gmail.com>
2026-10-08 00:33:32 -04:00
b2863c0de9 [Kimi-K3] Use the fused K/V pack kernel on the decode-context-parallel prefill path (#56881)
Signed-off-by: Brandon Zhang <31413216+brandonfzhang@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
2026-10-06 00:13:55 +00:00
Vadim GimpelsonCursor Agentmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
0eac152707 [Bugfix] Route every speculation-capable row through the speculative path so recurrent state stays consistent (#56531)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-06 03:03:44 +04:00
cjackal 0e468adb43 [Model] Extend device-side mm normalization to Kimi K2.5 / K3 (#59278)
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com>
2026-10-05 21:28:13 +08:00
cb6458c5ce [Core] Remove deprecated mamba_cache_mode "all" (#58997)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Thomas Parnell <tpa@zurich.ibm.com>
2026-10-01 02:59:37 -07:00
Duncan MossRoger Wangmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Thien Tran
766b4ae54c [Kimi-K3] Add FlashInfer speculative KDA backend (#54255)
Signed-off-by: Duncan Moss <dmoss@nvidia.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-01 02:28:50 +00:00
Chauncey 3e2a7e74a5 [Feat][Model] Enable KDA prefill checkpoints for GLM-5.3-Flash (#56960)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-09-29 09:14:30 +00:00
Oxana KorzhandClaude Opus 5.5 94d1462924 [Bugfix][Kimi-K3] Refresh DSpark context KV cache pointers after the KV cache is re-bound (#58814)
Signed-off-by: Oxana Korzh <okorzh@amd.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 11:05:10 -05:00
Yongye ZhuandClaude Opus 5.5 378504a544 [MoE] Defer the TRTLLM-Gen top-k finalize on the modular path (#58635)
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 15:38:29 +08:00
aoshen02andYongye Zhu 973a3be780 [Refactor][Quantization] Make FP8 and MLA weight transforms reusable pure functions (#57732)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
2026-09-23 01:16:20 -07:00
Albert Chengmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Benjamin Chislett
88aa0d287d [Spec decode] Support variable-length decode for Kimi-K3 adaptive ver (#52988)
Signed-off-by: Albert Cheng <albecheng@nvidia.com>
Signed-off-by: Albert Cheng (Engrg-Hardware 1) <albecheng@login-bia01.bia.clusters.nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com>
2026-09-22 20:43:32 -07:00
f92b78f6ef [Kernel][DSV4.1] Fuse MXFP8 wo_b GEMM with sequence-parallel reduce-scatter (#57428)
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Signed-off-by: Canlin <canlinguosdu@gmail.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
2026-09-22 01:07:17 -07:00
382970ee6c [Kimi-K3][AMD] Return KDA and MLA projection outputs directly (#50592)
Signed-off-by: Liuyinfeng01 <yinfeliu@amd.com>
Co-authored-by: Liuyinfeng01 <199041580+LiuYinfeng01@users.noreply.github.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-21 15:44:41 -05:00
Chaunceyandzjy0516 3df4ae153e [Core][KDA] Generalize Mamba prefill checkpoint builder and exporter (#57783)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
2026-09-21 04:46:11 +00:00
Ariel BereslavskyAriel BereslavskyCodexMisha GoinYongye Zhumergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
a7fda4c88b [Bugfix] Fix SM100 fp8_ds_mla cache scales (#49435)
Signed-off-by: Ariel Bereslavsky <arielb@getimpala.ai>
Signed-off-by: Ariel Bereslavsky <ariel@getimpala.ai>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Ariel Bereslavsky <arielb@getimpala.ai>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-19 17:15:07 -07:00
Julien DebacheandHarry Mellor 2bdae2a5a8 [fix] Mistral-Large-3 accuracy regression on main (#57563)
Signed-off-by: jdebache <jdebache@nvidia.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-09-18 14:19:16 +00:00
Kyle SayersandMisha Goin 76d517fd12 [Quantization] Support kimi-k3 routed expert quant (#57430)
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
2026-09-18 08:03:59 -06:00
Elvir CrnčevićOpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Tyler Michael Smith
ac2f0ea82c [MoE][Bugfix] Skip SP padded rows in grouped MoE routing (#56079)
Signed-off-by: Elvir Crncevic <elvircrn@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
2026-09-17 19:36:19 +00:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
sashko-zakharchukandzjy0516 f8b5c11468 [Bugfix][Kimi-K3] Do not classify a stateless first chunk as a decode (#51483)
Signed-off-by: Oleksandr Zakharchuk <oleksandr.zakharchuk@gmail.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
2026-09-16 21:24:48 +08:00
Wentao Yeandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 18f8aa0465 [Refactor] Remove dead kernel code (#56845)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-15 13:17:53 -07:00
Yongye ZhuandClaude Opus 5 8263ea12bd [Build] Move DeepGEMM pin to the vLLM fork and update the Mega MoE call convention (#56876)
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-15 00:57:18 -07:00
chengchengpeiandTyler Michael Smith 0c1e89ceb9 [Bugfix][Kimi-K3] Fix KDA projection overlap on Hopper (#55426)
Signed-off-by: Chengcheng Pei <5881383+chengchengpei@users.noreply.github.com>
Co-authored-by: Chengcheng Pei <5881383+chengchengpei@users.noreply.github.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
2026-09-11 19:07:30 +00:00
Wentao Ye 1e1060f998 [Kimi Perf] Group fp8 mla cahche insertion, 4~6x kernel level performance improvement for small batch (#55356)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-09-11 08:16:57 -07:00
siyuandIsotr0py 73fb19151f [Fast Start] Support fp4 (#55465)
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
2026-09-10 13:22:17 +00:00
Wentao Ye 86aca66191 [Kimi K3 Perf] Avoid KDA mixed-batch gather/scatter, 5.2%~7.7% E2E Throughput Improvement (#56159)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-09-10 17:57:01 +08:00
Yunzhu LuandIsotr0py 442d36031c [MM][CG] Enable encoder CUDA Graph for MiniCPM-V (#42785)
Signed-off-by: YunzhuLu <lucia.yunzhu@gmail.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
2026-09-10 06:28:48 +00:00
Wentao Ye bfb443a6b6 [Kimi Bug] Fix kda ima Triton Error [CUDA]: an illegal memory access was encountered (#55924)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-09-08 14:07:54 -07:00
Sherif WalyandNicolò Lucchesi f998862d46 [Bugfix] Fix Kimi K3 loading with interleaved weight streams (#53379)
Signed-off-by: Sherif Waly <sherif.waly@mistral.ai>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
2026-09-08 16:08:51 +02:00
Wei Zhao a69402aaa8 [Perf] Integrate FlashInfer KDA kernels (#55364)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
2026-09-07 20:29:30 -07:00
Ziming HuangandOpenAI Codex 144e79c810 [Kimi K3] Support internal prefix checkpoints with partial prefix caching and spec-decoding (#53614)
Signed-off-by: Ziming Huang <zelda.huanghuang@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-06 13:18:15 +08:00
d87a440f88 [Core][MRV2] Support eagle3 spec decode with pipeline parallel (#50514)
Signed-off-by: Yongqin Wang <yongqinwang@roblox.com>
Co-authored-by: Cursor <noreply@cursor.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-09-05 11:45:34 +08:00
Wei ZhaoandOpenAI Codex 16328c7a77 [Perf] Kimi K3 nvfp4 Align in_proj weights by 128 to avoid elementwise copy (#55242)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-05 10:46:47 +08:00
5690b02c03 [Quantization] Support online quantization with partially pre-quantized checkpoints (#51392)
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
2026-09-04 11:46:06 -07:00
Bolin SunandClaude Fable 5 d410fc12f3 [Kernel] Enable Kimi-K3 SiTU on the CuteDSL MoE backend and the SM107 low-latency GEMM plan (#54606)
Signed-off-by: Bolin Sun <bolins@nvidia.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-03 16:49:31 +00:00