MOVZX
e0fe5df731
Merge branch 'main' into patches
2026-10-17 08:06:56 +00:00
f019e3e105
[ROCm] attn res + sigmoid_mul + conv fusions ( #52968 )
...
Signed-off-by: Omar Muhammad <omar.muhammad@amd.com >
Signed-off-by: Nicholas Susanto <nicholas.susanto@amd.com >
Co-authored-by: Nicholas Susanto <nicholas.susanto@amd.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-17 00:54:54 +00:00
28e92eeff2
[ROCm][Perf][Kimi-K3] Use AITER Gluon chunk_kda for KDA prefill ( #59552 )
...
Signed-off-by: Omar Muhammad <omar.muhammad@amd.com >
Signed-off-by: Nicholas Susanto <nicholas.susanto@amd.com >
Co-authored-by: Nicholas Susanto <nicholas.susanto@amd.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-16 22:58:56 +00:00
Oxana Korzh and Claude Opus 5.5
5bf3d9f665
[ROCm][Kimi-K3] Support DSpark adaptive verification on MI355X ( #59824 )
...
Signed-off-by: Oxana Korzh <okorzh@amd.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-16 13:53:49 -05:00
MOVZX
be95edb7f6
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-16 17:35:42 +07:00
3f1e57225a
[ROCm][Perf][Kimi-K3] Use AITER Gluon KDA kernels for decode and spec decode ( #57392 )
...
Signed-off-by: omuhamma <omar.muhammad@amd.com >
Signed-off-by: Nicholas Susanto <nicholas.susanto@amd.com >
Co-authored-by: Nicholas Susanto <nicholas.susanto@amd.com >
Co-authored-by: Simon Danielsson <70206058+simondanielsson@users.noreply.github.com >
2026-10-15 17:54:24 -05:00
MOVZX
76df77c04e
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:16:39 +07:00
Rohan Potdar and Claude Opus 5.5
52a8cc5316
[ROCm][CI] Add dtype to the KDA metadata test's fake model config ( #61650 )
...
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 18:04:14 +00:00
Andreas Karatzas and Claude
67516b6c42
[Bugfix][ROCm][Kimi-K3] Stage fused KDA decode weights after loading ( #61207 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-13 14:55:57 -05:00
c5bda658d2
[Model][MoE] Load MegaMoE experts directly into the DeepGEMM layout ( #60096 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Aaron Hao <hao-aaron@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Aaron Hao <hao-aaron@users.noreply.github.com >
2026-10-13 11:52:08 -07:00
dingdangmao and Codex
124b89d5b7
[Bugfix][RecoverSSM] Correct aligned state indices at exact block boundaries ( #59962 )
...
Signed-off-by: apex-mochen <2756823972@qq.com >
Co-authored-by: Codex <noreply@openai.com >
2026-10-12 02:08:14 +00:00
Nick Hill
c059bee812
[MRV2] Switch CI tests still running on MRV1 unnecessarily ( #61014 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-10-11 09:57:07 -07:00
f8e6e365ae
[Bugfix][Mooncake] Save the EAGLE attention proof with hybrid Mamba partial tails ( #60695 )
...
Signed-off-by: Dao Le <Dao007forever@gmail.com >
Signed-off-by: Dao007forever <daole@inferact.ai >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
2026-10-10 00:08:08 +00:00
MOVZX
ac305c44cb
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-09 17:41:13 +00:00
Rohan Potdar and Claude Opus 5.5
88a419649e
[ROCm][Kimi-K3] Use vLLM's FP8 max in the fused AttnRes quantization ( #60717 )
...
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 04:45:13 +00:00
d8fa463e2b
[Bugfix][Prefix caching] Fix inconsistent hybrid Mamba prefix cache boundaries ( #60533 )
...
Signed-off-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-08 18:33:48 +00:00
Elvir Crnčević and Misha Goin
8674a794c7
[Perf] Enable Kimi K3 shared expert sharding with DeepEP v2 ( #56875 )
...
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-10-08 16:42:15 +00:00
9f3d70e081
[Quantization] Enable shared expert fusion compatibility with online shared_expert quantization (showcase: along Quark MXFP4 routed experts) ( #55686 )
...
Signed-off-by: Felix Marty <Felix.Marty@amd.com >
Signed-off-by: fxmarty-amd <felmarty@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-10-08 12:04:46 -04:00
kliuae
d80ddaac73
[ROCm][Perf] Kimi-K3 Store only the current rank's shards in latent MoE up-proj ( #59591 )
...
Signed-off-by: kliuae <kuanfu.liu@embeddedllm.com >
2026-10-08 14:55:20 +00:00
458ba2edf8
[ROCm][Kimi-K3] Fuse AttnRes output with per-token FP8 input quantization ( #59069 )
...
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com >
2026-10-08 12:01:47 +00:00
b2863c0de9
[Kimi-K3] Use the fused K/V pack kernel on the decode-context-parallel prefill path ( #56881 )
...
Signed-off-by: Brandon Zhang <31413216+brandonfzhang@users.noreply.github.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-10-06 00:13:55 +00:00
0eac152707
[Bugfix] Route every speculation-capable row through the speculative path so recurrent state stays consistent ( #56531 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
Co-authored-by: Cursor Agent <cursoragent@cursor.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-06 03:03:44 +04:00
MOVZX
85c31d47ad
Merge remote-tracking branch 'upstream/main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-03 22:06:10 +07:00
MOVZX
2877121b2d
Merge remote-tracking branch 'upstream/main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-03 21:59:03 +07:00
Markus Hartikainen
b0e21b3083
[ROCm][Kimi-K3] Add opt-in gfx942 MXFP4-to-int4 conversion ( #51274 )
...
Signed-off-by: Markus Hartikainen <markus.hartikainen@amd.com >
Signed-off-by: Hartikainen <mahartik@amd.com >
2026-10-03 14:28:02 +08:00
Jiangyun Zhu and Claude Opus 5.5
d648847fe4
[CI][Kimi-K3] Test prefix cache reuse with KV offload, P/D and DCP ( #59229 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-02 20:58:35 +08:00
kliuae
0154979656
[ROCm][Perf] Kimi-K3 enable prefill checkpoints on ROCm ( #58344 )
...
Signed-off-by: kliuae <kuanfu.liu@embeddedllm.com >
2026-10-02 06:58:04 +00:00
766b4ae54c
[Kimi-K3] Add FlashInfer speculative KDA backend ( #54255 )
...
Signed-off-by: Duncan Moss <dmoss@nvidia.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-10-01 02:28:50 +00:00
MOVZX and Hinata
ae0350c0de
[Spec Decode][MRV2] RecoverSSM for Qwen GDN and PLE short conv (PR #58863 )
...
Intake of open upstream PR vllm-project/vllm#58863 . Active only with
VLLM_USE_V2_MODEL_RUNNER=1; inert otherwise.
Speculative decoding on Gated DeltaNet snapshotted the full recurrent
state after every draft position and reserved one Mamba block per draft
position. On a 24 GB card that is reserved VRAM we never use, plus
per-step copy traffic.
RecoverSSM, the protocol Kimi-K3 KDA already uses, replaces this:
verify runs from a read-only checkpoint and stores a small per-token
replay record [correction (V), normalized key (K), gate (1)] in fp32;
commit replays only the accepted tokens and writes the recurrent state
once, plus the block-boundary state in align mode. The PLE dilated
short conv compacts its accepted window with the same plan.
Conflict in vllm/models/qwen4_exp/nvidia/mtp.py: our get_top_tokens()
work put LocalArgmaxMixin on Qwen4ExpMTP, the PR adds SupportsReplaySSM
to the same base list. Kept both.
Co-Authored-By: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-09-29 13:39:09 +07:00
xaguilar-amd and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
a7ff44355c
[ROCm][Perf] Kimi-K3 Enable sharded latent MoE up-projection under EP ( #54956 )
...
Signed-off-by: Xavier Aguilar <xavier.aguilarfruto@amd.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-28 10:31:29 -04:00
dd3bc9c72f
[Perf][MRV2] Reuse Mamba/GDN metadata across KV cache groups ( #58762 )
...
Signed-off-by: Luca Motz <luca.motz@icloud.com >
Co-authored-by: Xin Yang <xyangx@amazon.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-09-28 04:36:58 +00:00
5567cc1b25
[Mamba] RecoverSSM align: keep a boundary-ending state in its block
...
A speculative window whose accepted tokens end exactly on a recurrent block
boundary selected the following block-table column for the committed state
(commit plan) and recorded it as the running column (postprocess). That
column can still be unallocated, so the next step reads an uninitialized
page. Select the block that holds the last committed token, matching the
(n - 1) // block convention used elsewhere in align mode. Applies to
Kimi-K3 KDA and to the Qwen GDN/PLE RecoverSSM paths that share the plan.
Fix and test from local-inference-lab/vllm a02f585d (Martin Vit, root cause
reported by astra); the GDN/PLE align tests now expect the same convention.
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Martin Vit <martin@voipmonitor.org >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 11:12:40 +02:00
Jürgen Schmied and Claude
a80e41d690
[Mamba] Share RecoverSSM commit kernels and helpers across models
...
RecoverSSM's commit-plan and conv-window compaction kernels are model independent.
Move them from models/kimi_k3 into model_executor/layers/mamba/ops/recoverssm.py
(Kimi-K3 keeps its private names as import aliases; kernel source unchanged), add
mamba/recoverssm_utils.py (activation predicate, argument validation) and the
generic spec-row/request mapping helpers to v1/attention/backends/recoverssm_metadata.py.
CacheConfig.use_kda_recoverssm becomes use_recoverssm: the switch now selects RecoverSSM
for any model whose recurrent layers implement it, not only KDA.
No functional change for Kimi-K3.
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 00:07:04 +02:00
1417022c3d
[ROCm][Kimi-K3] Optimize low-concurrency speculative KDA ( #58045 )
...
Signed-off-by: jiacao-amd <jiahui.cao@amd.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-25 12:33:30 -05:00
Yongye Zhu and Claude Opus 5.5
378504a544
[MoE] Defer the TRTLLM-Gen top-k finalize on the modular path ( #58635 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-25 15:38:29 +08:00
c4cfd7007b
[CI][ROCm] Add an MI355 Kimi-K3 unit test group ( #58012 )
...
Signed-off-by: Oxana Korzh <okorzh@amd.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-23 11:25:51 -05:00
382970ee6c
[Kimi-K3][AMD] Return KDA and MLA projection outputs directly ( #50592 )
...
Signed-off-by: Liuyinfeng01 <yinfeliu@amd.com >
Co-authored-by: Liuyinfeng01 <199041580+LiuYinfeng01@users.noreply.github.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-21 15:44:41 -05:00
Chauncey and zjy0516
3df4ae153e
[Core][KDA] Generalize Mamba prefill checkpoint builder and exporter ( #57783 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
2026-09-21 04:46:11 +00:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
sashko-zakharchuk and zjy0516
f8b5c11468
[Bugfix][Kimi-K3] Do not classify a stateless first chunk as a decode ( #51483 )
...
Signed-off-by: Oleksandr Zakharchuk <oleksandr.zakharchuk@gmail.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
2026-09-16 21:24:48 +08:00
Wei Zhao
9d88ceb026
[KDA] Update flashKDA to support bf16 checkpoint state ( #56485 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
2026-09-11 23:53:24 +00:00
siyu and Isotr0py
73fb19151f
[Fast Start] Support fp4 ( #55465 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-09-10 13:22:17 +00:00
Wentao Ye
86aca66191
[Kimi K3 Perf] Avoid KDA mixed-batch gather/scatter, 5.2%~7.7% E2E Throughput Improvement ( #56159 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-10 17:57:01 +08:00
9521c60bdc
[ROCm][Perf] Kimi-K3 Fused kernels for KDA prefill reland ( #54038 )
...
Signed-off-by: kliuae <kuanfu.liu@embeddedllm.com >
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com >
Signed-off-by: kliuae <17350011+kliuae@users.noreply.github.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: TJian <tunjian.tan@embeddedllm.com >
2026-09-10 15:37:54 +08:00
Sherif Waly and Nicolò Lucchesi
f998862d46
[Bugfix] Fix Kimi K3 loading with interleaved weight streams ( #53379 )
...
Signed-off-by: Sherif Waly <sherif.waly@mistral.ai >
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
2026-09-08 16:08:51 +02:00
Wei Zhao
a69402aaa8
[Perf] Integrate FlashInfer KDA kernels ( #55364 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
2026-09-07 20:29:30 -07:00
Ziming Huang and OpenAI Codex
144e79c810
[Kimi K3] Support internal prefix checkpoints with partial prefix caching and spec-decoding ( #53614 )
...
Signed-off-by: Ziming Huang <zelda.huanghuang@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-06 13:18:15 +08:00
d87a440f88
[Core][MRV2] Support eagle3 spec decode with pipeline parallel ( #50514 )
...
Signed-off-by: Yongqin Wang <yongqinwang@roblox.com >
Co-authored-by: Cursor <noreply@cursor.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-09-05 11:45:34 +08:00
6cbb3c154e
[Perf][GDN] Build cudagraph-capture metadata without a device sync ( #55404 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Roger Wang <rogerw@inferact.ai >
Co-authored-by: Kimi Code <noreply@moonshot.cn >
2026-09-04 17:40:17 -07:00