MOVZX
e0fe5df731
Merge branch 'main' into patches
2026-10-17 08:06:56 +00:00
Vadim Gimpelson and Claude
783380af39
[Bugfix] Fix NaN output of the Mamba2 chunk scan kernel on sm90 ( #61718 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-16 17:24:51 -07:00
MOVZX
76df77c04e
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:16:39 +07:00
f9b65c30c7
[ROCm][Perf] Make AITER FlyDSL GDN prefill default on gfx942/gfx950 ( #60645 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-14 10:11:57 +00:00
Jiangyun Zhu and Claude Opus 5.5
badfcdbf91
[Perf][GDN] Internal prefill checkpoints for the FlashInfer GDN prefill backend ( #60050 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 14:49:46 +08:00
Vadim Gimpelson and Claude
cd7517fb17
[Bugfix] Fix ReplaySSM ring shape assertion when TP does not divide Mamba2 n_groups ( #60704 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-13 11:31:53 +00:00
c7a305ec10
[Bugfix][Mamba] Store conv_kernel - 1 inputs in Mamba2 prefill checkpoints ( #60797 )
...
Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com >
2026-10-13 10:09:17 +00:00
2cc2e68d1c
[Core][Mamba] Remove leftovers of mamba_cache_mode="all" ( #60043 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Jiangyun Zhu <zhujiangyun@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-13 00:58:59 -07:00
liuzhenwei and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
ff440c7805
[BreakableCG] Add eager break for Mamba-family ops ( #57881 )
...
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-10 14:06:57 +08:00
MOVZX
ac305c44cb
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-09 17:41:13 +00:00
Harry Mellor and Claude Opus 5.5
2c99ee9333
[Chore] Remove vendored code made redundant by Transformers 5.16.1 ( #60623 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 09:29:03 +00:00
Paweł Mańczak and Kunshang Ji
f0a5f111f2
[XPU] Add tuned Mamba SSU configs for Intel Arc Pro B60 ( #56765 )
...
Signed-off-by: pmanczak <pawel.manczak@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-10-09 14:29:29 +08:00
d8fa463e2b
[Bugfix][Prefix caching] Fix inconsistent hybrid Mamba prefix cache boundaries ( #60533 )
...
Signed-off-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-08 18:33:48 +00:00
Paweł Mańczak
4cf1307e72
[XPU] Add tuned Mamba SSU configs for 2 more B70 shapes ( #57565 )
...
Signed-off-by: pmanczak <pawel.manczak@intel.com >
2026-10-08 16:36:05 +08:00
Mikko Tukiainen and Cursor
548597367e
[ROCm][Perf] Add AITER FlyDSL GDN prefill backend ( #57560 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-10-07 09:44:22 -05:00
roikoren755 and zjy0516
e11962fc1c
[Feat][Mamba2] Enable internal prefill checkpoints ( #57329 )
...
Signed-off-by: Roi Koren <roik@nvidia.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
2026-10-06 07:53:51 +00:00
MOVZX
85c31d47ad
Merge remote-tracking branch 'upstream/main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-03 22:06:10 +07:00
MOVZX
2877121b2d
Merge remote-tracking branch 'upstream/main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-03 21:59:03 +07:00
e4294144a8
[Perf] Keep non-speculative GDN decode on the standard path ( #59735 )
...
Signed-off-by: DimensionSTP <65501090+DimensionSTP@users.noreply.github.com >
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Kimi Code <noreply@moonshot.ai >
2026-10-02 13:44:53 +00:00
b538d807ff
[Bugfix][MRV2] Keep GDN prefill checkpoint metadata local to each cache group ( #59536 )
...
Signed-off-by: Jingqiao Zhang <ai-jz@users.noreply.github.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Jingqiao Zhang <ai-jz@users.noreply.github.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
2026-10-01 22:59:15 +08:00
cb6458c5ce
[Core] Remove deprecated mamba_cache_mode "all" ( #58997 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Thomas Parnell <tpa@zurich.ibm.com >
2026-10-01 02:59:37 -07:00
Andrii Skliar and Andrii Skliar
deb148e730
[Mamba] Add FlashInfer ReplaySSM support for MTP ( #52928 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
2026-09-30 21:41:17 +00:00
almayne and Li, Jiang <jiang1.li@intel.com>
93dd80f636
[CPU] Conv1d optimised kernel for aarch64 ( #54093 )
...
Signed-off-by: Anna Mayne <anna.mayne@arm.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-09-30 20:23:42 +08:00
d71f662606
[Bugfix][Model][CPU] Fix Mamba2 quantized in_proj weight and scale loading for TP >1 ( #58083 )
...
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com >
Co-authored-by: Akash kaothalkar <akash.kaothalkar@ibm.com >
Co-authored-by: Antigravity <antigravity@google.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-09-30 17:59:02 +08:00
Chauncey
3e2a7e74a5
[Feat][Model] Enable KDA prefill checkpoints for GLM-5.3-Flash ( #56960 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
2026-09-29 09:14:30 +00:00
5567cc1b25
[Mamba] RecoverSSM align: keep a boundary-ending state in its block
...
A speculative window whose accepted tokens end exactly on a recurrent block
boundary selected the following block-table column for the committed state
(commit plan) and recorded it as the running column (postprocess). That
column can still be unallocated, so the next step reads an uninitialized
page. Select the block that holds the last committed token, matching the
(n - 1) // block convention used elsewhere in align mode. Applies to
Kimi-K3 KDA and to the Qwen GDN/PLE RecoverSSM paths that share the plan.
Fix and test from local-inference-lab/vllm a02f585d (Martin Vit, root cause
reported by astra); the GDN/PLE align tests now expect the same convention.
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Martin Vit <martin@voipmonitor.org >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 11:12:40 +02:00
Jürgen Schmied and Claude
03cae7837f
[Spec Decode][Qwen4Exp] RecoverSSM for Qwen GDN and the PLE short conv
...
With --use-replayssm and speculative decoding, Qwen4Exp's Gated DeltaNet layers verify the
draft window from a read-only checkpoint and keep a per-token replay record
[correction(V), normalized key(K), gate(1)] in fp32. After sampling, one commit replays the
accepted tokens and writes the recurrent state once (plus the block-boundary state in align
mode), instead of snapshotting the state after every draft position. The PLE dilated short
conv follows the same protocol (accepted-suffix compaction). num_speculative_blocks = 0,
so no per-draft Mamba blocks are reserved.
- GDN verify/commit kernels: model_executor/layers/mamba/gdn/recoverssm_gdn.py
- metadata/backends: v1/attention/backends/gdn_recoverssm.py, ple_recoverssm.py
- the verify reads strided q/k/v views of the conv output and writes in place
- builders declare AttentionCGSupport.NEVER: FULL_AND_PIECEWISE falls back to PIECEWISE;
the fallback now also counts breakable CUDA graphs (empty splitting_ops, with or
without torch.compile) as piecewise
attention instead of dropping to NONE
- the generic Mamba metadata builder skips the Mamba2 ReplaySSM ring-buffer workspaces
when --use-replayssm selects RecoverSSM (the PLE short conv uses that builder)
- tests: GDN verify/commit vs the native kernels (fp32 and bf16 state, align boundaries,
mixed and ragged batches, extreme gates), PLE vs token-by-token decode, config selection
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 01:17:24 +02:00
Jürgen Schmied and Claude
a80e41d690
[Mamba] Share RecoverSSM commit kernels and helpers across models
...
RecoverSSM's commit-plan and conv-window compaction kernels are model independent.
Move them from models/kimi_k3 into model_executor/layers/mamba/ops/recoverssm.py
(Kimi-K3 keeps its private names as import aliases; kernel source unchanged), add
mamba/recoverssm_utils.py (activation predicate, argument validation) and the
generic spec-row/request mapping helpers to v1/attention/backends/recoverssm_metadata.py.
CacheConfig.use_kda_recoverssm becomes use_recoverssm: the switch now selects RecoverSSM
for any model whose recurrent layers implement it, not only KDA.
No functional change for Kimi-K3.
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 00:07:04 +02:00
Samuel Kim and Claude
34ee85d5ca
[Perf] Batch Mamba2 prefill SSM state saves, removing GPU<->CPU syncs ( #49371 )
...
Signed-off-by: samuelkim7 <samuelmwkim@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-25 01:11:19 +00:00
Harry Mellor and Claude Opus 5.5
d881812500
[Chore] Use Transformers v5 names and drop redundant processor use_fast ( #58550 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-09-24 14:14:10 +00:00
67ddac3ea8
[CPU][GDN] Support NIXL DS convolution-state layout ( #53300 )
...
Signed-off-by: Li, Tianmu <tianmu.li@intel.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-09-24 14:55:20 +08:00
44bf011f44
[Dependency] Upgrade FlashInfer version to 0.7.0 ( #58069 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-09-23 22:45:23 -07:00
Wentao Ye
fae5cbfc8b
[Refactor] Remove dead code multiple places ( #58002 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-22 16:36:03 -04:00
Mikko Tukiainen and Cursor
68363d64ec
[ROCm][Perf] Avoid extra reshape kernel in Qwen GDN output norm ( #47842 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-22 14:38:16 +08:00
382970ee6c
[Kimi-K3][AMD] Return KDA and MLA projection outputs directly ( #50592 )
...
Signed-off-by: Liuyinfeng01 <yinfeliu@amd.com >
Co-authored-by: Liuyinfeng01 <199041580+LiuYinfeng01@users.noreply.github.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-21 15:44:41 -05:00
Chauncey and zjy0516
3df4ae153e
[Core][KDA] Generalize Mamba prefill checkpoint builder and exporter ( #57783 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
2026-09-21 04:46:11 +00:00
Yan Ma
d05da62e9c
[XPU] fix incorrect gdn kernel log for XPU path ( #54871 )
...
Signed-off-by: Yan Ma <yan.ma@intel.com >
2026-09-20 14:00:31 +08:00
fbe8a157fb
[ROCm][Perf] Enable the AITER GDN decode fast path for flat qkvz layouts ( #53623 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: TJian <tunjian.tan@embeddedllm.com >
2026-09-19 06:05:04 -07:00
Taneem Ibrahim
2b8a5495a8
[Mypy] Fix mypy typing for N/O models ( #54142 )
...
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
2026-09-18 12:50:01 +01:00
Zoe923
6a7de3ba3a
[Bugfix][GDN] Fix CuteDSL BF16 KKT inversion divergence ( #53864 )
...
Signed-off-by: Zoe923 <871844693@qq.com >
2026-09-17 08:49:56 -07:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
975dca5bb5
[Perf][GDN] Scatter mixed speculative outputs into the caller buffer ( #57140 )
...
Signed-off-by: Arpan Tripathi <tripathiarpan20@gmail.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Xavier Lyu <9938360+xavierlyu@users.noreply.github.com >
Co-authored-by: Bohdan Podziubanchuk <114604629+Danbog32@users.noreply.github.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-16 13:03:57 +00:00
Wentao Ye and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
18f8aa0465
[Refactor] Remove dead kernel code ( #56845 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-15 13:17:53 -07:00
aed894c190
[6/N][warmup][DSv4] Migrate sampling, and DFlash JIT kernels ( #56323 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-12 17:24:24 -07:00
Marceli Fylcek and Claude Opus 5
5b6cf93e8e
[XPU] Add forward_xpu to Mixer2RMSNormGated and FusedRMSNormGated ( #54968 )
...
Signed-off-by: Marceli Fylcek <marceli.fylcek@intel.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-09-11 07:58:28 +08:00
Stefano Castagnetta
f6326f53bd
[Perf][GDN] Enable the FlashInfer GDN prefill kernel on SM12x ( #55715 )
...
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
2026-09-08 12:26:03 +00:00
Wei Zhao
a69402aaa8
[Perf] Integrate FlashInfer KDA kernels ( #55364 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
2026-09-07 20:29:30 -07:00
Thien Tran
d9105ea800
[Qwen3.8-Flash-Next] Remove torch.compile for NVIDIA implementation ( #55272 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-09-06 21:57:56 -07:00
Jee Jee Li
199cb9b964
[Kernel] Remove unused fake implementation ( #55535 )
...
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
2026-09-06 17:54:32 -07:00