MOVZX
e0fe5df731
Merge branch 'main' into patches
2026-10-17 08:06:56 +00:00
Vadim Gimpelson and Claude
783380af39
[Bugfix] Fix NaN output of the Mamba2 chunk scan kernel on sm90 ( #61718 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-16 17:24:51 -07:00
Guan-Ming Chiu and vllm-agent
3dc6def360
[Attention][Spec Decode] Enable varlen Mamba2 decode for adaptive verification ( #53036 )
...
Signed-off-by: Guan-Ming (Wesley) Chiu <105915352+guan404ming@users.noreply.github.com >
Co-authored-by: vllm-agent <claw@inferact.ai >
2026-10-16 11:23:59 -07:00
MOVZX
76df77c04e
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:16:39 +07:00
f9b65c30c7
[ROCm][Perf] Make AITER FlyDSL GDN prefill default on gfx942/gfx950 ( #60645 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-14 10:11:57 +00:00
Jiangyun Zhu and Claude Opus 5.5
badfcdbf91
[Perf][GDN] Internal prefill checkpoints for the FlashInfer GDN prefill backend ( #60050 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 14:49:46 +08:00
Vadim Gimpelson and Claude
cd7517fb17
[Bugfix] Fix ReplaySSM ring shape assertion when TP does not divide Mamba2 n_groups ( #60704 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-13 11:31:53 +00:00
c7a305ec10
[Bugfix][Mamba] Store conv_kernel - 1 inputs in Mamba2 prefill checkpoints ( #60797 )
...
Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com >
2026-10-13 10:09:17 +00:00
2cc2e68d1c
[Core][Mamba] Remove leftovers of mamba_cache_mode="all" ( #60043 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Jiangyun Zhu <zhujiangyun@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-13 00:58:59 -07:00
MOVZX
ac305c44cb
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-09 17:41:13 +00:00
8a26869bca
[Bugfix] Index Mamba align state copies by request slot in MRV2 ( #60210 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Karl0007 <44745209+Karl0007@users.noreply.github.com >
Co-authored-by: Yongqin Wang <yongqinwang@roblox.com >
2026-10-07 02:32:46 +00:00
roikoren755 and zjy0516
e11962fc1c
[Feat][Mamba2] Enable internal prefill checkpoints ( #57329 )
...
Signed-off-by: Roi Koren <roik@nvidia.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
2026-10-06 07:53:51 +00:00
MOVZX
85c31d47ad
Merge remote-tracking branch 'upstream/main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-03 22:06:10 +07:00
MOVZX
2877121b2d
Merge remote-tracking branch 'upstream/main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-03 21:59:03 +07:00
e4294144a8
[Perf] Keep non-speculative GDN decode on the standard path ( #59735 )
...
Signed-off-by: DimensionSTP <65501090+DimensionSTP@users.noreply.github.com >
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Kimi Code <noreply@moonshot.ai >
2026-10-02 13:44:53 +00:00
cb6458c5ce
[Core] Remove deprecated mamba_cache_mode "all" ( #58997 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Thomas Parnell <tpa@zurich.ibm.com >
2026-10-01 02:59:37 -07:00
Andrii Skliar and Andrii Skliar
deb148e730
[Mamba] Add FlashInfer ReplaySSM support for MTP ( #52928 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
2026-09-30 21:41:17 +00:00
Harry Mellor and Claude Opus 5.5
fc2c801ad9
Disable mypy arg-type and assignment checks in tests ( #59428 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 15:18:50 +00:00
almayne and Li, Jiang <jiang1.li@intel.com>
93dd80f636
[CPU] Conv1d optimised kernel for aarch64 ( #54093 )
...
Signed-off-by: Anna Mayne <anna.mayne@arm.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-09-30 20:23:42 +08:00
5567cc1b25
[Mamba] RecoverSSM align: keep a boundary-ending state in its block
...
A speculative window whose accepted tokens end exactly on a recurrent block
boundary selected the following block-table column for the committed state
(commit plan) and recorded it as the running column (postprocess). That
column can still be unallocated, so the next step reads an uninitialized
page. Select the block that holds the last committed token, matching the
(n - 1) // block convention used elsewhere in align mode. Applies to
Kimi-K3 KDA and to the Qwen GDN/PLE RecoverSSM paths that share the plan.
Fix and test from local-inference-lab/vllm a02f585d (Martin Vit, root cause
reported by astra); the GDN/PLE align tests now expect the same convention.
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Martin Vit <martin@voipmonitor.org >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 11:12:40 +02:00
Jürgen Schmied and Claude
03cae7837f
[Spec Decode][Qwen4Exp] RecoverSSM for Qwen GDN and the PLE short conv
...
With --use-replayssm and speculative decoding, Qwen4Exp's Gated DeltaNet layers verify the
draft window from a read-only checkpoint and keep a per-token replay record
[correction(V), normalized key(K), gate(1)] in fp32. After sampling, one commit replays the
accepted tokens and writes the recurrent state once (plus the block-boundary state in align
mode), instead of snapshotting the state after every draft position. The PLE dilated short
conv follows the same protocol (accepted-suffix compaction). num_speculative_blocks = 0,
so no per-draft Mamba blocks are reserved.
- GDN verify/commit kernels: model_executor/layers/mamba/gdn/recoverssm_gdn.py
- metadata/backends: v1/attention/backends/gdn_recoverssm.py, ple_recoverssm.py
- the verify reads strided q/k/v views of the conv output and writes in place
- builders declare AttentionCGSupport.NEVER: FULL_AND_PIECEWISE falls back to PIECEWISE;
the fallback now also counts breakable CUDA graphs (empty splitting_ops, with or
without torch.compile) as piecewise
attention instead of dropping to NONE
- the generic Mamba metadata builder skips the Mamba2 ReplaySSM ring-buffer workspaces
when --use-replayssm selects RecoverSSM (the PLE short conv uses that builder)
- tests: GDN verify/commit vs the native kernels (fp32 and bf16 state, align boundaries,
mixed and ragged batches, extreme gates), PLE vs token-by-token decode, config selection
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 01:17:24 +02:00
Wentao Ye
bf93e1f92c
[Refactor] Remove dead tests utils ( #58803 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-26 09:16:53 -04:00
wangxiyuan
44257c2e35
[Feature] Triton kernel dispatcher ( #43048 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2026-09-25 04:07:43 +00:00
67ddac3ea8
[CPU][GDN] Support NIXL DS convolution-state layout ( #53300 )
...
Signed-off-by: Li, Tianmu <tianmu.li@intel.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-09-24 14:55:20 +08:00
Andreas Karatzas
a22c637ed9
[ROCm][Test] Check GDN prefill numerics and output ownership ( #58091 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-09-23 02:19:58 -07:00
Mikko Tukiainen and Cursor
68363d64ec
[ROCm][Perf] Avoid extra reshape kernel in Qwen GDN output norm ( #47842 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-22 14:38:16 +08:00
fbe8a157fb
[ROCm][Perf] Enable the AITER GDN decode fast path for flat qkvz layouts ( #53623 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: TJian <tunjian.tan@embeddedllm.com >
2026-09-19 06:05:04 -07:00
Zoe923
6a7de3ba3a
[Bugfix][GDN] Fix CuteDSL BF16 KKT inversion divergence ( #53864 )
...
Signed-off-by: Zoe923 <871844693@qq.com >
2026-09-17 08:49:56 -07:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
Jiangyun Zhu
98ed0856f3
[Model] add GLM-5.3-Flash support ( #53906 )
2026-09-03 09:40:35 -07:00
Andrii Skliar and Andrii Skliar
adebc41b7e
[Mamba] Add FlashInfer ReplaySSM backend ( #52506 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Signed-off-by: Andrii Skliar <andreyws96@gmail.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
2026-09-01 10:26:43 -04:00
Tianmu Li and OpenAI Codex
c6c33f2b1f
[CPU] Support FP16/BF16 persisted GDN state on AMX ( #52191 )
...
Signed-off-by: Li, Tianmu <tianmu.li@intel.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-08-31 16:32:20 +08:00
+5
e126687a9a
[Model] Support Qwen3.8-Flash-Next ( #53896 )
...
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com >
Signed-off-by: andyluo7 <andy.luo@amd.com >
Signed-off-by: lishunyang12 <lishunyang03@gmail.com >
Signed-off-by: Harry Huang <huanghaoyan.hhy@alibaba-inc.com >
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: KungYork <firelikest@163.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: andyluo7 <43718156+andyluo7@users.noreply.github.com >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: andyluo7 <andy.luo@amd.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: lishunyang12 <lishunyang03@gmail.com >
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-31 05:57:56 +00:00
pmanczak
44cf3f0466
[XPU][Tests] Make tests device-agnostic ( #51968 )
...
Signed-off-by: pmanczak <pawel.manczak@intel.com >
2026-08-20 02:25:41 -07:00
Dinesh Chitlangia
525b7bbb3a
[Bugfix][CPU] Enable C++ causal_conv1d GDN path and float32 SSM cache on non-AMX AVX-512BF16 CPUs ( #49688 )
...
Signed-off-by: Dinesh Chitlangia <dineshchitlangia@gmail.com >
2026-08-19 10:50:59 -05:00
kiroxu
cdb8545a91
[Kernel][Perf] Support Qwen head ratios in fused GDN MTP ( #52539 )
...
Signed-off-by: kiroxu <148877251+BabyDrangoner@users.noreply.github.com >
2026-08-17 18:17:40 -07:00
1be3628367
[Kernel][Perf] Add fused CUDA post-conv MTP decode kernel for Qwen3.5 GDN ( #51674 )
...
Signed-off-by: Jie Fang <jief@nvidia.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-08-14 11:44:45 +08:00
Francesco Fusco and Claude
fac808b36f
[Perf][Hybrid] 3D-grid tiling of the state-copy Triton kernels ( #49436 )
...
Signed-off-by: Francesco Fusco <ffu@zurich.ibm.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-08-10 17:02:49 +02:00
amitz-nv and OpenAI Codex
52c0e3cb08
[Kernel] Add support for Flashinfer Mamba SSU algorithm selection ( #50157 )
...
Signed-off-by: amitz-nv <203509407+amitz-nv@users.noreply.github.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-04 09:40:23 -07:00
Martin Hickey and Harry Mellor
38a267cdd5
[MyPy][1/N] Fix mypy errors in some tests/ directories and enforce follow-imports=silent ( #49570 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-30 05:02:51 -07:00
Li, Jiang and Claude
072a472782
[CPU] Bump up CPU kernels to latest version ( #50387 )
...
Signed-off-by: jiang1.li <jiang1.li@intel.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-07-30 19:18:43 +08:00
f51193b9ae
[Kernel][Mamba] Fused-kernel support for align-mode DS-conv state migration with num_accepted_tokens > 1 ( #49291 )
...
Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com >
2026-07-29 18:54:52 +08:00
544cb724c8
[CPU][Spec Decode] Optimize GDN conv path for speculative decoding ( #48577 )
...
Signed-off-by: Li, Tianmu <tianmu.li@intel.com >
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-07-27 06:20:14 +00:00
866fea2b99
[Kernel] ReplaySSM: cache SSM inputs for faster Mamba2 standard decode ( #48018 )
...
Signed-off-by: Johnny-Liou <a897111@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-07-24 09:39:49 -07:00
Thien Tran and Codex
b44311b6ef
[CI] stabilize GDN prefill CuTeDSL test ( #49388 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Codex <noreply@openai.com >
2026-07-22 09:03:46 -07:00
5c9f6557d7
[Hardware][CPU] Enable granite-4 model on cpu ( #47641 )
...
Signed-off-by: Akash Kaothalkar <akashkaothalkar@akashs-mbp.bl1-in.ibm.com >
Signed-off-by: Akash Kaothalkar <akashkaothalkar@dhcp-9-123-5-76.bl1-in.ibm.com >
Signed-off-by: Akash Kaothalkar <akashkaothalkar@Akashs-MBP.lan >
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com >
Co-authored-by: Akash Kaothalkar <akashkaothalkar@dhcp-9-123-5-76.bl1-in.ibm.com >
Co-authored-by: Akash Kaothalkar <akashkaothalkar@Akashs-MBP.lan >
Co-authored-by: Akash Kaothalkar <akashkaothalkar@akashs-mbp.bl1-in.ibm.com >
Co-authored-by: Akash kaothalkar <akash.kaothalkar@ibm.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-07-20 06:15:16 +00:00
1dcbbd9cac
[CI] Move compatible 1xL4 jobs to H200 35GB MIG ( #43024 )
...
Signed-off-by: Simon Mo <simon@inferact.ai >
Co-authored-by: Simon Mo <simon@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-07-19 19:21:25 -07:00
Wentao Ye
626c90b2d5
[Refactor] Move fla to third party ( #48500 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-07-16 19:22:36 +01:00
Rahul Vishwakarma
f7efab58ec
[CPU][Bugfix] Fix flaky ShortConv prefill test on ARM (uninitialized weights) ( #47848 )
...
Signed-off-by: Rahul Vishwakarma <Rahul.Vishwakarma2@ibm.com >
2026-07-07 18:20:09 -07:00