MOVZX
e0fe5df731
Merge branch 'main' into patches
2026-10-17 08:06:56 +00:00
Vadim Gimpelson and Claude
783380af39
[Bugfix] Fix NaN output of the Mamba2 chunk scan kernel on sm90 ( #61718 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-16 17:24:51 -07:00
MOVZX
76df77c04e
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:16:39 +07:00
c7a305ec10
[Bugfix][Mamba] Store conv_kernel - 1 inputs in Mamba2 prefill checkpoints ( #60797 )
...
Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com >
2026-10-13 10:09:17 +00:00
2cc2e68d1c
[Core][Mamba] Remove leftovers of mamba_cache_mode="all" ( #60043 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Jiangyun Zhu <zhujiangyun@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-13 00:58:59 -07:00
MOVZX
ac305c44cb
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-09 17:41:13 +00:00
Paweł Mańczak and Kunshang Ji
f0a5f111f2
[XPU] Add tuned Mamba SSU configs for Intel Arc Pro B60 ( #56765 )
...
Signed-off-by: pmanczak <pawel.manczak@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-10-09 14:29:29 +08:00
Paweł Mańczak
4cf1307e72
[XPU] Add tuned Mamba SSU configs for 2 more B70 shapes ( #57565 )
...
Signed-off-by: pmanczak <pawel.manczak@intel.com >
2026-10-08 16:36:05 +08:00
roikoren755 and zjy0516
e11962fc1c
[Feat][Mamba2] Enable internal prefill checkpoints ( #57329 )
...
Signed-off-by: Roi Koren <roik@nvidia.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
2026-10-06 07:53:51 +00:00
MOVZX
2877121b2d
Merge remote-tracking branch 'upstream/main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-03 21:59:03 +07:00
Andrii Skliar and Andrii Skliar
deb148e730
[Mamba] Add FlashInfer ReplaySSM support for MTP ( #52928 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
2026-09-30 21:41:17 +00:00
almayne and Li, Jiang <jiang1.li@intel.com>
93dd80f636
[CPU] Conv1d optimised kernel for aarch64 ( #54093 )
...
Signed-off-by: Anna Mayne <anna.mayne@arm.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-09-30 20:23:42 +08:00
5567cc1b25
[Mamba] RecoverSSM align: keep a boundary-ending state in its block
...
A speculative window whose accepted tokens end exactly on a recurrent block
boundary selected the following block-table column for the committed state
(commit plan) and recorded it as the running column (postprocess). That
column can still be unallocated, so the next step reads an uninitialized
page. Select the block that holds the last committed token, matching the
(n - 1) // block convention used elsewhere in align mode. Applies to
Kimi-K3 KDA and to the Qwen GDN/PLE RecoverSSM paths that share the plan.
Fix and test from local-inference-lab/vllm a02f585d (Martin Vit, root cause
reported by astra); the GDN/PLE align tests now expect the same convention.
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Martin Vit <martin@voipmonitor.org >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 11:12:40 +02:00
Jürgen Schmied and Claude
a80e41d690
[Mamba] Share RecoverSSM commit kernels and helpers across models
...
RecoverSSM's commit-plan and conv-window compaction kernels are model independent.
Move them from models/kimi_k3 into model_executor/layers/mamba/ops/recoverssm.py
(Kimi-K3 keeps its private names as import aliases; kernel source unchanged), add
mamba/recoverssm_utils.py (activation predicate, argument validation) and the
generic spec-row/request mapping helpers to v1/attention/backends/recoverssm_metadata.py.
CacheConfig.use_kda_recoverssm becomes use_recoverssm: the switch now selects RecoverSSM
for any model whose recurrent layers implement it, not only KDA.
No functional change for Kimi-K3.
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 00:07:04 +02:00
67ddac3ea8
[CPU][GDN] Support NIXL DS convolution-state layout ( #53300 )
...
Signed-off-by: Li, Tianmu <tianmu.li@intel.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-09-24 14:55:20 +08:00
Zoe923
6a7de3ba3a
[Bugfix][GDN] Fix CuteDSL BF16 KKT inversion divergence ( #53864 )
...
Signed-off-by: Zoe923 <871844693@qq.com >
2026-09-17 08:49:56 -07:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
Wentao Ye and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
18f8aa0465
[Refactor] Remove dead kernel code ( #56845 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-15 13:17:53 -07:00
aed894c190
[6/N][warmup][DSv4] Migrate sampling, and DFlash JIT kernels ( #56323 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-12 17:24:24 -07:00
Jee Jee Li
199cb9b964
[Kernel] Remove unused fake implementation ( #55535 )
...
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
2026-09-06 17:54:32 -07:00
Jiangyun Zhu
98ed0856f3
[Model] add GLM-5.3-Flash support ( #53906 )
2026-09-03 09:40:35 -07:00
Andrii Skliar and Andrii Skliar
adebc41b7e
[Mamba] Add FlashInfer ReplaySSM backend ( #52506 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Signed-off-by: Andrii Skliar <andreyws96@gmail.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
2026-09-01 10:26:43 -04:00
Dinesh Chitlangia
525b7bbb3a
[Bugfix][CPU] Enable C++ causal_conv1d GDN path and float32 SSM cache on non-AMX AVX-512BF16 CPUs ( #49688 )
...
Signed-off-by: Dinesh Chitlangia <dineshchitlangia@gmail.com >
2026-08-19 10:50:59 -05:00
pmanczak
cc7cf71fc8
[XPU] Enable Kimi K3 KDA kernel tests on XPU ( #51809 )
...
Signed-off-by: pmanczak <pawel.manczak@intel.com >
2026-08-17 17:21:34 +08:00
pmanczak
c4e969294e
[XPU] Add tuned Mamba SSU configs for Intel Arc Pro B70 ( #50534 )
...
Signed-off-by: pmanczak <pawel.manczak@intel.com >
2026-08-13 12:36:55 +00:00
vanshbhatia-amd
e229fdbc87
[ROCm] Add tuned selective_state_update float32 config for AMD Instinct MI325X ( #50007 )
...
Signed-off-by: vanshbhatia-amd <vansh.bhatia@amd.com >
2026-08-07 12:58:36 -04:00
amitz-nv and OpenAI Codex
52c0e3cb08
[Kernel] Add support for Flashinfer Mamba SSU algorithm selection ( #50157 )
...
Signed-off-by: amitz-nv <203509407+amitz-nv@users.noreply.github.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-04 09:40:23 -07:00
vanshbhatia-amd
4689c7dd61
[ROCm] Add tuned selective_state_update float16 config for AMD Instinct MI325X ( #50006 )
...
Signed-off-by: vanshbhatia-amd <vansh.bhatia@amd.com >
2026-07-31 02:43:53 +00:00
Li, Jiang and Claude
072a472782
[CPU] Bump up CPU kernels to latest version ( #50387 )
...
Signed-off-by: jiang1.li <jiang1.li@intel.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-07-30 19:18:43 +08:00
+13
aeeb36b1f1
[New model] Kimi K3 ( #50000 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Summer Yang <girasoleyang@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Bowen Wang <abmfy@icloud.com >
Co-authored-by: gnovack <novackgm@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com >
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-07-30 03:49:58 -07:00
+12
7c6729b769
[Model] Add Kimi K3 support: model files and kernels [1/N] ( #50089 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Summer Yang <girasoleyang@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Bowen Wang <abmfy@icloud.com >
Co-authored-by: gnovack <novackgm@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com >
Co-authored-by: Codex <codex@openai.com >
2026-07-29 14:10:58 +08:00
Andreas Karatzas
30fbd05537
[ROCm] Use backend-default dot precision for ReplaySSM ( #49909 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-07-27 18:11:06 +08:00
544cb724c8
[CPU][Spec Decode] Optimize GDN conv path for speculative decoding ( #48577 )
...
Signed-off-by: Li, Tianmu <tianmu.li@intel.com >
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-07-27 06:20:14 +00:00
866fea2b99
[Kernel] ReplaySSM: cache SSM inputs for faster Mamba2 standard decode ( #48018 )
...
Signed-off-by: Johnny-Liou <a897111@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-07-24 09:39:49 -07:00
5c9f6557d7
[Hardware][CPU] Enable granite-4 model on cpu ( #47641 )
...
Signed-off-by: Akash Kaothalkar <akashkaothalkar@akashs-mbp.bl1-in.ibm.com >
Signed-off-by: Akash Kaothalkar <akashkaothalkar@dhcp-9-123-5-76.bl1-in.ibm.com >
Signed-off-by: Akash Kaothalkar <akashkaothalkar@Akashs-MBP.lan >
Signed-off-by: Akash kaothalkar <akash.kaothalkar@ibm.com >
Co-authored-by: Akash Kaothalkar <akashkaothalkar@dhcp-9-123-5-76.bl1-in.ibm.com >
Co-authored-by: Akash Kaothalkar <akashkaothalkar@Akashs-MBP.lan >
Co-authored-by: Akash Kaothalkar <akashkaothalkar@akashs-mbp.bl1-in.ibm.com >
Co-authored-by: Akash kaothalkar <akash.kaothalkar@ibm.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-07-20 06:15:16 +00:00
Thien Tran
fe784ff22e
[M3] Improve indexer for long-context decode (sm100) ( #48582 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-07-16 18:12:15 -07:00
Giuseppe Grossi
d119beb1b9
[ROCm] Add tuned selective_state_update config for AMD MI350 ( #48159 )
...
Signed-off-by: Giuseppe Grossi <ggrossi@amd.com >
2026-07-15 10:18:09 +03:00
vanshbhatia-amd and vanshbhatia-amd
7ffb98e248
[ROCm] Retune MI355 selective_state_update float32 config on the unified effective_batch grid ( #48373 )
...
Signed-off-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com >
Co-authored-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com >
2026-07-14 18:26:35 +00:00
vanshbhatia-amd and vanshbhatia-amd
31be872f55
[ROCm] Retune MI355 selective_state_update float16 config on the unified effective_batch grid ( #48372 )
...
Signed-off-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com >
Co-authored-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com >
2026-07-14 05:16:29 +00:00
05fa8183a6
[CPU][Spec Decode] Support DFlash speculative decoding for GDN models on CPU ( #46090 )
...
Signed-off-by: guybd <guy.boudoukh@intel.com >
Signed-off-by: Guy Boudoukh <guy.boudoukh@intel.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-07-13 04:16:18 +00:00
rasmith
934eeaecfb
[CI/Build][BugFix][The Rock] Fix get_ssm_device_name to return sanitized, usable filename ( #47781 )
...
Signed-off-by: Randall Smith <Randall.Smith@amd.com >
2026-07-08 12:12:54 +00:00
vanshbhatia-amd and vanshbhatia-amd
db39d60010
Add tuned selective_state_update float32 config for AMD Instinct MI355 ( #47943 )
...
Signed-off-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com >
Co-authored-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com >
2026-07-08 11:41:26 +00:00
vanshbhatia-amd and vanshbhatia-amd
eeaf23107f
[ROCm] Add tuned selective_state_update float32 config for AMD Instinct MI300X ( #47947 )
...
Signed-off-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com >
Co-authored-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com >
2026-07-08 11:09:02 +00:00
vanshbhatia-amd and vanshbhatia-amd
1f4ad059d1
[ROCm] Add tuned selective_state_update float16 config for AMD Instinct MI300X ( #47945 )
...
Signed-off-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com >
Co-authored-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com >
2026-07-08 11:03:58 +00:00
vanshbhatia-amd and vanshbhatia-amd
c8c2f838e7
Add tuned selective_state_update config for AMD Instinct MI355 ( #47767 )
...
Signed-off-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com >
Co-authored-by: vanshbhatia-amd <210711135+vanshbhatia-amd@users.noreply.github.com >
2026-07-07 22:19:08 +00:00
Thien Tran
43916891b2
[GDN] Improve kkt kernel of CuteDSL prefill backend ( #46346 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-06-29 18:34:18 -07:00
9af53a3c13
[Perf] Add tuned selective_state_update configs for H200 and RTX PRO … ( #44251 )
...
Signed-off-by: Majid Taheri Andani <tahemaji@amazon.com >
Co-authored-by: Majid Taheri Andani <tahemaji@amazon.com >
Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com >
2026-06-02 23:59:01 -07:00
Marceli Fylcek and Kunshang Ji
f69ede495b
[XPU][Mamba] Triton-based selective scan forward op for XPU ( #43421 )
...
Signed-off-by: Marceli Fylcek <marceli.fylcek@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-06-02 03:50:26 -07:00
Fadi Arafeh and Li, Jiang <jiang1.li@intel.com>
0b25cf4419
[CPU][Perf] Enable fused kernels for GDN's gated delta rules ( #43534 )
...
Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-06-02 08:00:48 +00:00