MOVZX and Hinata
281491646d
[Worker] E2-identity and rank-tagged e2-toggles startup lines
...
Each worker logs its launch identity and, after warm-up and capture, one
e2-toggles line with the effective toggle state read back from the C++
extensions through two new tensor-free query ops. Both extensions must be
rebuilt.
Co-Authored-By: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-09 19:56:00 +07:00
MOVZX and Hinata
cff46af086
[TopK] Follow-up fixes to the persistent top-k kernels
...
- Counter enum declared above the decode template (two-phase lookup).
- Emission-fault flag set atomically.
- Close the legacy large tie loop with a block barrier (inherited).
- Large-path fault injectors registered as hashed passthroughs.
- The tensor-free counter-slots op registered for CompositeExplicitAutograd.
- Tie measurements recorded only on rows neither fault verdict failed.
- Tests scoped to the properties the build promises.
Co-Authored-By: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-09 19:56:00 +07:00
MOVZX and Hinata
2918e3fda1
[TopK] Determinism toggles for greedy repeatability
...
- MERLIN_FULL_K: deterministic Marlin MoE split-K (after unmerged upstream #46639 ).
- MERLIN_QSA_SORT: canonical order for the QSA selected blocks.
- MERLIN_TIE_DET: deterministic tie-break at the persistent_topk cutoff.
- MERLIN_TIE_RECENT: cutoff ties keep the highest indices.
- Decode candidate truncation is detected; one tie direction across the
medium path; the kernel switches are registered.
Co-Authored-By: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-09 19:56:00 +07:00
MOVZX and Hinata
754fb3e29d
[TopK] Exact restart of the medium path on candidate-buffer overflow
...
When the persistent top-k medium-path candidate buffer overflows, the row is
restarted exactly instead of being truncated. Capture-safe counters with atomic
publication and a coherent snapshot record every restart.
Co-Authored-By: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-09 19:43:58 +07:00
Yongye Zhu and Claude Opus 5.5
7566d83bd3
[Attention][MiniMax-M3] NVFP4 KV cache on the MSA sparse attention path ( #59300 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 14:39:55 -07:00
Matthew Bonanni and Claude Opus 5.5
2798f66860
[Bugfix][HiSparse] Resolve MTP verification rows with a union residency kernel ( #59235 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 03:01:45 +00:00
Misha Goin
25965ee583
[Kernel] Remove AllSpark INT8 W8A16 GEMM backend ( #58001 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
2026-09-22 14:07:34 -07:00
d6a1677d55
[Model][DSv4.1] FlashMLA mega attention and the NVFP4 compressed KV cache ( #56935 )
...
Signed-off-by: Yongye Zhu <yongye@inferact.ai >
Signed-off-by: 云挚 <ningyunxiao.nyx@antgroup.com >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <yongye@inferact.ai >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Yunxiao Ning <73625538+foraxe@users.noreply.github.com >
Co-authored-by: AI Assistant <noreply@openai.com >
2026-09-16 03:07:50 -07:00
6ecd97f1b7
[Perf][DSpark] Add KV-only context insertion across V4.1 cache formats ( #56441 )
...
Signed-off-by: 0z5a <0z5a@users.noreply.github.com >
Signed-off-by: Yongye Zhu <yongye@inferact.ai >
Co-authored-by: 0z5a <0z5a@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-15 21:34:55 -07:00
e6eb0d120c
[Model][DSv4.1] Store the whole KV in MXFP8 (FlashMLA V4.1 record) ( #56893 )
...
Signed-off-by: Yongye Zhu <yongye@inferact.ai >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <yongye@inferact.ai >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-14 20:12:55 -07:00
29332cf936
[4/N] Expose HiSparse cache metrics via KV connector stats ( #56061 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-12 10:47:55 -07:00
d43bb2f37f
[3/N] HiSparse: host-resident sparse-MLA decode hot-buffering ( #53781 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Signed-off-by: Matthew Bonanni <matthew@neuralmagic.com >
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-09-12 00:36:47 -04:00
Wentao Ye
1e1060f998
[Kimi Perf] Group fp8 mla cahche insertion, 4~6x kernel level performance improvement for small batch ( #55356 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-11 08:16:57 -07:00
9521c60bdc
[ROCm][Perf] Kimi-K3 Fused kernels for KDA prefill reland ( #54038 )
...
Signed-off-by: kliuae <kuanfu.liu@embeddedllm.com >
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com >
Signed-off-by: kliuae <17350011+kliuae@users.noreply.github.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: TJian <tunjian.tan@embeddedllm.com >
2026-09-10 15:37:54 +08:00
Yongye Zhu and Claude Opus 5
be1cb9834b
[Kernel] Optional Q-norm in fused DSv4 MLA epilogue; group_size=32 for packed FP8 quant ( #56215 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-10 00:35:59 -07:00
Roderick Wu and mgoin
2c9d68f80b
[Quant][Kernel] Remove GPTQ Group/Dynamic Activation Ordering ( #54809 )
...
Signed-off-by: Roderick-Wu <roderickwu2003@gmail.com >
Signed-off-by: Roderick Wu <roderickwu2003@gmail.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
2026-09-08 12:45:12 -07:00
Misha Goin and OpenAI Codex
82b7d49a6e
[MoE] Generalize masked activation for padded layouts ( #51217 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-01 11:34:13 -04:00
sychen52
7c5dc571cb
[Attention][DSA] Enable W4A16 DSA ( #51724 )
...
Signed-off-by: Shiyang Chen <shiychen@nvidia.com >
2026-09-01 12:37:47 +08:00
Yongye Zhu
d6d6658543
[Kimi-K3][Perf] Make native CUDA AttnRes the SM100 default ( #54261 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
2026-08-31 23:18:12 +00:00
Jee Jee Li
e2c8eeac40
[kernel] Fused embedding kernel ( #53677 )
...
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
2026-08-31 11:34:14 +00:00
+5
e126687a9a
[Model] Support Qwen3.8-Flash-Next ( #53896 )
...
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com >
Signed-off-by: andyluo7 <andy.luo@amd.com >
Signed-off-by: lishunyang12 <lishunyang03@gmail.com >
Signed-off-by: Harry Huang <huanghaoyan.hhy@alibaba-inc.com >
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: KungYork <firelikest@163.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: andyluo7 <43718156+andyluo7@users.noreply.github.com >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: andyluo7 <andy.luo@amd.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: lishunyang12 <lishunyang03@gmail.com >
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-31 05:57:56 +00:00
ca29cd48ba
[Quantization][Humming] Support MXFP4 weight + block-FP8 activation for MoE ( #51332 )
...
Signed-off-by: Elvir Crncevic <elvircrn@gmail.com >
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Signed-off-by: Elvir Crnčević <elvircrn@gmail.com >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-25 10:21:00 -04:00
Kevin H. Luu and Andreas Karatzas
592e06f2ae
Revert "[ROCm][Perf] Kimi-K3 Fused kernels for KDA prefill" ( #53294 )
...
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-08-21 13:34:13 -07:00
463aa5e30f
[ROCm][Perf] Kimi-K3 Fused kernels for KDA prefill ( #52606 )
...
Signed-off-by: kliuae <kuanfu.liu@embeddedllm.com >
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: TJian <tunjian.tan@embeddedllm.com >
2026-08-21 09:07:13 -07:00
Woosuk Kwon and OpenAI Codex
f1178f3a06
Revert DSv4 eager workspace reuse ( #52836 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-18 17:02:59 -07:00
1be3628367
[Kernel][Perf] Add fused CUDA post-conv MTP decode kernel for Qwen3.5 GDN ( #51674 )
...
Signed-off-by: Jie Fang <jief@nvidia.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-08-14 11:44:45 +08:00
903d2efe7e
[Attention][MLA] Fuse Kimi-K3 chunked-context K/V packing ( #51772 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-12 22:42:57 -07:00
Jason Li
0055b8bfa3
[Attention][MiniMax-M3] Add MSA speculative decode verification ( #50032 )
2026-08-02 13:46:55 -07:00
Wentao Ye and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
df71917cf1
[DSv4 Perf] Optimize workspace reuse for eager break, 3.9% E2E TTFT improvement. ( #49236 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-31 08:45:58 -07:00
+12
7c6729b769
[Model] Add Kimi K3 support: model files and kernels [1/N] ( #50089 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Summer Yang <girasoleyang@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Bowen Wang <abmfy@icloud.com >
Co-authored-by: gnovack <novackgm@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com >
Co-authored-by: Codex <codex@openai.com >
2026-07-29 14:10:58 +08:00
Thien Tran
61ac368021
[Kimi-K3] Add AttnRes kernels ( #50090 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-07-28 00:22:09 -07:00
Woosuk Kwon
213f681f81
Revert "[Perf][GLM-5.2] Blackwell decode optimizations" ( #49768 )
2026-07-24 16:04:59 -07:00
dd72658e7d
[Perf][GLM-5.2] Blackwell decode optimizations ( #48597 )
...
Signed-off-by: Peiyuan Zhou <peiyuanzhou1994@gmail.com >
Signed-off-by: zhou <zhou@zhoudeMacBook-Pro.local >
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: zhou <zhou@zhoudeMacBook-Pro.local >
Co-authored-by: stefankoncarevic <Stefan.Koncarevic@amd.com >
2026-07-23 21:36:27 -07:00
67f9046e4a
[Bugfix] Sparse MLA: enable fp8_ds_mla dense prefill ( #48642 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-16 22:44:04 +00:00
ee5a89f4d7
[ROCm][MiniMax-M3] Add AITER sparse paged attention ( #47287 )
...
Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com >
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com >
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com >
Co-authored-by: Hongxia Yang <hongxia.yang@amd.com >
Co-authored-by: Jun Kang Chow <junkangchow@gmail.com >
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com >
2026-07-12 19:27:29 -07:00
Tanish Malekar and Claude Sonnet 4.6
e26264f3ef
[Kernel] Implement CUDA kernel for ReLUSquaredActivation (relu^2) ( #39058 )
...
Signed-off-by: Tanish Malekar <tanishmalekar32@gmail.com >
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-07-12 19:18:03 -07:00
Michael Goin
08dfd68610
[Model] Add LongCat-Flash-Lite (n-gram embedding) ( #47857 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
2026-07-10 07:17:50 -07:00
Michael Goin
d715b3aa1e
Delete PagedAttention ( #47361 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
2026-07-02 12:31:26 -07:00
TJian
de2a8fc042
[ROCm] [PyTorch] Move to stable abi since ROCm upgraded to torch 2.11 ( #47128 )
...
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com >
2026-07-02 18:34:07 +08:00
Wentao Ye
9e84ec8648
[Refactor] Remove dead minimax allreduce rms kernel ( #46842 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-06-30 08:29:21 -07:00
Roberto L. Castro
855cd4d787
[Perf][DSv4/DSv3.2] Add cluster-cooperative topK kernel for low-latency scenarios ( #43008 )
...
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
2026-06-23 16:11:00 -07:00
Chris Leonard
b9a7cd464c
[12/n] final _C library kernel migration ( #45415 )
2026-06-19 06:57:26 -07:00
Wentao Ye and Shengqi Chen
b4c80ec0fd
[Refactor] Remove dead cutlass mxfp8 code ( #44681 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Co-authored-by: Shengqi Chen <harry-chen@outlook.com >
2026-06-17 21:18:25 -07:00
arghyadeep sarkar
93bbe94d3a
[Kernel] Add weightless RMSNorm CUDA kernels for has_weight=False ( #41430 ) ( #44109 )
...
Signed-off-by: hello-args <args.sarkar@gmail.com >
2026-06-16 23:45:55 -07:00
Thien Tran and Yongye Zhu
4c62663315
[M3] Enable FP8 sparse GQA ( #45744 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
2026-06-16 21:38:03 -07:00
0a1c5034f5
[Model] Add MiniMax M3 support ( #45381 )
...
Signed-off-by: youkaichao <youkaichao@gmail.com >
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com >
Signed-off-by: functionstackx <47992694+functionstackx@users.noreply.github.com >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
2026-06-16 01:01:25 +08:00
Isotr0py
6635279d8a
[Migration] Migrate GGUF quantization support to plugin ( #39612 )
...
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
2026-06-12 12:02:21 -07:00
Chris Leonard and Shengqi Chen
a014dddbaa
[11b/n] Migrate Machete kernels to torch stable ABI ( #45304 )
...
Signed-off-by: Chris Leonard <chleonar@redhat.com >
Signed-off-by: Shengqi Chen <harry-chen@outlook.com >
Co-authored-by: Shengqi Chen <harry-chen@outlook.com >
2026-06-12 10:36:49 +00:00
Chris Leonard
7021be66e8
[11a/n] Migrate Marlin kernels to torch stable ABI ( #45176 )
...
Signed-off-by: Chris Leonard <chleonar@redhat.com >
2026-06-11 21:22:37 -07:00
Chris Leonard and Shengqi Chen
6e64c1bab1
[10c/n] Migrate MoE kernels to torch stable ABI ( #44565 )
...
Signed-off-by: Chris Leonard <chleonar@redhat.com >
Co-authored-by: Shengqi Chen <harry-chen@outlook.com >
2026-06-10 23:02:26 -07:00