72 Commits
Author SHA1 Message Date
Wei Zhao 90d6d871d6 [Attention] Add CuTe DSL backend selection for MLA context prefill (#62069)
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
2026-10-17 01:21:55 -04:00
Wei ZhaoandCodex be25977532 [CI][Tests] Account for FP8 KV-cache rounding in MLA reference (#62067)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-16 17:10:08 -07:00
Oxana KorzhandClaude Opus 5.5 5bf3d9f665 [ROCm][Kimi-K3] Support DSpark adaptive verification on MI355X (#59824)
Signed-off-by: Oxana Korzh <okorzh@amd.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 13:53:49 -05:00
81e1c3f183 [XPU][MLA] Integrate XPU MLA support into the existing FLASH_ATTN_MLA backend (#58297)
Signed-off-by: baodii <di.bao@intel.com>
Signed-off-by: Zhouxiaoya25 <xiaoya.zhou@intel.com>
Signed-off-by: ZhouXiaoya25 <xiaoya.zhou@intel.com>
Co-authored-by: baodii <di.bao@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-10-15 12:47:37 +08:00
Wei ZhaoandCodex 03a7d3776f [Perf] Support minimum KV splits configuration for TokenSpeed MLA decode (#60345)
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-09 23:33:47 +00:00
Nicolò LucchesiandLeoyzen ff53f32409 [Attention][MLA] Support fp8_ds_mla KV cache for NoPE-512 models on SM90 (#59246)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Co-authored-by: Leoyzen <leoyzen@gmail.com>
2026-10-05 13:12:00 +00:00
Wei Zhaoandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 84bcbc6264 [DCP] Enable TokenSpeed MLA with block-interleaved DCP (#59462)
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-03 17:26:07 +00:00
lzOpenAI CodexLucas WilkinsonRobert Shawmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
29468dde8b [Bugfix][KV Cache][MLA] Align packed block strides for V3.2 sparse MLA (#55528)
Signed-off-by: lz <145014769+200lz@users.noreply.github.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-25 05:33:36 +00:00
Albert Chengmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Benjamin Chislett
88aa0d287d [Spec decode] Support variable-length decode for Kimi-K3 adaptive ver (#52988)
Signed-off-by: Albert Cheng <albecheng@nvidia.com>
Signed-off-by: Albert Cheng (Engrg-Hardware 1) <albecheng@login-bia01.bia.clusters.nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com>
2026-09-22 20:43:32 -07:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
Jared WenClaude Fable 5.1mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
238cb2b191 [Perf][GLM-5.3-Flash] Dense/masked-MHA sparse prefill for the NoPE (256, 0, 256) layout + skip the NoPE K concat (#55738)
Signed-off-by: Jared Wen <jaredwen@inferact.ai>
Signed-off-by: Jared Wen <w13431838023@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-14 09:36:39 +00:00
d43bb2f37f [3/N] HiSparse: host-resident sparse-MLA decode hot-buffering (#53781)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Matthew Bonanni <matthew@neuralmagic.com>
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
2026-09-12 00:36:47 -04:00
sychen52 7c5dc571cb [Attention][DSA] Enable W4A16 DSA (#51724)
Signed-off-by: Shiyang Chen <shiychen@nvidia.com>
2026-09-01 12:37:47 +08:00
Summer Yang 6d4562c59b [Attention][DCP] Enable FlashInfer MLA for DSpark drafting (#54277)
Signed-off-by: Summer Yang <girasoleyang@gmail.com>
2026-08-28 21:11:28 -07:00
Matthew BonanniandOpenAI Codex de9250ac9e [Attention] Enable masked MHA for GLM-5 head dimensions (#53785)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-08-27 21:17:12 +00:00
Summer Yang 07242faa46 [Attention][DCP] Use FlashInfer native CP for MLA decode (#54012)
Signed-off-by: Summer Yang <girasoleyang@gmail.com>
2026-08-27 10:32:45 -07:00
Andreas Karatzas 30b34171b1 [ROCm][CI] Restore attention coverage after KV-cache layout refactor (#53351)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
2026-08-22 20:00:25 -05:00
Lucas Wilkinson 8bdc70ec7b [6/N][KV-Cache Layout Refactor] Standardize KV cache layout (#51718)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
2026-08-21 17:47:49 -07:00
Summer Yang d6c2fec9fd [Cleanup][MLA] Remove FlashInfer DSpark DCP support (#53139)
Signed-off-by: Summer Yang <girasoleyang@gmail.com>
2026-08-21 11:42:52 -07:00
yimdevandusberkeley 574e6a00ef [Bugfix][Attention] Normalize FlashInfer prefill LSE before merging (#52796)
Signed-off-by: yimdev <5779256+yimdev@users.noreply.github.com>
Co-authored-by: usberkeley <150880684+usberkeley@users.noreply.github.com>
2026-08-21 11:31:14 -04:00
Li, Jiang e4d61d0d22 [CPU] Add AMX-only high-performance MLA backend for DeepSeek V2/V3/R1 (#52616)
Signed-off-by: jiang1.li <jiang1.li@intel.com>
2026-08-19 10:28:52 +08:00
Summer Yangmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Yifan Qiao
0db502c8d8 [Kimi-K3] support DCP partial prefix cache hit (#50493)
Signed-off-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
2026-08-17 17:05:51 -07:00
Wei Zhao d1e3eee6fb [Spec decode] Support Kimi-K3 DCP with DSpark (#52188)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
2026-08-17 13:08:45 -07:00
9035151d6c [Model] Add native Dots3 NOTE multimodal support (#51255)
Signed-off-by: KurodaKanbei <mistergalahad@gmail.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
2026-08-12 23:13:20 +08:00
Matthew BonanniClaude Opus 5OpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
b38e111d3e [Attention][MLA] Per-request scheduling for MLA chunked context (#50613)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-06 16:45:44 +00:00
NVShreyasandOpenAI Codex 8ae8337ffa [Spec Decode] Enable fused non-causal TokenSpeed MLA for DSpark (#50911)
Signed-off-by: Shreyas Misra <shreyasm@nvidia.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-08-04 18:05:40 +00:00
Matthew Bonanni a0cd2b69b3 [Bugfix] Universally align block table width to 128 tokens (#50302)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
2026-07-31 08:52:17 -07:00
+13 aeeb36b1f1 [New model] Kimi K3 (#50000)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Isotr0py <mozf@inferact.ai>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Bowen Wang <abmfy@icloud.com>
Co-authored-by: gnovack <novackgm@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com>
Co-authored-by: Roy Wang <yasong.wang@inferact.ai>
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
2026-07-30 03:49:58 -07:00
Andreas Karatzas ac87549cbd [CI][ROCm] Reduce V1 attention test runtime (#49916)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
2026-07-27 11:12:45 +08:00
2396a61108 [Attention][MLA][DCP] Query replication for MLA decode (DeepSeek-V2/R1 + Kimi-K2.5) (#45964)
Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
2026-07-20 23:51:27 +00:00
aoshen02 17fdd42100 [Bugfix][Attention] Preserve post-load tensors across weight reloads (#48251)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
2026-07-17 14:15:26 +08:00
Matthew BonanniOpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
67f9046e4a [Bugfix] Sparse MLA: enable fp8_ds_mla dense prefill (#48642)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-16 22:44:04 +00:00
Matthew BonanniandClaude Opus 4.8 c4f5cd60da [1/N] Add dense MHA path for sparse MLA short sequences (#47327)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-14 00:29:56 +00:00
Pavani MajetyandJingyi Yang 7fc97042c3 Add DCP + Eagle support for Tokenspeed MLA backends (#48180)
Signed-off-by: Pavani Majety <pmajety@nvidia.com>
Signed-off-by: Jingyi Yang <girasoleyang@gmail.com>
Co-authored-by: Jingyi Yang <girasoleyang@gmail.com>
2026-07-13 11:46:02 -07:00
shivamprMatthew Bonannimergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
49f2104c53 [Feature] Support DCP with FP8 KV cache in MLA decode path (#44044)
Signed-off-by: shivampr <shivampr.dev@gmail.com>
Signed-off-by: Shivam <shivamprasad91@gmail.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-24 19:28:17 +00:00
Mohammad Miadh AngkadandMatthew Bonanni 9d4dc4ca2f [Kernel] Support GLM-5 dimensions for TRT-LLM ragged MLA prefill (#43525)
Signed-off-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
2026-06-16 20:49:47 +00:00
Yongye ZhuandRoger Wang 0d2732dd91 [MLA Attention Backend] Add TOKENSPEED_MLA backend for DSR1/Kimi K25 prefill + decode on Blackwell (#41778)
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roger Wang <hey@rogerw.io>
2026-05-13 23:48:02 -07:00
Matthew Bonanni 01b9b5af67 [Attention] Minor refactor: layer takes ownership of the MLA prefill backend (#41744)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
2026-05-05 23:22:41 +00:00
Andreas Karatzas 1cb0838721 [ROCm][CI] Fix MLA prefill scale for DeepSeek GSM8K (#41569)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
2026-05-04 16:32:55 -07:00
Matthew BonanniMichael GoinLucas Wilkinsonmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f3fef12350 [Attention] Abstract the MLA prefill backends and eliminate cuDNN (#32623)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-01 13:36:20 -04:00
wliao2 32e0c0bfa2 refactor hard coded device string in test files under tests/v1 and tests/lora (#37566)
Signed-off-by: Liao, Wei <wei.liao@intel.com>
2026-04-03 11:21:47 +08:00
Andy Lo 577df69b26 [Bugfix] Fix KV scales inconsistency in fp8 MLA & FlashInfer kv_cache_dtype "auto" leading to gibberish (#37054)
Signed-off-by: Andy Lo <andy@mistral.ai>
2026-03-18 23:07:29 +00:00
Wei Zhao 379689d533 [Perf] Support FP8 KV cache for Flashinfer MLA Sparse (#35891) 2026-03-07 13:51:54 -08:00
Wei Zhao f24b2de3d3 [Test] Add FP8 KV Cache Testing for MLA Backends (#34473)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
2026-02-20 18:51:58 +00:00
Pavani Majety 578977bb5e [SM100] Resubmit FMHA FP8 prefill for MLA (#31195)
Signed-off-by: Pavani Majety <pmajety@nvidia.com>
2026-02-10 16:18:43 -05:00
Matthew Bonanni aaa901ad55 [Attention] Move MLA forward from backend to layer (#33284)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
2026-01-30 19:30:00 -08:00
Vadim Gimpelson 0727cc9ecf [BUGFIX] Fix test_mla_backends.py. Scale MLA projection weights to prevent numerical instability (#32529)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
2026-01-19 13:49:29 -05:00
Matthew BonanniandLuka Govedič 2263d44b68 [4/N][Attention] Move MLA common to model_executor (#32060)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
2026-01-13 09:08:45 -08:00
Matthew Bonanni 20228cb851 [3/N][Attention] Move AttentionMetadata-related code from utils.py to backend.py (#32054)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
2026-01-12 09:13:56 -08:00