 
|
c825e1d235
|
[Perf] Fuse the mamba align block-table gather into one Triton kernel, 7 kernels -> 1, 6x speedup on all shapes for this kernel (#58737)
Signed-off-by: lifulu <fululi12@amd.com>
Signed-off-by: lifulu <lifulu@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: lifulu <lifulu@users.noreply.github.com>
|
2026-10-15 11:02:53 -05:00 |
|
Nicolò Lucchesi
|
30d4032363
|
[Model][GLM-5.3-Flash] Support DCP for the kpool sparse indexer (#59211)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
|
2026-10-05 16:44:14 +02:00 |
|
 
|
f7999d2e44
|
[Bugfix][HiSparse] Size the KV cache from the groups HiSparse allocates (#59450)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
|
2026-10-02 04:11:55 +00:00 |
|
 
|
cb6458c5ce
|
[Core] Remove deprecated mamba_cache_mode "all" (#58997)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Thomas Parnell <tpa@zurich.ibm.com>
|
2026-10-01 02:59:37 -07:00 |
|
 
|
927c87b347
|
[PCP][DCP] Support DCP target model with non-DCP Dspark (#56723)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
|
2026-09-26 18:12:35 -04:00 |
|
![mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>](/assets/img/avatar_default.png)     
|
e6dc16cebd
|
[PCP] Support prefill context parallelism with data parallelism (#57075)
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: QiuChunshuo <qiuchunshuo@huawei.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-09-23 22:32:36 -04:00 |
|
 Jiangyun ZhuandOpenAI Codex
|
6632ed559e
|
[Perf][Attention] Avoid CPU-GPU sync in DCP sequence lengths (#58169)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
|
2026-09-23 16:28:43 +08:00 |
|
 Harry MellorandClaude Opus 5
|
8c1557a79c
|
Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
2026-09-16 16:44:29 +00:00 |
|
  
|
241e9391ed
|
[Bugfix][Attention] Stabilize sparse-MLA DCP for GLM PCP evals (#55879)
Signed-off-by: khluu <khluu000@gmail.com>
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Jason Yao <wsyjh8@gmail.com>
|
2026-09-15 09:58:30 -07:00 |
|
   
|
82a85dc1d2
|
[Attention] Add Triton/FlashInfer composite for multimodal prefix attention (#56305)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-09-13 14:29:41 +00:00 |
|
Wentao Ye
|
5fe77aecfc
|
[Deprecation] Deprecate items scheduled for 0.29 (#55353)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
|
2026-09-11 08:17:02 -07:00 |
|
Nick Hill
|
e6cb56337b
|
[Core] MRV2 support for fast-prefill (#56145)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-09-10 10:36:04 -07:00 |
|
Wentao Ye
|
6b15bea080
|
[Refactor] Remove utils dead code (#53941)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
|
2026-09-08 11:22:09 -04:00 |
|
Nick Hill
|
4aab2b0ebe
|
[Model Runner V2] Use MRV2 for all models by default (#53183)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-08-27 11:59:34 -07:00 |
|
Misha Goin
|
f620499ee3
|
[Bugfix][Spec Decode] Reapply group geometry for FlashAttention metadata (#53336)
Signed-off-by: mgoin <mgoin64@gmail.com>
|
2026-08-24 08:08:43 -07:00 |
|
Lucas Wilkinson
|
8bdc70ec7b
|
[6/N][KV-Cache Layout Refactor] Standardize KV cache layout (#51718)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
|
2026-08-21 17:47:49 -07:00 |
|
Misha Goin
|
9ff7041b55
|
[Bugfix][Spec Decode] Use group geometry for FlashAttention metadata (#53002)
Signed-off-by: mgoin <mgoin64@gmail.com>
|
2026-08-21 17:07:19 -07:00 |
|
 yimdevandusberkeley
|
574e6a00ef
|
[Bugfix][Attention] Normalize FlashInfer prefill LSE before merging (#52796)
Signed-off-by: yimdev <5779256+yimdev@users.noreply.github.com>
Co-authored-by: usberkeley <150880684+usberkeley@users.noreply.github.com>
|
2026-08-21 11:31:14 -04:00 |
|
Nick Hill
|
f97e502969
|
[Perf] Avoid more GPU<->CPU syncs on the model execution path (#51738)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-08-11 19:30:34 -07:00 |
|
![mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>](/assets/img/avatar_default.png)  
|
66b3c0e61f
|
[Kernel][Model] Optimize FA4 mm_prefix range lookup (#50294)
Signed-off-by: Juqi Li <2223621784@qq.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-08-05 07:33:49 -07:00 |
|
 Michael GoinandOpenAI Codex
|
90329913e9
|
[UX] Reduce startup log noise (#50590)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
|
2026-07-31 13:04:34 -07:00 |
|
  
|
b6ff8a2f50
|
[Core] Add MRV2 virtual-batch PCP for MLA (#46570)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
|
2026-07-19 02:53:15 +00:00 |
|
![mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>](/assets/img/avatar_default.png) Wentao Yeandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
f17be06fbe
|
[Perf] Optimize clamp to clamp_ (#48143)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-07-16 18:41:07 -04:00 |
|
  
|
c5200d3565
|
[Attention][DSA] support dcp for FLASHINFER_MLA_SPARSE (#46076)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: Jingyi Yang <girasoleyang@gmail.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: GirasoleY <girasoleyang@gmail.com>
Co-authored-by: Jingyi Yang <girasoleyang@gmail.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
2026-07-01 00:32:20 -04:00 |
|
  
|
a7732537f4
|
[Bugfix] Restore part of bugfix #42650 after accidental deletion in #43241 (#47039)
Signed-off-by: zhanda <zhandazhu@gmail.com>
Signed-off-by: Nikita Shapovalov <nikita@poolside.ai>
Co-authored-by: Zhanda Zhu <49645678+zhandaz@users.noreply.github.com>
Co-authored-by: Shang Wang <shangw@nvidia.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
|
2026-06-30 11:07:59 -07:00 |
|
Nick Hill
|
7df3d7dada
|
[Core] Ensure memory is pinned prior to async h2d copy (#45424)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-06-20 20:02:24 -07:00 |
|
  
|
6deb05e0e4
|
[Core][Model] Gemma4: Unified FA4 for all layers + FlashAttention mm_prefix support (#42175)
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
|
2026-06-09 17:45:39 -07:00 |
|
Micah Williamson
|
c9c1540e61
|
[ROCm][V2] Fix failed assertion in Llama models when using EAGLE with ROCM_AITER_FA (#44936)
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
|
2026-06-09 13:30:52 -05:00 |
|
Giancarlo Delfin
|
ceb0111a90
|
[Model Runner V2][Spec Decode] Add Gemma4 MTP support (#43241)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-06-04 00:51:06 +00:00 |
|
 Zhanda ZhuandShang Wang
|
c7624bea5e
|
[Bugfix] Source num_qo_heads from Attention layers in Flashinfer/Triton metadata builders (#42650)
Signed-off-by: zhanda <zhandazhu@gmail.com>
Co-authored-by: Shang Wang <shangw@nvidia.com>
|
2026-05-22 16:10:03 +00:00 |
|
roikoren755
|
737bfa3a43
|
[Bugfix][Hybrid][NemotronH] Fix mamba_cache_mode=all + speculative decoding crash (#41233)
Signed-off-by: Roi Koren <roik@nvidia.com>
|
2026-05-18 14:54:00 +03:00 |
|
Nick Hill
|
989c176c0a
|
[Perf][3/n] Eliminate GPU<->CPU syncs in attention impls (#41434)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-05-07 19:44:24 -07:00 |
|
+7        
|
4d51588e23
|
[Feat] DeepSeek V4 Rebased (#40860)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: qizixi <zixi@inferact.ai>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <yongye@inferact.ai>
Co-authored-by: Simon Mo <simon@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roy Wang <yasong.wang@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com>
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com>
Co-authored-by: khluu <khluu000@gmail.com>
Co-authored-by: qizixi <zixi@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
|
2026-04-26 18:31:08 -07:00 |
|
Nick Hill
|
fe85a92e86
|
[Core] Avoid seq_lens_cpu GPU->CPU sync (#40654)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-04-24 00:35:55 +00:00 |
|
 
|
bcc6f67447
|
[Bugfix] Use null block (0) for padded block table entries (#35431)
Signed-off-by: SandishKumarHN <sandish@fb.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
|
2026-03-30 14:02:51 -07:00 |
|
Or Ozeri
|
7cc302dd87
|
[kv_offload+HMA][7/N]: Support register_kv_caches for hybrid models (#37853)
Signed-off-by: Or Ozeri <oro@il.ibm.com>
|
2026-03-27 08:38:33 +03:00 |
|
Chuan (Richard) Li
|
cb2263218e
|
[Bugfix][Minor] Fix potential NameError in mamba backend selector and misc typos (#35886)
Signed-off-by: Li <chuali@amd.com>
|
2026-03-26 11:59:24 -04:00 |
|
Lucas Wilkinson
|
e1d85e5c24
|
[Attention] Support distinguishing between short extends and decodes (#37303)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
|
2026-03-20 10:49:36 -07:00 |
|
Woosuk Kwon
|
006aea17d7
|
[BugFix] Remove incorrect assert in split_decodes_and_prefills (#36553)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-09 20:02:02 -07:00 |
|
 Huamin LiandCyrus Leung
|
1fe462168c
|
[perf] Avoid dtype promotion sync in mamba_get_block_table_tensor (#34870)
Signed-off-by: Huamin Li <3ericli@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
|
2026-02-20 06:21:56 -08:00 |
|
Woosuk Kwon
|
0916e7960b
|
[GDN] Use CPU tensors to build GDN metadata (#34498)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-13 01:24:45 -08:00 |
|
Benjamin Chislett
|
af3162d3aa
|
[Spec Decode] Unified Parallel Drafting (#32887)
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
|
2026-02-05 12:37:18 -05:00 |
|
 Harry HuangandChen Zhang
|
5206e5e28c
|
[V1][Hybrid] Mamba Prefix Caching with align mode (#30877)
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
|
2026-01-23 09:56:48 -08:00 |
|
tianshu-Michael-yu
|
13d8746c54
|
[Feature]: Remove DtoH Copy for lfm2_vl On Default Stream (#32815)
Signed-off-by: Tianshu Yu <tianshuyu.formal@gmail.com>
|
2026-01-23 13:20:30 +00:00 |
|
Tomas Ruiz
|
4a5299c93f
|
feat: spec decode with draft models (#24322)
Signed-off-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
|
2026-01-19 16:05:46 -05:00 |
|
Matthew Bonanni
|
98f60e5acb
|
[6/N][Attention] Move utils to more appropriate locations (#32215)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
|
2026-01-13 05:38:52 -08:00 |
|
Matthew Bonanni
|
20228cb851
|
[3/N][Attention] Move AttentionMetadata-related code from utils.py to backend.py (#32054)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
|
2026-01-12 09:13:56 -08:00 |
|
Asaf Joseph Gardin
|
8fb2c135be
|
[Bugfix] Fix stale SSM state for new Mamba requests scheduled as decode (#32118)
Signed-off-by: Josephasafg <ajgard7@gmail.com>
|
2026-01-12 17:02:38 +00:00 |
|
Lucas Wilkinson
|
da6709c9fe
|
[Misc] Delay deprecation of CommonAttentionMetadata properties (#32074)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
|
2026-01-09 21:06:44 -08:00 |
|
Matthew Bonanni
|
2612ba9285
|
[1/N][Attention] Restructure attention: move files (#31916)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
|
2026-01-09 13:10:24 -08:00 |
|