Jiahan Chang (Cyrus) and OpenAI Codex
4e5ffda19d
[Bugfix] Isolate supplemental FlashInfer BF16 autotuning ( #57285 )
...
Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-17 07:38:58 +00:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
Roberto L. Castro and Codex
e52be1a62d
[9/N][warmup][DSv4] Migrate MHC TileLang kernels ( #50178 )
...
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Signed-off-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com >
Co-authored-by: Codex <codex@openai.com >
2026-09-12 17:25:23 -07:00
aed894c190
[6/N][warmup][DSv4] Migrate sampling, and DFlash JIT kernels ( #56323 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-12 17:24:24 -07:00
7ee8a6dd01
[watermarking] Dual-key gumbel-max watermarking for speculative decoding support ( #56122 )
...
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai >
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Signed-off-by: Tomas Ruiz <tomas.ruiz.te@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Tomas Ruiz <tomas.ruiz.te@gmail.com >
2026-09-12 18:42:01 +00:00
d43bb2f37f
[3/N] HiSparse: host-resident sparse-MLA decode hot-buffering ( #53781 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Signed-off-by: Matthew Bonanni <matthew@neuralmagic.com >
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-09-12 00:36:47 -04:00
cc4210f671
[Perf] Improve BF16x3 router GEMM accuracy and make it default on sm100 ( #55899 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
2026-09-09 07:33:12 -07:00
7fa2c63796
[Bugfix] Autotune FlashInfer deferred MoE decode kernels before CUDA graph capture ( #55377 )
...
Signed-off-by: ming <ming@inferact.ai >
Co-authored-by: Kimi Code <kimi-code@moonshot.cn >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-08 04:02:23 +00:00
42801b3a6b
[3/N][warmup][DSv4] Migrate FA4 MLA and shared CuTeDSL kernels ( #53565 )
...
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-07 09:36:35 -04:00
vhagor and Isotr0py
3dc7a68ce4
[Perf] Extend Qwen Triton warmup to avoid first-request latency spikes ( #54797 )
...
Signed-off-by: Juqi Li <2223621784@qq.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-09-07 11:46:00 +08:00
Jiangyun Zhu
98ed0856f3
[Model] add GLM-5.3-Flash support ( #53906 )
2026-09-03 09:40:35 -07:00
Gabriel Wu and Kimi Code
396c5a5632
[Bugfix] Lazy-import FlashInfer PCIe IPC all-reduce in kernel_warmup ( #54869 )
...
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
Co-authored-by: Kimi Code <noreply@moonshot.cn >
2026-09-02 04:31:03 +00:00
Yongye Zhu and OpenAI Codex
3ba9907a1d
[Kimi-K3] Overlap low-M TP8 KDA projections ( #54697 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-02 03:35:28 +00:00
Thien Tran
003e34341a
[Qwen3.8-Flash-Next] Separate prefill and decode paths for QSA indexer ( #54513 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-09-02 09:08:10 +08:00
Andrii Skliar and Andrii Skliar
adebc41b7e
[Mamba] Add FlashInfer ReplaySSM backend ( #52506 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Signed-off-by: Andrii Skliar <andreyws96@gmail.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
2026-09-01 10:26:43 -04:00
446c769482
[Distributed] Add opt-in FlashInfer PCIe IPC all-reduce backend ( #53576 )
...
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-01 03:34:56 +00:00
Roberto L. Castro and Codex
e16b5e518d
[1/N][warmup][DSv4] Migrate generic MLA metadata and indexing kernels ( #50175 )
...
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Co-authored-by: Codex <codex@openai.com >
2026-08-31 23:26:05 -04:00
7c877062ac
[kernel] Integrate FlashInfer BF16 CuTeDSL Low Latency GEMM ( #50572 )
...
Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
2026-08-27 22:29:12 +00:00
Ning Xie
6648eb118d
[Core] drop duplicate VLLM_USE_DEEP_GEMM check ( #48687 )
...
Signed-off-by: Andy Xie <andy.xning@gmail.com >
2026-08-24 16:30:50 +00:00
2785c72a14
[Kimi-K3] Extend GEMM-RS to GEMM-AR ( #53053 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: wzhao18 <wzhao18.sz@gmail.com >
2026-08-21 03:44:15 +00:00
Roberto L. Castro and Codex
49905ad94d
[3/N][Feat][Perf] Add new warmup infrastructure for JITs. Add provider registry and orchestration for JIT warmup ( #50174 )
...
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Co-authored-by: Codex <codex@openai.com >
2026-08-17 10:11:14 -07:00
Luke Alonso
3c79b1a8bf
[Kernel] Add B12X dense linear backends ( #52016 )
...
Signed-off-by: Luke Alonso <lalonso@gmail.com >
2026-08-13 19:48:33 -07:00
f18e10a7e1
Bump Flashinfer version to 0.6.16.post3 ( #50892 )
...
Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com >
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: jiahanc <173873397+jiahanc@users.noreply.github.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-08-09 00:54:07 -07:00
Michael Goin and OpenAI Codex
90329913e9
[UX] Reduce startup log noise ( #50590 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-07-31 13:04:34 -07:00
+13
aeeb36b1f1
[New model] Kimi K3 ( #50000 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Summer Yang <girasoleyang@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Bowen Wang <abmfy@icloud.com >
Co-authored-by: gnovack <novackgm@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com >
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-07-30 03:49:58 -07:00
Nick Hill
0d0504b54c
[Core] Warm up runner-owned Triton kernels before the first request ( #49903 )
2026-07-28 07:58:02 -07:00
Itay Alroy
35efdf6b34
[Elastic EP] Async preparation ( #47288 )
...
Signed-off-by: Itay Alroy <ialroy@nvidia.com >
2026-07-28 05:19:13 -07:00
Roberto L. Castro
56c96b0d91
[Perf] Tune LL BF16 Router GEMM ( #48774 )
...
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Signed-off-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com >
2026-07-27 10:25:37 -04:00
neweyes
96fa3f42c9
[Perf] Skip ll_bf16 router GEMM warmup for non-MoE models ( #49659 )
...
Signed-off-by: neweyes <328719365@qq.com >
2026-07-27 05:16:42 -07:00
Roberto L. Castro and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
7ca017778f
[Feat][Perf] Add new warmup infrastructure for JITs ( #47451 )
...
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Signed-off-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-20 13:21:55 -07:00
6570c9800c
[Model] Add Inkling model support [1/N] ( #48799 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-07-15 23:40:07 -07:00
Roberto L. Castro and Lucas Wilkinson
0762f2afeb
[Perf][Feat] Add generic cuteDSL LL BF16 router (GEMM) ( #42562 )
...
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
2026-07-13 23:01:21 -07:00
Michael Goin and Claude Fable 5
29fd688892
Add VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS and skip CuTeDSL fp4_gemm autotuning by default ( #48268 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Signed-off-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-07-10 18:13:08 -07:00
Harry Mellor
ae098abe3f
[CI] Fix some errors on main ( #47726 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-06 19:40:23 +00:00
Roberto L. Castro and Lucas Wilkinson
248d1fbb71
[Feat][1/N] CuTeDSL warmup infrastructure, FA4 MLA ( #46182 )
...
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Signed-off-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-06-30 12:17:34 -07:00
Roberto L. Castro and Lucas Wilkinson
030c9523bd
[Perf][1/N] Expand Triton kernel warmup coverage, DSv4 ( #46634 )
...
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Signed-off-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-06-29 16:40:34 +00:00
Wei Zhao and Jee Jee Li
4708292d48
Bump flashinfer version to 0.6.13 ( #46683 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
2026-06-29 09:30:57 -07:00
Roberto L. Castro
eddfd4cf21
[Perf][2/N] Expand Triton kernel warmup coverage, Qwen ( #46750 )
...
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
2026-06-29 10:10:07 +00:00
44d95069e9
Enable DeepSeek V4 and GLM-5.1 on SM120 ( #43477 )
...
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
2026-06-22 11:54:14 -07:00
0a1c5034f5
[Model] Add MiniMax M3 support ( #45381 )
...
Signed-off-by: youkaichao <youkaichao@gmail.com >
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com >
Signed-off-by: functionstackx <47992694+functionstackx@users.noreply.github.com >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
2026-06-16 01:01:25 +08:00
Wei Zhao and gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
9aaf83ef50
[CI failure] Temporarily disable using persistent cache for flashinfer autotune ( #43119 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-05-19 11:44:32 -07:00
Mohammad Miadh Angkad
da03e549b3
[UX] Add a persistent cache for FlashInfer autotuning ( #42537 )
...
Signed-off-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com >
2026-05-18 20:25:37 -07:00
Wei Zhao
8c296de63b
[Perf] Re-enable flashinfer autotune by default and cleanup ( #42857 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
2026-05-18 09:12:27 -07:00
97995f6376
[MoE Refactor] Create MK for TRTLLM Kernels ( #32564 )
...
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Signed-off-by: Robert Shaw <rshaw@neuralmagic.com >
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com >
Co-authored-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com >
2026-03-03 10:39:50 -08:00
Mohammad Miadh Angkad and Luka Govedič
dd6a6e1190
[Kernel] Add KernelConfig flag to enable/disable FlashInfer autotune ( #34006 )
...
Signed-off-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com >
Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com >
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com >
2026-02-07 05:24:44 -08:00
Michael Goin
d4d2751732
Update note comment for flashinfer attention warmup ( #30711 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
2025-12-16 21:29:03 -08:00
4ccffe561f
[Core] Encoder separation for Encode-Prefill-Decode Disaggregation ( #25233 )
...
Signed-off-by: n00909098 <nguyen.kha.long@huawei.com >
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com >
Signed-off-by: herotai214 <herotai214@gmail.com >
Signed-off-by: Khuong Le <khuong.le.manh@huawei.com >
Signed-off-by: Khuong Le <lemanhkhuong2611@gmail.com >
Co-authored-by: n00909098 <nguyen.kha.long@huawei.com >
Co-authored-by: knlnguyen1802 <knlnguyen1802@gmail.com >
Co-authored-by: herotai214 <herotai214@gmail.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Khuong Le <khuong.le.manh@huawei.com >
Co-authored-by: Khuong Le <lemanhkhuong2611@gmail.com >
2025-11-11 18:58:33 -08:00
Varun Sundar Rabindranath
4022a9d279
[BugFix][Performance] Restore flashinfer autotuning for all scenarios ( #27904 )
2025-11-04 15:56:21 +08:00
Varun Sundar Rabindranath and Varun Sundar Rabindranath
e5e076cad7
[BugFix] Stopgap - Flashinfer Autotuner + GPT-OSS + DP/TP ( #27762 )
...
Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com >
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com >
2025-10-30 08:24:31 -07:00
5ff5d94e77
[Bugfix] Fix gpt-oss w4a8 DP/EP on B200 ( #26729 )
...
Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com >
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com >
Co-authored-by: Michael Goin <mgoin64@gmail.com >
2025-10-21 01:51:14 -04:00