Commit Graph
490 Commits
Author SHA1 Message Date
zack 0136df94b0 [Bugfix][Spec Decode][MoE] Avoid uninitialized EPLB state in DeepSeek V4.1 DSpark drafter (#56387)
Signed-off-by: zack <51604064+luoyuctl@users.noreply.github.com>
2026-09-15 14:35:47 +00:00
Qiu ChunshuoCodexLucas Wilkinsonmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
676650397f [Feature][PCP] Support decode-only FULL CUDA graphs (#53867)
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
Signed-off-by: QiuChunshuo <chunshuoq@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-15 16:27:46 +08:00
Nick Hill f84b0c4bce [Model Runner V2] Tolerate lack of pinned memory support (#56908)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-09-14 21:28:32 -07:00
Giancarlo Delfin 3bb0a03f35 [Model Runner V2] Acceptance estimation for adaptive verification (#52228)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
2026-09-14 15:58:48 -07:00
鐘天楽andNick Hill 0a5747d410 [Profiler] Add Proton CUDA graph attribution for MRV2 (#51084)
Signed-off-by: Luossu <zhluosuu@outlook.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-14 19:02:07 +00:00
Anjie HouandCodex 73d2a8cf86 [2/2][Model Runner V2] FULL CUDA graph capture for microbatched steps (DBO) (#51700)
Signed-off-by: specture724 <specture724@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-13 22:42:33 -07:00
Roberto L. CastroandCodex 186a1e6222 [Warmup] Gemma 4 de-JITification (#55768)
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-13 16:47:21 -04:00
Taneem Ibrahim 0cf266a469 [Pooling] Support prompt embeddings in MRV2 decoder pooling (#56639)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
2026-09-13 05:49:45 +00:00
Woosuk KwonandOpenAI Codex 2671fedfc7 [MRV2] Revert explicit Triton JIT warmup migration (#56654)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-13 04:07:56 +00:00
aed894c190 [6/N][warmup][DSv4] Migrate sampling, and DFlash JIT kernels (#56323)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-12 17:24:24 -07:00
ebe1dec2da [PCP][DCP] Enable PCP+DCP on sparse-MLA models (#56157)
Signed-off-by: patryk.saffer <patryk.saffer@blc-login-1.blc1.mistralcompute.ai>
Signed-off-by: patryk.saffer <patryk.saffer@blc-login-0.blc1.mistralcompute.ai>
Signed-off-by: PatrykSaffer <patryk.saffer@mistral.ai>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: patryk.saffer <patryk.saffer@blc-login-1.blc1.mistralcompute.ai>
Co-authored-by: patryk.saffer <patryk.saffer@blc-login-0.blc1.mistralcompute.ai>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Patryk Saffer <PatrykSaffer@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-12 23:24:44 +00:00
7ee8a6dd01 [watermarking] Dual-key gumbel-max watermarking for speculative decoding support (#56122)
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Signed-off-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
2026-09-12 18:42:01 +00:00
Itay Alroy 658c8131c7 [Elastic EP] Reuse CUDA graphs across reconfiguration (#54985) 2026-09-12 10:26:39 -04:00
Canlin GuoandCodex dca96bf97b [Bugfix][Spec Decode] Avoid fastsafetensors deadlock for PP draft models (#54416)
Signed-off-by: Canlin <canlinguosdu@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-12 14:22:30 +08:00
d43bb2f37f [3/N] HiSparse: host-resident sparse-MLA decode hot-buffering (#53781)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Matthew Bonanni <matthew@neuralmagic.com>
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
2026-09-12 00:36:47 -04:00
Giancarlo Delfin 9dcf6bf344 [BugFix] Fix DP token padding in dflash attention metadata (#56181)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
2026-09-11 14:33:53 -07:00
Chris EastwoodNick Hillmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
6fe67cbbf3 [Spec][V2] Support MTP speculative decoding under pipeline parallelism (#46994)
Signed-off-by: Chris Eastwood <chris.eastwood@pwn4g3.dev>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-11 17:55:21 +00:00
e77daef89e [Model] Support DeepSeek-V4.1-Flash (#56214)
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Signed-off-by: andyluo7 <andy.luo@amd.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-11 02:11:19 -07:00
d0dfe587d5 [Bugfix] Fall back to full decode graphs for noncompiled models (#55095)
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
2026-09-11 01:56:05 -07:00
980c16c8e4 [PCP][Spec Decode] Adds PCP support for single-module MTP and replicated DSpark. (#56107)
Signed-off-by: Summer Yang <girasoleyang@gmail.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-09-10 22:07:53 -07:00
Nick Hill e6cb56337b [Core] MRV2 support for fast-prefill (#56145)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-09-10 10:36:04 -07:00
bjf-frzandNick Hill 2e0ee66cab [Perf] Use UVA-backed contents for MRV2 apply_write (#55819)
Signed-off-by: bjf-frz <frz123db@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-10 07:27:16 -07:00
Sy03Raphael RiallandZiyang ZhangCodexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
83990f5bcc [Bugfix][MRV2] Initialize DCP metadata after batch partitioning (#55212)
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
Signed-off-by: Sy03 <1370724210@qq.com>
Co-authored-by: Raphael Rialland <raphael.rialland@mistral.ai>
Co-authored-by: Ziyang Zhang <hafuhafu@qq.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-10 06:50:09 +00:00
Raphaël RiallandOpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
ea40bb9e90 [feature] Watermarked generation and detection (Gumbel-max algorithm) (#54053)
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-09 21:00:15 -07:00
Chris EastwoodandNick Hill e8064a96d0 [Bugfix][V2] record_stream idx_mapping in the PP draft broadcast (#55745)
Signed-off-by: Chris Eastwood <chris.eastwood@pwn4g3.dev>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-09 12:55:56 -07:00
Harry MellorandClaude Opus 5 719284fe15 [Core][Model] Unify XD-RoPE into M-RoPE and derive the channel count (#56078)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 09:23:30 -07:00
Kevin H. LuuCodexWentao Yemergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
5acd95906b [Bugfix] Exclude DP token padding from draft attention metadata (#55458)
Signed-off-by: khluu <khluu000@gmail.com>
Signed-off-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-09 01:47:45 -07:00
Jing Wang c268198715 [Bugfix][Spec Decode] Preserve target parallel config (DCP) for DSpark (#55472)
Signed-off-by: Jing Wang <jingwang96@qq.com>
2026-09-08 16:59:10 -07:00
Stefano CastagnettaNick Hillmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
9c2d21046b [Bugfix][Spec Decode] Honour the draft's moe_backend on Model Runner V2 (#54788)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-08 16:50:07 -07:00
Wentao Ye 07950d4734 [Kimi Bug] Fix kimi k3 startup cuda graph issue with recoverSSM (#55774)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-09-08 10:29:53 -04:00
6fbb00b188 [EPD] Add ECMooncakeConnector for encoder cache over Mooncake TransferEngine (#41567)
Signed-off-by: Teng Ma <sima.mt@alibaba-inc.com>
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Tianyu Guo <guoty@inferact.ai>
Co-authored-by: Zhou ziheng <jiaranran2@gmail.com>
Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-07 06:21:43 +00:00
Kevin H. LuuandCodex 6865e67f0b [Bugfix] Defer adaptive verification until after kernel warmup (#55455)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-06 11:31:40 -07:00
d87a440f88 [Core][MRV2] Support eagle3 spec decode with pipeline parallel (#50514)
Signed-off-by: Yongqin Wang <yongqinwang@roblox.com>
Co-authored-by: Cursor <noreply@cursor.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-09-05 11:45:34 +08:00
685074cd61 [Bugfix][V2] Warm up kernels before capturing CUDA graphs (#55341)
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-04 22:44:18 +00:00
3f41d102c5 [1/2][Model Runner V2] DBO support, eager mode (#50945)
Signed-off-by: specture724 <specture724@gmail.com>
Signed-off-by: Anjie Hou <149605198+specture724@users.noreply.github.com>
Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
2026-09-04 14:17:39 -07:00
Zihan Zhang 5093e4844a [Bugfix][Spec Decode] Drop FlashAttention's AOT schedule for a sliding-window DFlash drafter (#54374) 2026-09-04 16:42:01 -04:00
Stefano Castagnetta 2524051385 [Bugfix][Spec Decode] Honour the draft's attention_backend on Model Runner V2 (#54826)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
2026-09-04 12:23:19 -07:00
YizhouandOpenAI Codex 8f816a3f66 [MRV2][Metrics] Support CUDAGraphStat in MRV2 (#52358)
Signed-off-by: Yizhou Liu <liu_yizhou@outlook.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-09-04 07:19:50 +00:00
560ef78bfe [Perf][Model Runner V2] Compact sampling masks on GPU instead of unpacking the full-vocab bitmask on CPU (#54901)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: vx120 <893600387@qq.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: vx120 <893600387@qq.com>
2026-09-03 19:03:36 -07:00
Jiangyun Zhu 98ed0856f3 [Model] add GLM-5.3-Flash support (#53906) 2026-09-03 09:40:35 -07:00
Giancarlo Delfin facd9a74a1 [Model Runner V2][Spec Decode] Skip DP sync for all speculator uniform decodes (#54856) 2026-09-03 10:32:33 +00:00
Isotr0py 1c26e57d3c [Bugfix] Raise for unavailable piecewise CUDA graphs (#54782)
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
2026-09-02 08:52:24 +00:00
Andrii SkliarandAndrii Skliar adebc41b7e [Mamba] Add FlashInfer ReplaySSM backend (#52506)
Signed-off-by: Andrii Skliar <askliar@nvidia.com>
Signed-off-by: Andrii Skliar <andreyws96@gmail.com>
Co-authored-by: Andrii Skliar <askliar@nvidia.com>
2026-09-01 10:26:43 -04:00
Roberto L. CastroandCodex e16b5e518d [1/N][warmup][DSv4] Migrate generic MLA metadata and indexing kernels (#50175)
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com>
Co-authored-by: Codex <codex@openai.com>
2026-08-31 23:26:05 -04:00
Nick Hill c28feab989 [Core][MRV2] Freeze gc during V2 CG capture; skip per-descriptor cleanup (#54646)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-08-31 19:37:46 -07:00
AusarandClaude 6bafc049aa [Bugfix][PP] Never drop a decoding request from the sampled-token broadcast (#54436)
Signed-off-by: ArcheyChen <45146080+ArcheyChen@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-08-31 23:49:54 +00:00
Zihan ZhangandClaude Opus 5 d61b6e1878 [Bugfix][Spec Decode] Take the DFlash draft's RoPE layout from its own config (#54373)
Signed-off-by: SubSir <tiancaizhangdaxian@sjtu.edu.cn>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-31 18:13:20 -04:00
Artem Perevedentsev 39e276eaeb [Structured Output] Let terminal grammars stop under min_tokens (II) (#54218)
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
2026-08-31 20:39:27 +04:00
+5 e126687a9a [Model] Support Qwen3.8-Flash-Next (#53896)
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
Signed-off-by: andyluo7 <andy.luo@amd.com>
Signed-off-by: lishunyang12 <lishunyang03@gmail.com>
Signed-off-by: Harry Huang <huanghaoyan.hhy@alibaba-inc.com>
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: KungYork <firelikest@163.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: andyluo7 <43718156+andyluo7@users.noreply.github.com>
Co-authored-by: Isotr0py <mozf@inferact.ai>
Co-authored-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: lishunyang12 <lishunyang03@gmail.com>
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-08-31 05:57:56 +00:00
Canlin Guoandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> b383e16396 [Bugfix] Reset cached Mamba align metadata on profiling teardown (#54044)
Signed-off-by: Canlin Guo <canlinguosdu@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-30 00:43:20 -07:00