 Cheese CakeandClaude Opus 5.5
|
b6d8e8afd9
|
[Bugfix][MRV2][Spec Decode] Accept num_speculative_tokens in NgramGPUSpeculator.propose (#60300)
Signed-off-by: cheesecake <farzanaman99@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
2026-10-06 15:02:04 -07:00 |
|
   
|
819852df6b
|
[ModelRunner V2] Speculative Decoding NGram GPU Implementations (#40704)
Signed-off-by: PatchouliTaisa <patchychen@tencent.com>
Signed-off-by: PatchouliTaisa <pyramkar@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Xuanan Chen <xuananchenc@nvidia.com>
Co-authored-by: PatchouliTaisa <patchychen@tencent.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Xuanan Chen <xuananchenc@nvidia.com>
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com>
|
2026-10-06 13:32:59 -07:00 |
|
Giancarlo Delfin
|
fcf53d240c
|
[Spec Decode] Remove eager metadata rebuild during MTP fused multi-step decode (#58463)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-10-06 12:17:30 -04:00 |
|
Wentao Ye
|
74c5cbcd7b
|
[Bugfix][MRV2][Spec Decode] Honor dynamic K in autoregressive speculators, 1.2~1.3x kernel perf improvement (#57053)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
|
2026-10-05 14:54:11 -04:00 |
|
Wentao Ye
|
8bdd8d847d
|
[GLM 5.3 Perf] Enable fused multi-step decode, 13.3% E2E throughput improvement for concurrency 1 (#57443)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
|
2026-10-02 20:05:27 -04:00 |
|
 
|
4056c8ac1f
|
[Bugfix][HiSparse] Fix MTP acceptance collapse under FULL graphs with a saturated GPU pool (#59309)
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
|
2026-10-01 23:04:21 -07:00 |
|
 
|
c4fc3c7d32
|
[Bugfix][Spec Decode] Per-module LM heads for multi-layer MTP on Model Runner V2 (#58921)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: chanh <channguyen@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
2026-10-01 01:09:14 -07:00 |
|
  
|
70be6aae19
|
[Feature][Spec Decode] Support sampling mask replay for MRV2 MTP (#59359)
Signed-off-by: Cuiping Cheng <chengcuipingswu@163.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Cuiping Cheng <chengcuipingswu@163.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
2026-10-01 13:51:39 +08:00 |
|
   
|
34c458d27f
|
[Bugfix] Don't let a structured-output request sample from an unmasked row (#54442)
Signed-off-by: ArcheyChen <45146080+ArcheyChen@users.noreply.github.com>
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
Co-authored-by: GPT-5.6 <noreply@openai.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
|
2026-10-01 02:11:44 +00:00 |
|
 
|
0f8b398158
|
[watermarking] add context deduplication support to speculative decoding (#56807)
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
|
2026-09-30 12:22:10 -04:00 |
|
 Giancarlo DelfinandWoosuk Kwon
|
3d5f4d4cd5
|
[Perf][Spec Decode] Avoid triton recompiles in the acceptance estimator (#57107)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-09-28 13:46:25 -07:00 |
|
 
|
fedbc3b564
|
[Bugfix][MRV2][Spec Decode] Reject draft slots that were never proposed (#58784)
Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
|
2026-09-28 13:07:15 -07:00 |
|
 ![mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>](/assets/img/avatar_default.png)  
|
e7c903609f
|
[Perf][MRV2] Allow FULL decode graphs for one-token prompt tails (#58400)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
|
2026-09-28 12:48:52 -07:00 |
|
![mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>](/assets/img/avatar_default.png) Xiaoshuang Wangandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
ef42093a36
|
[Model Runner V2][Spec Decode] Support spec decode with draft model (#43091)
Signed-off-by: Icey <1790571317@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-09-28 19:38:05 +00:00 |
|
 
|
927c87b347
|
[PCP][DCP] Support DCP target model with non-DCP Dspark (#56723)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
|
2026-09-26 18:12:35 -04:00 |
|
  
|
a4eb3f25d6
|
[Spec Decode] Enable Gemma4 DSpark adaptive verification with FlashInfer (#57263)
Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
|
2026-09-26 17:01:17 +08:00 |
|
  
|
73a78e6f1f
|
[SpecDecode] Add LiLiCorr drafter (#57934)
Signed-off-by: Andrii Skliar <askliar@nvidia.com>
Signed-off-by: Andrii Skliar <andreyws96@gmail.com>
Co-authored-by: Andrii Skliar <askliar@nvidia.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
|
2026-09-25 10:08:28 -04:00 |
|
 Juntian LiuandClaude Opus 5.5
|
04730e8270
|
[DFlash] Capture the context K/V precompute in the draft CUDA graph (#57632)
Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
2026-09-24 21:38:04 +00:00 |
|
Nick Hill
|
09fe178dba
|
Revert "[DSpark] Support pipeline-parallel targets in aggregated serving (#56956)" (#58484)
|
2026-09-24 08:48:36 -07:00 |
|
 Andreas KaratzasandOpenAI Codex
|
23110a0c9e
|
[Bugfix] Capture prefill kernels for mixed FULL graphs (#58275)
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-09-24 02:05:07 -07:00 |
|
Jhao-Ting Chen
|
4fb767ff0a
|
[Kernel] Resubmit PR 48666 - Gemma4 FP8 KV FA4 head dim 512 backend selection (#53175)
Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com>
|
2026-09-23 20:29:06 -07:00 |
|
![mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>](/assets/img/avatar_default.png)     
|
e6dc16cebd
|
[PCP] Support prefill context parallelism with data parallelism (#57075)
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: QiuChunshuo <qiuchunshuo@huawei.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-09-23 22:32:36 -04:00 |
|
 
|
157bcb7c48
|
[DSpark] Support pipeline-parallel targets in aggregated serving (#56956)
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
|
2026-09-23 04:52:28 -07:00 |
|
Nick Hill
|
afec265bca
|
[MRV2] Miscellaneous code cleanup (#57980)
|
2026-09-23 01:51:52 -07:00 |
|
 Lucas WilkinsonandClaude Fable 5.1
|
5c121e962a
|
[SpecDecode] Restore residual-logits comments in _resample_kernel (#58166)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
|
2026-09-22 08:07:31 -07:00 |
|
Yicong Wang
|
c4d424c2e2
|
[Bugfix][Spec Decode] Cap DFlash/DSpark profiling query batch (#56448)
Signed-off-by: wangyicong <wangyicong@bytedance.com>
|
2026-09-22 12:36:32 +08:00 |
|
 siyuandIsotr0py
|
c723a831a8
|
[Fast Start] Cache the MTP draft model in a separate daemon group (#57312)
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
|
2026-09-22 03:13:38 +00:00 |
|
Cyrus Leung
|
c8824a1e42
|
[MM] Move supports_multimodal_inputs and cache out of registry (#57913)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
|
2026-09-21 07:43:07 -07:00 |
|
 Yifan QiaoandClaude Fable 5.1
|
d2983f2f16
|
[Bugfix][Spec Decode] Stop dummy draft decode steps from writing KV through stale block-table rows (#56734)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
|
2026-09-21 01:06:25 +00:00 |
|
 
|
5302d1fe40
|
[Model Runner V2] Support custom logits processors (#56497)
Signed-off-by: David Cheung <d7cheung@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
|
2026-09-19 17:46:49 +00:00 |
|
 Harry MellorandClaude Opus 5
|
651a88c09a
|
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
2026-09-16 10:57:24 -07:00 |
|
 Harry MellorandClaude Opus 5
|
8c1557a79c
|
Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
2026-09-16 16:44:29 +00:00 |
|
Nick Hill
|
f2aad6aa70
|
[MRV2] Buffer util simplifications (#56888)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-09-15 20:36:51 +00:00 |
|
zack
|
0136df94b0
|
[Bugfix][Spec Decode][MoE] Avoid uninitialized EPLB state in DeepSeek V4.1 DSpark drafter (#56387)
Signed-off-by: zack <51604064+luoyuctl@users.noreply.github.com>
|
2026-09-15 14:35:47 +00:00 |
|
Giancarlo Delfin
|
3bb0a03f35
|
[Model Runner V2] Acceptance estimation for adaptive verification (#52228)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-09-14 15:58:48 -07:00 |
|
 Woosuk KwonandOpenAI Codex
|
2671fedfc7
|
[MRV2] Revert explicit Triton JIT warmup migration (#56654)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-09-13 04:07:56 +00:00 |
|
 
|
aed894c190
|
[6/N][warmup][DSv4] Migrate sampling, and DFlash JIT kernels (#56323)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-09-12 17:24:24 -07:00 |
|
  
|
7ee8a6dd01
|
[watermarking] Dual-key gumbel-max watermarking for speculative decoding support (#56122)
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Signed-off-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
|
2026-09-12 18:42:01 +00:00 |
|
 Canlin GuoandCodex
|
dca96bf97b
|
[Bugfix][Spec Decode] Avoid fastsafetensors deadlock for PP draft models (#54416)
Signed-off-by: Canlin <canlinguosdu@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
|
2026-09-12 14:22:30 +08:00 |
|
Giancarlo Delfin
|
9dcf6bf344
|
[BugFix] Fix DP token padding in dflash attention metadata (#56181)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-09-11 14:33:53 -07:00 |
|
    
|
e77daef89e
|
[Model] Support DeepSeek-V4.1-Flash (#56214)
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Signed-off-by: andyluo7 <andy.luo@amd.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
2026-09-11 02:11:19 -07:00 |
|
 
|
d0dfe587d5
|
[Bugfix] Fall back to full decode graphs for noncompiled models (#55095)
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
|
2026-09-11 01:56:05 -07:00 |
|
  
|
980c16c8e4
|
[PCP][Spec Decode] Adds PCP support for single-module MTP and replicated DSpark. (#56107)
Signed-off-by: Summer Yang <girasoleyang@gmail.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
|
2026-09-10 22:07:53 -07:00 |
|
![mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>](/assets/img/avatar_default.png)   
|
83990f5bcc
|
[Bugfix][MRV2] Initialize DCP metadata after batch partitioning (#55212)
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
Signed-off-by: Sy03 <1370724210@qq.com>
Co-authored-by: Raphael Rialland <raphael.rialland@mistral.ai>
Co-authored-by: Ziyang Zhang <hafuhafu@qq.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-09-10 06:50:09 +00:00 |
|
![mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>](/assets/img/avatar_default.png)  
|
5acd95906b
|
[Bugfix] Exclude DP token padding from draft attention metadata (#55458)
Signed-off-by: khluu <khluu000@gmail.com>
Signed-off-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-09-09 01:47:45 -07:00 |
|
Jing Wang
|
c268198715
|
[Bugfix][Spec Decode] Preserve target parallel config (DCP) for DSpark (#55472)
Signed-off-by: Jing Wang <jingwang96@qq.com>
|
2026-09-08 16:59:10 -07:00 |
|
![mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>](/assets/img/avatar_default.png) 
|
9c2d21046b
|
[Bugfix][Spec Decode] Honour the draft's moe_backend on Model Runner V2 (#54788)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-09-08 16:50:07 -07:00 |
|
 
|
d87a440f88
|
[Core][MRV2] Support eagle3 spec decode with pipeline parallel (#50514)
Signed-off-by: Yongqin Wang <yongqinwang@roblox.com>
Co-authored-by: Cursor <noreply@cursor.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
|
2026-09-05 11:45:34 +08:00 |
|
Zihan Zhang
|
5093e4844a
|
[Bugfix][Spec Decode] Drop FlashAttention's AOT schedule for a sliding-window DFlash drafter (#54374)
|
2026-09-04 16:42:01 -04:00 |
|
Stefano Castagnetta
|
2524051385
|
[Bugfix][Spec Decode] Honour the draft's attention_backend on Model Runner V2 (#54826)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
|
2026-09-04 12:23:19 -07:00 |
|