        
|
251f7e478e
|
[Model] Add PW CUDA graph support for Inkling [2/N] (#48822)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <mozf@inferact.ai>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-07-16 10:28:56 -07:00 |
|
 Michael GoinandClaude Opus 4.8
|
8bfd683901
|
[Spec Decode] Add kv_cache_dtype to speculative_config to control separately from target (#48787)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
2026-07-15 23:49:23 -06:00 |
|
 Michael GoinandClaude Fable 5
|
ecf4aa5ce2
|
[Bugfix] Fix FlashInfer non-causal draft attention (DFlash/DSpark) on Blackwell (#48167)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
2026-07-15 12:44:01 -07:00 |
|
Nick Hill
|
26587f9519
|
[BugFix][ModelRunner V2] Fix stale attn metadata in speculator prefill cudagraph capture (#48261)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-07-13 09:39:15 -07:00 |
|
 Michael GoinandClaude Opus 4.8
|
0d12618e98
|
[Spec Decode] Support hybrid (SWA + full attention) DFlash drafters (#47914)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
2026-07-08 11:12:45 -04:00 |
|
Benjamin Chislett
|
7bd154375d
|
[Bugfix] Fix mamba+dflash for MRV2 (#47698)
|
2026-07-07 15:59:13 -04:00 |
|
Summer Yang
|
d3e69fd671
|
[Perf] Use blocking CUDA events to avoid busy polling cuda driver lock (#47081)
Signed-off-by: Jingyi Yang <girasoleyang@gmail.com>
|
2026-07-07 09:36:10 +00:00 |
|
 Kunshang JiandHarry Mellor
|
07f9baf756
|
Revert "[Platform] Replace torch.cuda.Event with torch.Event (#47140)" (#47668)
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
|
2026-07-06 14:18:33 +01:00 |
|
 Woosuk KwonandClaude Fable 5
|
a47f38f825
|
[Bugfix][Model Runner V2][Spec Decode] Fix int32 offset overflow in block verification kernels (#47383)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
2026-07-02 07:32:38 -07:00 |
|
Michael Goin
|
2b753ad200
|
[Spec Decode] DSpark speculators checkpoint support (#47093)
Signed-off-by: mgoin <mgoin64@gmail.com>
|
2026-07-01 17:32:27 -07:00 |
|
 
|
f5a8d73377
|
[Spec Decode] DSpark (#46995)
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: mgoin <mgoin64@gmail.com>
|
2026-07-01 08:30:24 -07:00 |
|
Giancarlo Delfin
|
3cecee40f3
|
[Model Runner V2][Spec Decode] Fix stale values in idx_mapping from CG num reqs padding (#47066)
|
2026-06-30 11:25:32 -07:00 |
|
 Kunshang JiandHarry Mellor
|
e840f0d3f5
|
[Platform] Replace torch.cuda.Event with torch.Event (#47140)
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
|
2026-06-30 08:39:59 -07:00 |
|
Giancarlo Delfin
|
db808b3961
|
[Model Runner V2][Spec Decode] Implement block verification for rejection sampling (#46781)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-06-30 08:07:24 -07:00 |
|
Kunshang Ji
|
fb42e5219e
|
[Platform] Replace torch.cuda.mem_get_info with torch.accelerator.get_memory_info (#44825)
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
|
2026-06-30 14:39:52 +08:00 |
|
 Ilya MarkovandMarkov Ilya
|
a2abce646f
|
[EPLB] Mask padding in EPLB load recording (#38128)
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Markov Ilya <markovilya19@gmail.com>
|
2026-06-28 19:43:58 -07:00 |
|
Woosuk Kwon
|
0472436541
|
[Spec Decode] Avoid redundant hidden-states gather in draft prefill (#46968)
|
2026-06-28 17:04:01 -07:00 |
|
Chauncey
|
11a12305c0
|
[Model Runner V2][Spec Decode] Handle tuple hidden states from MTP draft models (#46786)
|
2026-06-27 18:38:07 -07:00 |
|
Giancarlo Delfin
|
b6caeb5a09
|
[Model Runner V2][Spec Decode] Use fp32 uniform threshold for acceptance (#46878)
|
2026-06-27 14:09:25 -07:00 |
|
![mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>](/assets/img/avatar_default.png) yihengandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
652d962bc9
|
[Model Runner V2][Spec Decode] Reduce TP communication for draft token generation (#46448)
Signed-off-by: EanWang211123 <wangyiheng@sangfor.com.cn>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-06-25 19:30:07 -07:00 |
|
Giancarlo Delfin
|
5314665bad
|
[Model Runner V2][DFlash] Enable dflash attention backend selection (#46770)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-06-25 19:29:25 -07:00 |
|
Giancarlo Delfin
|
c53994e134
|
[Model Runner V2][Spec Decode] Use log1p to compute residual during rejection sampling (#46665)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-06-25 23:46:10 +00:00 |
|
 
|
3c43237233
|
[Bugfix][Model Runner V2][Spec Decode] Fix int32 offset overflow in sampler kernels (#46560)
Signed-off-by: xiaojun.wei <jessiewei747@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
|
2026-06-24 11:00:57 -07:00 |
|
Nick Hill
|
7ee4d22009
|
[Spec Decode] Reject placeholder (-1) draft tokens in rejection sampler (#46533)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-06-24 03:32:32 +00:00 |
|
Giancarlo Delfin
|
ef361de916
|
[Model Runer V2][DFlash] Fix lm head sharing for dflash (#46435)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-06-23 19:09:06 +00:00 |
|
Jee Jee Li
|
ebbb2d55ac
|
[CI/Build][Bugfix] Fix SD LoRA (#45941)
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
|
2026-06-18 00:34:15 +00:00 |
|
        
|
eb28452b10
|
[Model] Add DiffusionGemma Support (#45163)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Martin Kukla <martin.kukla@cantab.net>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Dipika Sikka <dsikka@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
Co-authored-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
Co-authored-by: Alec Kohlhoff <134344302+aleckohlhoff@users.noreply.github.com>
Co-authored-by: Porras Huang <20535584+porrashuang@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: scoootscooob <167050519+scoootscooob@users.noreply.github.com>
|
2026-06-11 22:17:35 -07:00 |
|
 Benjamin ChislettandGiancarlo Delfin
|
0bae1d3848
|
[MRV2][Spec Decode] DFlash (#44586)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-06-10 08:47:46 -07:00 |
|
![mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>](/assets/img/avatar_default.png) Jared Wenandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
32f34d3935
|
[feature] add index share feature for DSA MTP (#44420)
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-06-06 22:04:14 -07:00 |
|
Giancarlo Delfin
|
ceb0111a90
|
[Model Runner V2][Spec Decode] Add Gemma4 MTP support (#43241)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-06-04 00:51:06 +00:00 |
|
 Giancarlo DelfinandWoosuk Kwon
|
91945b6e4a
|
[Bug Fix][Model Runner V2][Spec Decode] Warmup & capture with different attention states for speculator prefill (#44253)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-06-03 13:32:40 -07:00 |
|
zhrrr
|
1edfd09ffd
|
[Model Runner V2] Use actual batch max_seq_len for attn metadata (#43991)
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
|
2026-06-02 06:07:56 +00:00 |
|
Lucas Wilkinson
|
4721bb3aa4
|
[MRV2] Remove Eagle's dedicated CUDA graph pool (#44078)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
|
2026-05-31 22:00:33 -07:00 |
|
Woosuk Kwon
|
27fa5aa3b9
|
[MRV2] Support breakable CUDA graph (#44050)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-05-30 09:40:52 -07:00 |
|
 Mengqing CaoandNick Hill
|
1223732dda
|
[ModelRunnerV2][Hybrid model] Support kernel block size in hybrid model (#38831)
Signed-off-by: MengqingCao <cmq0113@163.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Mengqing Cao <cmq0113@163.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
|
2026-05-28 00:55:55 +00:00 |
|
 Nicolò LucchesiandNick Hill
|
fba010dd74
|
[Bugfix][MRV2] Fix KVCache tensor explicit kernel_block_size dim (#42766)
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
|
2026-05-18 20:25:41 -07:00 |
|
 PatchyTISandPatchouliTaisa
|
0162596603
|
[Model Runner V2] FP32 gumbel sampling. (#41775)
Signed-off-by: PatchouliTaisa <patchychen@tencent.com>
Co-authored-by: PatchouliTaisa <patchychen@tencent.com>
|
2026-05-15 09:20:08 -07:00 |
|
Siddharth Bedekar
|
f51f6844f9
|
[Bugfix][Spec Decode] Wire draft_probs into probabilistic draft_model rejection (#40269)
|
2026-05-13 21:04:03 -04:00 |
|
Nick Hill
|
a505cf807e
|
[ModelRunner V2] Share identical MTP weights (#42538)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-05-13 18:57:04 +00:00 |
|
Giancarlo Delfin
|
fe5b4e0fe7
|
[Model Runner V2] Apply synthetic mode to probabilistic rejection sampler (#41035)
|
2026-05-12 13:37:03 -07:00 |
|
zhrrr
|
7a08b34fbf
|
[Model Runner V2] support qwen35 / mamba hybrid model (#35520)
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
|
2026-05-07 09:31:05 -07:00 |
|
Giancarlo Delfin
|
e1e4646b06
|
[Model Runner V2] Rebuild attn metadata between draft decode steps (#41162)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-05-05 00:44:55 +00:00 |
|
Giancarlo Delfin
|
4c7c69b4e0
|
[Model Runner V2] Skip attention metadata rebuild before draft prefill (#40410)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-04-27 15:38:05 -07:00 |
|
Giancarlo Delfin
|
f5f9878514
|
[Model Runner V2] Fix rejection sampling acceptance rate gap vs MRV1 (#40651)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-04-26 19:12:08 -07:00 |
|
+7        
|
4d51588e23
|
[Feat] DeepSeek V4 Rebased (#40860)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: qizixi <zixi@inferact.ai>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <yongye@inferact.ai>
Co-authored-by: Simon Mo <simon@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roy Wang <yasong.wang@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com>
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com>
Co-authored-by: khluu <khluu000@gmail.com>
Co-authored-by: qizixi <zixi@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
|
2026-04-26 18:31:08 -07:00 |
|
![gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>](/assets/img/avatar_default.png) Benjamin Chislettandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
|
626daa2076
|
[Feat] Unified Synthetic Acceptance Rate for V1 and V2 (#40662)
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
|
2026-04-24 00:48:08 +00:00 |
|
Giancarlo Delfin
|
343f65234b
|
[Model Runner V2][BugFix] fix num_sampled dtype for probabilistic rej… (#39951)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-04-15 18:09:11 -07:00 |
|
Giancarlo Delfin
|
3bfe55a037
|
[Model Runner V2] Disable piecewise cudagraph mode fallback for eagle draft decodes (#39773)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-04-14 17:47:57 -07:00 |
|
Giancarlo Delfin
|
ccf90ba784
|
[Model Runner V2] Add full cuda graph support for eagle prefill (#37588)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-04-13 16:01:24 -07:00 |
|
zhrrr
|
8f121f7879
|
[Model Runner V2] support auto resolve cudagraph mode/sizes based on attn backend (#32936)
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
|
2026-04-10 08:27:15 -07:00 |
|