 Woosuk Kwon
|
2f8b4ce0c0
|
[Model Runner V2] Do not initialize sampler for non-last PP ranks (#36824)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-12 03:55:28 +00:00 |
|
|
Woosuk Kwon
|
55eed6b7a5
|
[Model Runner V2] Add WhisperModelState [6/N] (#35790)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-11 14:20:38 -07:00 |
|
|
Giancarlo Delfin
|
c77181e534
|
[Model Runner V2] Add probabilistic rejection sampling for spec decoding (#35461)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-03-11 14:04:32 -07:00 |
|
|
Woosuk Kwon
|
8ccbcda5c0
|
[Model Runner V2] Remove unused warmup_for_prefill method (#36762)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-11 08:02:44 -07:00 |
|
|
Woosuk Kwon
|
f088a831dd
|
[Model Runner V2] Use unpadded num_tokens for PW CUDA graph attn metadata (#36626)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-10 09:30:56 -07:00 |
|
|
Lucas Wilkinson
|
483463f735
|
[MRV2] Extensible CG dispatch rework (#35959)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
|
2026-03-09 13:58:45 -07:00 |
|
|
Woosuk Kwon
|
10a5f4d53d
|
[Model Runner V2] Use NamedTuple for execute_model_state (#35930)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-09 11:17:34 -07:00 |
|
|
Woosuk Kwon
|
6e956d9eca
|
[Model Runner V2] Add dummy profile_cudagraph_memory API (#36520)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-09 10:20:13 -07:00 |
|
|
Kunshang Ji
|
66a2209645
|
[Hardware] Replace torch.cuda.synchronize() api with torch.accelerator.synchronize (#36085)
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
|
2026-03-05 10:36:39 +00:00 |
|
|
Nick Hill
|
a3299c3d1d
|
[Model Runner V2] Misc code simplification (#35941)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-03-04 15:26:35 -08:00 |
|
|
Nick Hill
|
417fd28fb1
|
[Model Runner V2] Fix pooling (#36019)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-03-04 10:53:17 -08:00 |
|
|
Kunshang JiandHarry Mellor
|
16d2ad1d38
|
[Hardware] Replace torch.cuda.empty_cache with torch.accelerator.empty_cache (#30681)
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
|
2026-03-04 09:49:47 +00:00 |
|
|
zhrrrandGiancarlo Delfin
|
97286a20ed
|
[Model Runner V2] support dp & ep for spec decoding (#35294)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-03-03 15:19:45 -08:00 |
|
|
Woosuk Kwon
|
467886a0c4
|
[Model Runner V2] Fix inputs_embeds=None bug for MM models (#35917)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-03 13:47:45 -08:00 |
|
|
Woosuk Kwon
|
a0a5178ab4
|
[Model Runner V2] Use ModelState.prepare_attn() for cuda graph capture [5/N] (#35774)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-02 20:06:27 -08:00 |
|
|
zhrrr
|
168ee03e1c
|
[Model Runner V2][Perf] align dummy_run tokens to uniform decode for dp cudagraph (#35376)
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
|
2026-03-02 17:10:47 -08:00 |
|
|
Woosuk Kwon
|
e3eb146f7a
|
[Model Runner V2] Add ModelStateInterface [4/N] (#35621)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-28 13:19:45 -08:00 |
|
|
Woosuk Kwon
|
1a014a0a93
|
[Model Runner V2] Move MM encoder to Model States [3/N] (#35564)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-27 18:32:38 -08:00 |
|
|
Woosuk Kwon
|
86ac7bcf84
|
[Model Runner V2] Support pooling models (#35120)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-27 18:03:01 -08:00 |
|
|
Woosuk Kwon
|
3d66502e1b
|
[Model Runner V2] Prepare attn metadata in ModelState [2/N] (#35383)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-26 11:47:02 -08:00 |
|
|
Woosuk Kwon
|
c66aa48e99
|
[Model Runner V2] Add model states [1/N] (#35350)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-26 11:20:35 -08:00 |
|
|
Nick Hill
|
b6d5a17298
|
[Model Runner V2] Fix error-handling (#35063)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-02-26 11:00:19 -08:00 |
|
|
Woosuk Kwon
|
13025e71e8
|
[Model Runner V2] Add coding style guide (#35325)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-25 20:42:40 -08:00 |
|
|
Woosuk Kwon
|
c645e9a214
|
[Model Runner V2] Remove propose_draft method (#35070)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-22 18:27:12 -08:00 |
|
|
Nick HillandWoosuk Kwon
|
944ffb5968
|
[Model Runner V2][Minor] Remove redundant do_spec_decode field (#35039)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-22 16:18:04 -08:00 |
|
|
Woosuk Kwon
|
2cbf9656ce
|
[Model Runner V2] Enable CUDA graph for Eagle3 (#35040)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-21 21:42:50 -08:00 |
|
|
Woosuk Kwon
|
b71fbd06e2
|
[Model Runner V2] Support attention group (#35036)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-21 16:42:53 -08:00 |
|
|
Woosuk Kwon
|
a4047d4ea9
|
[Model Runner V2] Support Eagle3 (no CUDA graph) (#35029)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-21 12:55:24 -08:00 |
|
|
Nick Hill
|
40b2f1c3d9
|
[Model Runner V2] Minor CPU optimizations (#34856)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-02-19 16:05:37 -08:00 |
|
|
zhrrr
|
11d3976b88
|
[Model Runner V2] support piecewise & mixed cudagraph (#32771)
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
|
2026-02-18 15:03:17 -08:00 |
|
|
Woosuk Kwon
|
95be2a7f22
|
[Model Runner V2] Minor simplification for DCP (#34786)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-18 11:04:53 -08:00 |
|
|
Nick Hill
|
c50e105a88
|
[Model Runner V2] Avoid prepare prefill kernel launch overhead (#34780)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-02-18 00:49:21 -08:00 |
|
|
Nick Hill
|
a49ea5a58f
|
[Model Runner V2] A bit more PP simplification (#34766)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-02-17 21:39:07 -08:00 |
|
|
Wentao Ye
|
ab33d2a629
|
[Feature] Decode Context Parallel support for GPU model runner v2 (#34179)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
|
2026-02-17 16:27:15 -08:00 |
|
|
Woosuk Kwon
|
be3af2d29e
|
[Model Runner V2] Further simplification for PP (#34724)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-17 15:18:18 -08:00 |
|
|
Woosuk Kwon
|
d00df624f3
|
[Model Runner V2] Minor refactoring for penalties (#34662)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-16 21:43:00 -08:00 |
|
|
Woosuk Kwon
|
04925b2202
|
[Model Runner V2] Minor cleanup for PP (#34666)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-16 19:15:31 -08:00 |
|
|
zhanqiuhu
|
9a8853f781
|
[Core] Pipeline Parallel support for Model Runner V2 (#33960)
Signed-off-by: Zhanqiu Hu <zh338@cornell.edu>
|
2026-02-16 17:48:16 -08:00 |
|
|
zhrrrandWoosuk Kwon
|
387a1898d9
|
[Model Runner V2] support bad_words sampling param (#33433)
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-16 16:36:06 -08:00 |
|
|
Nick Hill
|
e7e52781ff
|
[ModelRunner V2][BugFix] Fix max_query_len calculation (#34167)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-02-09 21:47:17 +00:00 |
|
|
zhrrr
|
16786da735
|
[Model Runner V2] support apply penalty for spec decode (#33251)
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
|
2026-02-06 10:56:48 -08:00 |
|
|
Nick Hill
|
cf0a99f84d
|
[ModelRunner V2] Support spec decode with structured outputs (#33374)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-02-02 00:19:59 +00:00 |
|
|
Nick Hill
|
e535d90deb
|
[ModelRunner V2] Misc minor simplifications and optimizations (#33467)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-02-01 22:17:14 +00:00 |
|
|
Cyrus Leung
|
79b6ec6aab
|
[Bugfix] Fix inconsistent handling of cache reset (#33481)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
|
2026-01-31 20:23:41 -08:00 |
|
|
Nick Hill
|
876a16f4fb
|
[ModelRunner V2] Fix spec decoding + logprobs (#33391)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-01-31 03:33:26 +00:00 |
|
|
Nick Hill
|
6bf3b46d78
|
[ModelRunner V2] Misc code simplification and cleanup (#33266)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-01-28 14:41:23 -08:00 |
|
|
Woosuk KwonandNick Hill
|
46ec6d71c7
|
[Model Runner V2] Use a different stream for grammar bitmask h2d copy (#33059)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Nick Hill <nhill@redhat.com>
|
2026-01-27 16:37:43 -08:00 |
|
|
Woosuk KwonandNick Hill
|
6d86fde09c
|
[Model Runner V2] Remove UvaBufferPool for cpu->gpu copy (#33055)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Nick Hill <nhill@redhat.com>
|
2026-01-26 16:47:35 -08:00 |
|
|
Woosuk Kwon
|
a9b53dd435
|
[Model Runner V2] Add LoRAState to consolidate lora logic (#33062)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-01-25 22:21:12 -08:00 |
|
|
Woosuk Kwon
|
2f0d3ba745
|
[Model Runner V2] Minor simplification for finish_requests (#33048)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-01-25 18:35:02 -08:00 |
|