 Benjamin Chislett
|
8b346309a5
|
[Refactor] Consolidate SupportsEagle (#36063)
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
|
2026-03-13 23:22:40 +00:00 |
|
|
Nick Hill
|
cd32d6f586
|
[Model Runner V2] Some code simplification (#36929)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-03-13 00:59:23 +00:00 |
|
|
Woosuk Kwon
|
2f8b4ce0c0
|
[Model Runner V2] Do not initialize sampler for non-last PP ranks (#36824)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-12 03:55:28 +00:00 |
|
|
Wentao Ye
|
c34ba6b961
|
[Perf] Optimize compute maxsim using batched version, 3.2% E2E throughput improvement (#36710)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
|
2026-03-12 08:37:01 +08:00 |
|
|
Woosuk Kwon
|
55eed6b7a5
|
[Model Runner V2] Add WhisperModelState [6/N] (#35790)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-11 14:20:38 -07:00 |
|
|
Giancarlo Delfin
|
c77181e534
|
[Model Runner V2] Add probabilistic rejection sampling for spec decoding (#35461)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-03-11 14:04:32 -07:00 |
|
|
Woosuk Kwon
|
8ccbcda5c0
|
[Model Runner V2] Remove unused warmup_for_prefill method (#36762)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-11 08:02:44 -07:00 |
|
|
Woosuk Kwon
|
f088a831dd
|
[Model Runner V2] Use unpadded num_tokens for PW CUDA graph attn metadata (#36626)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-10 09:30:56 -07:00 |
|
|
Nick Hill
|
ddbb0d230a
|
[Model Runner V2] Fix mm input embeddings lookup (#36588)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-03-10 00:24:58 -07:00 |
|
|
Nick Hill
|
9efc3bdcd6
|
[Model Runner V2] Fix _compute_slot_mappings_kernel for chunked prefill (#36580)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-03-10 00:23:42 -07:00 |
|
|
Wentao Ye
|
7279374f91
|
[Perf] Compute maxsim in worker side, reducing redundant copies, 2.7% E2E throughput improvement (#36159)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
|
2026-03-09 20:55:58 -07:00 |
|
|
Woosuk Kwon
|
2a194ddd72
|
[Model Runner V2] Add model_state inputs to CUDA graph capture (#36544)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-09 15:14:51 -07:00 |
|
|
Lucas Wilkinson
|
483463f735
|
[MRV2] Extensible CG dispatch rework (#35959)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
|
2026-03-09 13:58:45 -07:00 |
|
|
Woosuk Kwon
|
10a5f4d53d
|
[Model Runner V2] Use NamedTuple for execute_model_state (#35930)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-09 11:17:34 -07:00 |
|
|
Woosuk Kwon
|
6e956d9eca
|
[Model Runner V2] Add dummy profile_cudagraph_memory API (#36520)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-09 10:20:13 -07:00 |
|
|
Nick Hill
|
b354686524
|
[Model Runner V2] Fix warmup for pipeline parallel (#36280)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-03-06 16:58:51 -08:00 |
|
|
Cyrus Leung
|
6dd302653f
|
[Misc] Rename group_mm_kwargs_by_modality -> group_and_batch_mm_kwargs (#36158)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
|
2026-03-06 12:32:48 +08:00 |
|
|
Nick Hill
|
a73af584fe
|
[Model Runner V2] Fix warmup for very small kvcache and/or blocksizes (#36176)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-03-05 14:48:10 -08:00 |
|
|
Kunshang Ji
|
66a2209645
|
[Hardware] Replace torch.cuda.synchronize() api with torch.accelerator.synchronize (#36085)
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
|
2026-03-05 10:36:39 +00:00 |
|
|
Nick Hill
|
a3299c3d1d
|
[Model Runner V2] Misc code simplification (#35941)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-03-04 15:26:35 -08:00 |
|
|
Nick Hill
|
417fd28fb1
|
[Model Runner V2] Fix pooling (#36019)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-03-04 10:53:17 -08:00 |
|
|
Kunshang JiandHarry Mellor
|
16d2ad1d38
|
[Hardware] Replace torch.cuda.empty_cache with torch.accelerator.empty_cache (#30681)
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
|
2026-03-04 09:49:47 +00:00 |
|
|
zhrrrandGiancarlo Delfin
|
97286a20ed
|
[Model Runner V2] support dp & ep for spec decoding (#35294)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-03-03 15:19:45 -08:00 |
|
|
Woosuk Kwon
|
467886a0c4
|
[Model Runner V2] Fix inputs_embeds=None bug for MM models (#35917)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-03 13:47:45 -08:00 |
|
|
Woosuk Kwon
|
a0a5178ab4
|
[Model Runner V2] Use ModelState.prepare_attn() for cuda graph capture [5/N] (#35774)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-02 20:06:27 -08:00 |
|
|
Andy Lo
|
0a7165fd71
|
[ModelRunnerV2] Rename sampler functions and variables for clarity (#35459)
Signed-off-by: Andy Lo <andy@mistral.ai>
|
2026-03-02 19:48:56 -08:00 |
|
|
zhrrr
|
168ee03e1c
|
[Model Runner V2][Perf] align dummy_run tokens to uniform decode for dp cudagraph (#35376)
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
|
2026-03-02 17:10:47 -08:00 |
|
|
Woosuk Kwon
|
72f4d16262
|
[Model Runner V2] Use block table apis for capture inputs (#35671)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-01 10:31:13 -08:00 |
|
|
Woosuk Kwon
|
da543d1abe
|
[Model Runner V2] Minor refactoring for EncoderRunner (#35628)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-01 00:15:39 -08:00 |
|
|
Woosuk Kwon
|
e3eb146f7a
|
[Model Runner V2] Add ModelStateInterface [4/N] (#35621)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-28 13:19:45 -08:00 |
|
|
Woosuk Kwon
|
1a014a0a93
|
[Model Runner V2] Move MM encoder to Model States [3/N] (#35564)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-27 18:32:38 -08:00 |
|
|
Woosuk Kwon
|
86ac7bcf84
|
[Model Runner V2] Support pooling models (#35120)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-27 18:03:01 -08:00 |
|
|
Nick Hill
|
b1d9f5372d
|
[Model Runner V2] Warmup kernels (#35172)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-02-27 10:43:30 -08:00 |
|
|
Woosuk Kwon
|
3d66502e1b
|
[Model Runner V2] Prepare attn metadata in ModelState [2/N] (#35383)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-26 11:47:02 -08:00 |
|
|
Woosuk Kwon
|
c66aa48e99
|
[Model Runner V2] Add model states [1/N] (#35350)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-26 11:20:35 -08:00 |
|
|
Nick Hill
|
b6d5a17298
|
[Model Runner V2] Fix error-handling (#35063)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-02-26 11:00:19 -08:00 |
|
|
Woosuk Kwon
|
13025e71e8
|
[Model Runner V2] Add coding style guide (#35325)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-25 20:42:40 -08:00 |
|
|
Ming YangandMichael Goin
|
6831650c40
|
[offloader] v2: Hide weight onloading latency via prefetching (#29941)
Signed-off-by: Ming Yang <minos.future@gmail.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
|
2026-02-25 17:20:59 -08:00 |
|
|
Woosuk Kwon
|
c645e9a214
|
[Model Runner V2] Remove propose_draft method (#35070)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-22 18:27:12 -08:00 |
|
|
Nick HillandWoosuk Kwon
|
944ffb5968
|
[Model Runner V2][Minor] Remove redundant do_spec_decode field (#35039)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-22 16:18:04 -08:00 |
|
|
qizixiandLu Fang
|
b9c2a565cc
|
[Spec Decode] Defer clearing KV connector metadata for EAGLE3 speculative decode + prefill / decode disagg setup (#34529)
Signed-off-by: qizixi <qizixi@meta.com>
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
|
2026-02-22 08:08:32 -08:00 |
|
|
Woosuk Kwon
|
2cbf9656ce
|
[Model Runner V2] Enable CUDA graph for Eagle3 (#35040)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-21 21:42:50 -08:00 |
|
|
Woosuk Kwon
|
b71fbd06e2
|
[Model Runner V2] Support attention group (#35036)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-21 16:42:53 -08:00 |
|
|
Woosuk Kwon
|
a4047d4ea9
|
[Model Runner V2] Support Eagle3 (no CUDA graph) (#35029)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-21 12:55:24 -08:00 |
|
|
Nick Hill
|
40b2f1c3d9
|
[Model Runner V2] Minor CPU optimizations (#34856)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-02-19 16:05:37 -08:00 |
|
|
Woosuk Kwon
|
5fcb0cdd68
|
[Model Runner V2] Use FP32 for Gumbel Noise (#34854)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-18 17:07:37 -08:00 |
|
|
Woosuk Kwon
|
c878b43b64
|
[Model Runner V2] Remove unnecessary copies in PW CUDA graph capture (#34849)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-18 15:52:50 -08:00 |
|
|
zhrrr
|
11d3976b88
|
[Model Runner V2] support piecewise & mixed cudagraph (#32771)
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
|
2026-02-18 15:03:17 -08:00 |
|
|
Woosuk Kwon
|
95be2a7f22
|
[Model Runner V2] Minor simplification for DCP (#34786)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-02-18 11:04:53 -08:00 |
|
|
Nick Hill
|
c50e105a88
|
[Model Runner V2] Avoid prepare prefill kernel launch overhead (#34780)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-02-18 00:49:21 -08:00 |
|