 Nick Hill
|
47d4407d7c
|
[Model Runner V2] Support sharing kv cache layers (#35045)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-05-22 22:18:23 +00:00 |
|
|
haosdentandWentao Ye
|
5ea76fa89a
|
[CI] Fix test_lora_with_spec_decode on V2 model runner (#43314)
Signed-off-by: haosdent <haosdent@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
|
2026-05-22 14:24:18 +08:00 |
|
|
Wentao YeandNick Hill
|
9640970de2
|
[Model Runner V2] Fix lora Triton Error [CUDA]: device-side assert triggered (#43139)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
|
2026-05-21 01:00:30 +00:00 |
|
|
Nicolò LucchesiandNick Hill
|
fba010dd74
|
[Bugfix][MRV2] Fix KVCache tensor explicit kernel_block_size dim (#42766)
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
|
2026-05-18 20:25:41 -07:00 |
|
|
Michael GoinandClaude Opus 4.7
|
f5d3dc7115
|
[Model Runner v2] Support update_config (#42783)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
2026-05-18 10:26:07 -04:00 |
|
|
Wentao Yeandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
|
6147c70224
|
[Model Runner v2] Support reload weights (sleep mode) (#42673)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
|
2026-05-15 16:41:23 +00:00 |
|
|
PatchyTISandPatchouliTaisa
|
0162596603
|
[Model Runner V2] FP32 gumbel sampling. (#41775)
Signed-off-by: PatchouliTaisa <patchychen@tencent.com>
Co-authored-by: PatchouliTaisa <patchychen@tencent.com>
|
2026-05-15 09:20:08 -07:00 |
|
|
Nick Hill
|
9af6a5ed75
|
[Model Runner V2] Fix seq_lens_cpu_upper_bound (#42202)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-05-11 10:37:50 -07:00 |
|
|
zhrrr
|
7a08b34fbf
|
[Model Runner V2] support qwen35 / mamba hybrid model (#35520)
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
|
2026-05-07 09:31:05 -07:00 |
|
|
Woosuk Kwon
|
e6ff3e9c83
|
[MRV2] Add shutdown() method (#41297)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-05-02 21:06:30 -07:00 |
|
|
Wentao Ye
|
526927be94
|
[Model Runner v2] Fix v2 compile counter num_gpu_runner_capture_triggers and num_cudagraph_captured (#41285)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
|
2026-04-30 15:20:11 -07:00 |
|
|
Wentao YeandNick Hill
|
51fda1ba44
|
[Model Runner v2] Fix block table IMA issue (#40648)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
|
2026-04-29 08:30:33 -07:00 |
|
|
Giancarlo Delfin
|
4c7c69b4e0
|
[Model Runner V2] Skip attention metadata rebuild before draft prefill (#40410)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-04-27 15:38:05 -07:00 |
|
|
+7
|
4d51588e23
|
[Feat] DeepSeek V4 Rebased (#40860)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: qizixi <zixi@inferact.ai>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <yongye@inferact.ai>
Co-authored-by: Simon Mo <simon@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roy Wang <yasong.wang@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com>
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com>
Co-authored-by: khluu <khluu000@gmail.com>
Co-authored-by: qizixi <zixi@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
|
2026-04-26 18:31:08 -07:00 |
|
|
Benjamin Chislettandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
|
626daa2076
|
[Feat] Unified Synthetic Acceptance Rate for V1 and V2 (#40662)
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
|
2026-04-24 00:48:08 +00:00 |
|
|
Nick Hill
|
fe85a92e86
|
[Core] Avoid seq_lens_cpu GPU->CPU sync (#40654)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-04-24 00:35:55 +00:00 |
|
|
roikoren755
|
b897f00c9c
|
Gate SSU dispatch setup (#40039)
Signed-off-by: Roi Koren <roik@nvidia.com>
|
2026-04-16 13:06:01 -07:00 |
|
|
roikoren755
|
ecf5ff7ce3
|
[Mamba] Flashinfer selective_state_update (#36162)
Signed-off-by: Roi Koren <roik@nvidia.com>
|
2026-04-14 15:10:58 -04:00 |
|
|
Giancarlo Delfin
|
ccf90ba784
|
[Model Runner V2] Add full cuda graph support for eagle prefill (#37588)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-04-13 16:01:24 -07:00 |
|
|
zhrrr
|
8f121f7879
|
[Model Runner V2] support auto resolve cudagraph mode/sizes based on attn backend (#32936)
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
|
2026-04-10 08:27:15 -07:00 |
|
|
Nick Hill
|
62095e82c1
|
[BugFix][MRV2] Fix cuda event reuse race (#39115)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-04-07 00:21:09 +00:00 |
|
|
Woosuk Kwon
|
f186cfe75e
|
[MRV2] Fix hanging issue with DeepSeek V3.2 by setting skip_attn=False (#39098)
Signed-off-by: WoosukKwon <woosuk.kwon@berkeley.edu>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-04-06 12:55:13 -07:00 |
|
|
Giancarlo Delfin
|
c32e97602d
|
[Model Runner V2] Enable forcing a specific acceptance rate during rejection sampling (#38045)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-03-26 13:38:12 -07:00 |
|
|
Wentao YeandWoosuk Kwon
|
d7e93e13fb
|
[Feature] EPLB Support for GPU Model Runner v2 (#37488)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-25 08:16:39 -07:00 |
|
|
Nick Hill
|
4e824d1c83
|
[Model Runner V2][Minor] Simplify PP logic (#38031)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-03-24 13:57:17 -07:00 |
|
|
Giancarlo Delfin
|
8f4824b664
|
[Model Runner V2] Gather multimodal embeddings before draft model postprocess (#37932)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-03-23 18:14:13 -07:00 |
|
|
zhanqiuhuandWoosuk Kwon
|
63f49b8bd4
|
[Model Runner V2] Enable piecewise CUDA graphs for pipeline parallelism (#35162)
Signed-off-by: Zhanqiu Hu <zh338@cornell.edu>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-22 20:48:25 +00:00 |
|
|
Giancarlo DelfinandWoosuk Kwon
|
b3e846017d
|
[Model Runner V2] Support multi-modal embeddings for spec decode model (#36097)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-22 02:48:43 -07:00 |
|
|
Santino Ramos
|
85f671b8e1
|
[Model Runner V2] Support Streaming Inputs (#37028)
Signed-off-by: Santino Ramos <elsantinoramos@gmail.com>
|
2026-03-20 20:42:25 +00:00 |
|
|
Giancarlo Delfin
|
dcee9be95a
|
[Model Runner V2] Fix draft logits not populated during cudagraph replay (#37639)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-03-20 07:43:47 +00:00 |
|
|
Giancarlo Delfin
|
39474513f6
|
[Model Runner V2] fix draft attention metadata generation (#37364)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-03-19 21:05:15 -07:00 |
|
|
Woosuk Kwon
|
40b8363b45
|
[MRV2] Use fp32 for draft logits (#37526)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-19 08:41:21 -07:00 |
|
|
Giancarlo Delfin
|
04244fd0e1
|
[Model Runner V2] Spec decode rejection sampler greedy support (#37238)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-03-18 15:59:03 -07:00 |
|
|
Santino Ramos
|
3ed46f374b
|
[Model Runner V2] Add Support for XD-RoPE (#36817)
Signed-off-by: Santino Ramos <elsantinoramos@gmail.com>
|
2026-03-14 09:27:55 -07:00 |
|
|
Woosuk Kwon
|
2f8b4ce0c0
|
[Model Runner V2] Do not initialize sampler for non-last PP ranks (#36824)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-12 03:55:28 +00:00 |
|
|
Woosuk Kwon
|
55eed6b7a5
|
[Model Runner V2] Add WhisperModelState [6/N] (#35790)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-11 14:20:38 -07:00 |
|
|
Giancarlo Delfin
|
c77181e534
|
[Model Runner V2] Add probabilistic rejection sampling for spec decoding (#35461)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-03-11 14:04:32 -07:00 |
|
|
Woosuk Kwon
|
8ccbcda5c0
|
[Model Runner V2] Remove unused warmup_for_prefill method (#36762)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-11 08:02:44 -07:00 |
|
|
Woosuk Kwon
|
f088a831dd
|
[Model Runner V2] Use unpadded num_tokens for PW CUDA graph attn metadata (#36626)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-10 09:30:56 -07:00 |
|
|
Lucas Wilkinson
|
483463f735
|
[MRV2] Extensible CG dispatch rework (#35959)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
|
2026-03-09 13:58:45 -07:00 |
|
|
Woosuk Kwon
|
10a5f4d53d
|
[Model Runner V2] Use NamedTuple for execute_model_state (#35930)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-09 11:17:34 -07:00 |
|
|
Woosuk Kwon
|
6e956d9eca
|
[Model Runner V2] Add dummy profile_cudagraph_memory API (#36520)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-09 10:20:13 -07:00 |
|
|
Kunshang Ji
|
66a2209645
|
[Hardware] Replace torch.cuda.synchronize() api with torch.accelerator.synchronize (#36085)
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
|
2026-03-05 10:36:39 +00:00 |
|
|
Nick Hill
|
a3299c3d1d
|
[Model Runner V2] Misc code simplification (#35941)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-03-04 15:26:35 -08:00 |
|
|
Nick Hill
|
417fd28fb1
|
[Model Runner V2] Fix pooling (#36019)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-03-04 10:53:17 -08:00 |
|
|
Kunshang JiandHarry Mellor
|
16d2ad1d38
|
[Hardware] Replace torch.cuda.empty_cache with torch.accelerator.empty_cache (#30681)
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
|
2026-03-04 09:49:47 +00:00 |
|
|
zhrrrandGiancarlo Delfin
|
97286a20ed
|
[Model Runner V2] support dp & ep for spec decoding (#35294)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-03-03 15:19:45 -08:00 |
|
|
Woosuk Kwon
|
467886a0c4
|
[Model Runner V2] Fix inputs_embeds=None bug for MM models (#35917)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-03 13:47:45 -08:00 |
|
|
Woosuk Kwon
|
a0a5178ab4
|
[Model Runner V2] Use ModelState.prepare_attn() for cuda graph capture [5/N] (#35774)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-03-02 20:06:27 -08:00 |
|
|
zhrrr
|
168ee03e1c
|
[Model Runner V2][Perf] align dummy_run tokens to uniform decode for dp cudagraph (#35376)
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
|
2026-03-02 17:10:47 -08:00 |
|