28 Commits
Author SHA1 Message Date
7c9042c05a [Bugfix][MRV2] Fix speculative warmup deadlock with pipeline parallelism (#62129)
Signed-off-by: chenjr97 <chenjr97@mail2.sysu.edu.cn>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-17 23:51:32 -07:00
qizixiClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
c1407a7e86 [Spec Decode][MRV2] Warm up top-k/top-p and greedy rejection kernels for speculative decoding (#59260)
Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-14 14:59:21 -04:00
Woosuk KwonandClaude Opus 5.5 e1a5086256 [Attention] Remove cascade attention (#61126)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 10:41:07 -07:00
Wentao Ye 74c5cbcd7b [Bugfix][MRV2][Spec Decode] Honor dynamic K in autoregressive speculators, 1.2~1.3x kernel perf improvement (#57053)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-10-05 14:54:11 -04:00
Raphaël RiallandandSimon Veitner c32495d58f [watermark] compatibility validation (#56801)
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
2026-10-05 14:37:13 -04:00
Almog Tavor 171d1b9ef6 [Elastic EP] Support Model Runner V2 (#53934)
Signed-off-by: almogtavor <almogtavor@gmail.com>
Signed-off-by: Almog Tavor <70065337+almogtavor@users.noreply.github.com>
2026-09-29 00:20:40 +00:00
Nick Hill 09fe178dba Revert "[DSpark] Support pipeline-parallel targets in aggregated serving (#56956)" (#58484) 2026-09-24 08:48:36 -07:00
157bcb7c48 [DSpark] Support pipeline-parallel targets in aggregated serving (#56956)
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
2026-09-23 04:52:28 -07:00
d43bb2f37f [3/N] HiSparse: host-resident sparse-MLA decode hot-buffering (#53781)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Matthew Bonanni <matthew@neuralmagic.com>
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
2026-09-12 00:36:47 -04:00
Kevin H. LuuandCodex 6865e67f0b [Bugfix] Defer adaptive verification until after kernel warmup (#55455)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-06 11:31:40 -07:00
+5 e126687a9a [Model] Support Qwen3.8-Flash-Next (#53896)
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
Signed-off-by: andyluo7 <andy.luo@amd.com>
Signed-off-by: lishunyang12 <lishunyang03@gmail.com>
Signed-off-by: Harry Huang <huanghaoyan.hhy@alibaba-inc.com>
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: KungYork <firelikest@163.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: andyluo7 <43718156+andyluo7@users.noreply.github.com>
Co-authored-by: Isotr0py <mozf@inferact.ai>
Co-authored-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: lishunyang12 <lishunyang03@gmail.com>
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-08-31 05:57:56 +00:00
Tianyu Guo 7ca49fbe4b [Refactor][Model Runner V2][Multimodal] Move the encoder-only path out of the shared runner (#53176)
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
2026-08-22 05:11:44 +00:00
d608dfabfd [Bugfix][MRV2] Reserve spec-decode lookahead blocks in V2 warmup (#51438)
Signed-off-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com>
Co-authored-by: Janelle Cai <janelle.cai@modal.com>
2026-08-08 23:58:43 +00:00
Wentao Ye 4f819f801b [Model Runner v2] E/P/D disaggregation support (#38390)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-08-04 15:52:23 +00:00
Taneem Ibrahim 43eaefba5a [ModelRunner V2] Enable sequence pooling for embedding and classification models (#48791)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
2026-07-29 06:30:45 -07:00
Nick Hill 0d0504b54c [Core] Warm up runner-owned Triton kernels before the first request (#49903) 2026-07-28 07:58:02 -07:00
Nick HillandTaneem Ibrahim fdaa0d9e59 [ModelRunner V2] Support encoder-only attention (#49331)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
2026-07-27 00:38:44 +00:00
Nick Hill 8cfeb84dba [ModelRunner V2] Warmup cross-attn properly in encoder-decoder case (#47308) 2026-07-01 12:36:48 -07:00
Nick Hillandziminghuang fcaa84efa7 [BugFix] Gate MRV2 mixed sparse-MLA warmup on max_num_seqs > 1 (#47050)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: ziminghuang <ziminghuang@inferact.ai>
2026-06-30 16:31:27 +01:00
zhrrr 61ab70ec3b [Model Runner V2] support mamba hybrid models align prefix cache (#42406)
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
2026-06-29 14:09:16 -07:00
44d95069e9 Enable DeepSeek V4 and GLM-5.1 on SM120 (#43477)
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
2026-06-22 11:54:14 -07:00
eb28452b10 [Model] Add DiffusionGemma Support (#45163)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Martin Kukla <martin.kukla@cantab.net>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Dipika Sikka <dsikka@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
Co-authored-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
Co-authored-by: Alec Kohlhoff <134344302+aleckohlhoff@users.noreply.github.com>
Co-authored-by: Porras Huang <20535584+porrashuang@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: scoootscooob <167050519+scoootscooob@users.noreply.github.com>
2026-06-11 22:17:35 -07:00
Nick Hill e9f331d72e [MRV2] Ensure warmup covers prefill path (#40746)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-04-24 01:33:26 +00:00
Woosuk KwonandNick Hill ffb5b32b5f [MRV2] Consider spec decoding in warmup (#37812)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-03-23 17:45:43 +00:00
Nick Hill b354686524 [Model Runner V2] Fix warmup for pipeline parallel (#36280)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-03-06 16:58:51 -08:00
Nick Hill a73af584fe [Model Runner V2] Fix warmup for very small kvcache and/or blocksizes (#36176)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-03-05 14:48:10 -08:00
Kunshang Ji 66a2209645 [Hardware] Replace torch.cuda.synchronize() api with torch.accelerator.synchronize (#36085)
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
2026-03-05 10:36:39 +00:00
Nick Hill b1d9f5372d [Model Runner V2] Warmup kernels (#35172)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-02-27 10:43:30 -08:00