 Giancarlo Delfin
|
dec13a33b7
|
[Model Runner V2][Spec Decode] Add multi-layer MTP speculator (#48892)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-07-30 15:54:15 -07:00 |
|
|
Taneem Ibrahim
|
43eaefba5a
|
[ModelRunner V2] Enable sequence pooling for embedding and classification models (#48791)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
|
2026-07-29 06:30:45 -07:00 |
|
|
Nick HillandJeff Ma
|
6fbbcf2151
|
[BugFix] Stop dummy runs from writing mamba state through stale block-table rows (#49757)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Jeff Ma <jeffjma@umich.edu>
Co-authored-by: Jeff Ma <jeffjma@umich.edu>
|
2026-07-29 01:09:41 +00:00 |
|
|
Wentao Ye
|
b5bcb3ce88
|
[Refactor] Remove dead code in multiple files (#49745)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
|
2026-07-27 15:58:26 -04:00 |
|
|
Nick Hill
|
50aa830482
|
[BugFix][MRV2] Don't create dummy requests longer than max_model_len (#49751)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-07-27 02:04:20 +00:00 |
|
|
Nick HillandTaneem Ibrahim
|
fdaa0d9e59
|
[ModelRunner V2] Support encoder-only attention (#49331)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
|
2026-07-27 00:38:44 +00:00 |
|
|
|
0b0bd2b5f6
|
[Feature] Add fault tolerance framework (simplified) for DP+EP external LB deployments (#44428)
Signed-off-by: fangyuchu <fangyuchu@qq.com>
Signed-off-by: a798347923 <2645302020@qq.com>
Signed-off-by: TianZhuo <2770730562@qq.com>
Signed-off-by: a798347923 <39047817+a798347923@users.noreply.github.com>
Signed-off-by: 205150940 <112750056+205150940@users.noreply.github.com>
Signed-off-by: w00689259 <wangzhuo66@huawei.com>
Signed-off-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com>
Signed-off-by: zWaNg3 <389750525@qq.com>
Signed-off-by: yzchang-plus <1078477584@qq.com>
Signed-off-by: Jade Zheng <zheng.shoujian@outlook.com>
Co-authored-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com>
Co-authored-by: a798347923 <2645302020@qq.com>
Co-authored-by: TianZhuo <2770730562@qq.com>
Co-authored-by: 205150940 <112750056+205150940@users.noreply.github.com>
Co-authored-by: a798347923 <39047817+a798347923@users.noreply.github.com>
Co-authored-by: w00689259 <wangzhuo66@huawei.com>
Co-authored-by: zWaNg3 <389750525@qq.com>
Co-authored-by: yzchang-plus <1078477584@qq.com>
Co-authored-by: Jade Zheng <zheng.shoujian@outlook.com>
|
2026-07-25 11:49:09 -04:00 |
|
|
Nick Hillandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
521aa80f71
|
[Core] Simplify KVBlockZeroer index tensor handling (#48399)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-07-23 18:12:46 +08:00 |
|
|
Isotr0py
|
ea0e9c8f2e
|
[MRV2] Add encoder cache profiling implementation (#47985)
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
|
2026-07-20 20:18:14 -07:00 |
|
|
|
b6ff8a2f50
|
[Core] Add MRV2 virtual-batch PCP for MLA (#46570)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
|
2026-07-19 02:53:15 +00:00 |
|
|
|
41ea2dd44a
|
[Bugfix][V1/V2] Fix prompt_logprobs to respect logprobs_mode (#47680)
Signed-off-by: Wojciech Wais <wojciech.wais@gmail.com>
Signed-off-by: Federico Kamelhar <209537060+fede-kamel@users.noreply.github.com>
Signed-off-by: Allen Shen <aoshen@inferact.ai>
Co-authored-by: Wojciech Wais <wojciech.wais@gmail.com>
Co-authored-by: Federico Kamelhar <209537060+fede-kamel@users.noreply.github.com>
|
2026-07-17 21:58:59 +01:00 |
|
|
Michael GoinandClaude Fable 5
|
ecf4aa5ce2
|
[Bugfix] Fix FlashInfer non-causal draft attention (DFlash/DSpark) on Blackwell (#48167)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
2026-07-15 12:44:01 -07:00 |
|
|
Nick Hill
|
26587f9519
|
[BugFix][ModelRunner V2] Fix stale attn metadata in speculator prefill cudagraph capture (#48261)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-07-13 09:39:15 -07:00 |
|
|
|
fc1c548093
|
Runtime Draft Weight Update for Speculative Decoding (#46725)
Signed-off-by: vx120 <893600387@qq.com>
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: crp0128 <191679376@qq.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
2026-07-11 22:51:53 -07:00 |
|
|
Jiangyun ZhuandYifan Qiao
|
481e481be7
|
[2/N][Core] support partial prefix cache hit for hybrid model (#46384)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
|
2026-07-12 05:37:51 +00:00 |
|
|
|
85b3a7264b
|
[Bugfix][Model Runner V2] Order uniform decodes first so spec decodes aren't misclassified as prefills (#47381)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
|
2026-07-09 14:26:27 +01:00 |
|
|
Benjamin ChislettandWentao Ye
|
1cd75b3dd4
|
[Bugfix] Fix race condition in KVBlockZeroer (#48085)
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
|
2026-07-09 09:18:19 +00:00 |
|
|
|
07516fda67
|
[MRV2][SD] Make Dynamic SD comatible with Full Cuda Graphs (#45953)
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
|
2026-07-03 23:58:27 -04:00 |
|
|
Nick Hill
|
276b837dc4
|
[ModelRunner V2][BugFix] Free all model refs on shutdown (#47483)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-07-02 19:32:48 -07:00 |
|
|
|
f5a8d73377
|
[Spec Decode] DSpark (#46995)
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: mgoin <mgoin64@gmail.com>
|
2026-07-01 08:30:24 -07:00 |
|
|
Aaron HaoandClaude Sonnet 4.6
|
77a9c5ae28
|
Weight sync refactor + move sparse nccl engine (#44353)
Signed-off-by: hao-aaron <ahao@anyscale.com>
Signed-off-by: haoaaron <ahao@anyscale.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
2026-07-01 01:25:19 -07:00 |
|
|
Kunshang Ji
|
fb42e5219e
|
[Platform] Replace torch.cuda.mem_get_info with torch.accelerator.get_memory_info (#44825)
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
|
2026-06-30 14:39:52 +08:00 |
|
|
zhrrr
|
61ab70ec3b
|
[Model Runner V2] support mamba hybrid models align prefix cache (#42406)
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
|
2026-06-29 14:09:16 -07:00 |
|
|
Nick Hill
|
8ad4a01825
|
[ModelRunner V2] Simplify recent UnlimitedOCR-related changes (#46975)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-06-29 09:56:17 -07:00 |
|
|
Ilya MarkovandMarkov Ilya
|
a2abce646f
|
[EPLB] Mask padding in EPLB load recording (#38128)
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Markov Ilya <markovilya19@gmail.com>
|
2026-06-28 19:43:58 -07:00 |
|
|
|
c6741b2ad4
|
[Model] Support Unlimited OCR (#46564)
Signed-off-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Roger Wang <hey@rogerw.io>
|
2026-06-27 23:09:18 -07:00 |
|
|
Nick Hill
|
c6dd32a810
|
[ModelRunner V2] Support realtime embeddings (#46762)
|
2026-06-26 19:42:27 -07:00 |
|
|
Nick Hill
|
1d41009e81
|
[ModelRunner V2] Fix cross-attention block table sizing (#46753)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-06-26 16:34:21 -07:00 |
|
|
Nick Hillandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
84c62e1cbd
|
[Model Runner V2][MM] Support EVS (#46535)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-06-24 10:18:56 -04:00 |
|
|
Woosuk Kwonandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
0a3e2dbc09
|
[Optimization] Skip DP padding tokens in MoE (#46428)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-06-23 14:54:46 -07:00 |
|
|
|
3e6529cc0e
|
[Bugfix][Spec Decode] Fix EAGLE drafter multimodal encoder cache misses (#46315)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
|
2026-06-22 18:14:02 +00:00 |
|
|
Nick Hill
|
7df3d7dada
|
[Core] Ensure memory is pinned prior to async h2d copy (#45424)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-06-20 20:02:24 -07:00 |
|
|
Nick Hill
|
14b438a98b
|
[ModelRunnerV2] Various model/config compatibility fixes (#45868)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-06-17 03:23:01 +00:00 |
|
|
|
eacff17c8d
|
[Model Runner V2][Bugfix] Fix MRV2 LoRA warmup (#35536)
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-06-15 13:17:23 -07:00 |
|
|
Nick Hill
|
cf027b86af
|
[Core] Simplify MRV2 async output handling (#45442)
|
2026-06-13 18:15:36 -07:00 |
|
|
Wentao Yeandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
fbc3a1907a
|
[Bug] Migrate Reset cache for both v2 and v1 model runner (#42759)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-06-12 09:38:12 -04:00 |
|
|
|
eb28452b10
|
[Model] Add DiffusionGemma Support (#45163)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Martin Kukla <martin.kukla@cantab.net>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Dipika Sikka <dsikka@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
Co-authored-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
Co-authored-by: Alec Kohlhoff <134344302+aleckohlhoff@users.noreply.github.com>
Co-authored-by: Porras Huang <20535584+porrashuang@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: scoootscooob <167050519+scoootscooob@users.noreply.github.com>
|
2026-06-11 22:17:35 -07:00 |
|
|
Benjamin ChislettandGiancarlo Delfin
|
0bae1d3848
|
[MRV2][Spec Decode] DFlash (#44586)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-06-10 08:47:46 -07:00 |
|
|
Wentao Ye
|
2c27c294c0
|
[Model Runner V2] Fix mrv2 mm lora issue (#44450)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
|
2026-06-08 14:30:09 -04:00 |
|
|
Giancarlo Delfin
|
ceb0111a90
|
[Model Runner V2][Spec Decode] Add Gemma4 MTP support (#43241)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
|
2026-06-04 00:51:06 +00:00 |
|
|
Giancarlo DelfinandWoosuk Kwon
|
91945b6e4a
|
[Bug Fix][Model Runner V2][Spec Decode] Warmup & capture with different attention states for speculator prefill (#44253)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-06-03 13:32:40 -07:00 |
|
|
Nick Hill
|
e15f20258b
|
[ModelRunnerV2] Avoid pipeline parallel bubbles (#42187)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-06-02 14:02:01 -07:00 |
|
|
Nick Hillandyewentao256
|
da107a59e5
|
[MRV2] Also enable MRV2 for Llama and Mistral dense models (#43458)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: yewentao256 <zhyanwentao@126.com>
|
2026-06-02 11:18:46 -07:00 |
|
|
zhrrr
|
8a9eb40808
|
[Model Runner V2] Support zeroing freshly allocated KV blocks for hybrid + fp8 KVCache (#43990)
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
|
2026-06-02 05:56:53 +00:00 |
|
|
Woosuk Kwon
|
27fa5aa3b9
|
[MRV2] Support breakable CUDA graph (#44050)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
|
2026-05-30 09:40:52 -07:00 |
|
|
Nick Hillandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
7e53283b1c
|
[Core] Cleanup KVConnector handling with PP + fix MRV2 (#43732)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-05-28 13:12:03 -07:00 |
|
|
Mengqing CaoandNick Hill
|
1223732dda
|
[ModelRunnerV2][Hybrid model] Support kernel block size in hybrid model (#38831)
Signed-off-by: MengqingCao <cmq0113@163.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Mengqing Cao <cmq0113@163.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
|
2026-05-28 00:55:55 +00:00 |
|
|
Nick HillandWentao Ye
|
8c94938cfb
|
[MRV2][BugFix] Fix KV connector handling in spec decode case (#43719)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
|
2026-05-27 06:37:56 +00:00 |
|
|
Nick Hill
|
47d4407d7c
|
[Model Runner V2] Support sharing kv cache layers (#35045)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-05-22 22:18:23 +00:00 |
|
|
haosdentandWentao Ye
|
5ea76fa89a
|
[CI] Fix test_lora_with_spec_decode on V2 model runner (#43314)
Signed-off-by: haosdent <haosdent@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
|
2026-05-22 14:24:18 +08:00 |
|