MOVZX and Hinata
5aa2ac64f8
[Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
...
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.
- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
render routes, and vllm/config/speech_to_text.py. GenerationTask is
generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
Multi, Offloading and Example connectors stay; production uses the
CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
arch gate, declarations and op registrations. The CUDA build never
defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
fail with "The model ... does not exist. Valid aliases: ...".
Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 21:06:19 +07:00
Stefano Castagnetta and Kevin H. Luu
784d1fff0c
[MRV2][Spec Decode] Quantized draft lm_head for drafters that share the target head ( #59973 )
...
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 22:06:40 +00:00
Sandeep Agrawal and Kevin H. Luu
21e207efa1
[Bugfix][MRV2][Spec Decode] Honor EAGLE3 single-state draft configuration ( #61563 )
...
Signed-off-by: Sandeep Agrawal <sandeepa@nvidia.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-14 12:09:06 -07:00
Giancarlo Delfin
6ee9be6b3c
[Spec Decode] Rename AR speculators to StandaloneAR / TargetDependentAR ( #60335 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-10-07 14:01:59 -04:00
c4fc3c7d32
[Bugfix][Spec Decode] Per-module LM heads for multi-layer MTP on Model Runner V2 ( #58921 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: chanh <channguyen@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 01:09:14 -07:00
siyu and Isotr0py
c723a831a8
[Fast Start] Cache the MTP draft model in a separate daemon group ( #57312 )
...
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-09-22 03:13:38 +00:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
Canlin Guo and Codex
dca96bf97b
[Bugfix][Spec Decode] Avoid fastsafetensors deadlock for PP draft models ( #54416 )
...
Signed-off-by: Canlin <canlinguosdu@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-12 14:22:30 +08:00
e77daef89e
[Model] Support DeepSeek-V4.1-Flash ( #56214 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
Signed-off-by: andyluo7 <andy.luo@amd.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: andyluo7 <andy.luo@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-11 02:11:19 -07:00
9c2d21046b
[Bugfix][Spec Decode] Honour the draft's moe_backend on Model Runner V2 ( #54788 )
...
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-08 16:50:07 -07:00
d87a440f88
[Core][MRV2] Support eagle3 spec decode with pipeline parallel ( #50514 )
...
Signed-off-by: Yongqin Wang <yongqinwang@roblox.com >
Co-authored-by: Cursor <noreply@cursor.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-09-05 11:45:34 +08:00
Stefano Castagnetta
2524051385
[Bugfix][Spec Decode] Honour the draft's attention_backend on Model Runner V2 ( #54826 )
...
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
2026-09-04 12:23:19 -07:00
Michael Goin and Codex
a7d00ec051
[Bugfix] DFlash fc sized wrong when num_target_layers != num_hidden_layers ( #48524 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-07-21 14:42:52 -07:00
Michael Goin and Claude Opus 4.8
8bfd683901
[Spec Decode] Add kv_cache_dtype to speculative_config to control separately from target ( #48787 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
2026-07-15 23:49:23 -06:00
Michael Goin and Claude Opus 4.8
0d12618e98
[Spec Decode] Support hybrid (SWA + full attention) DFlash drafters ( #47914 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-07-08 11:12:45 -04:00
f5a8d73377
[Spec Decode] DSpark ( #46995 )
...
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com >
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai >
Co-authored-by: mgoin <mgoin64@gmail.com >
2026-07-01 08:30:24 -07:00
Kunshang Ji
fb42e5219e
[Platform] Replace torch.cuda.mem_get_info with torch.accelerator.get_memory_info ( #44825 )
...
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com >
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com >
2026-06-30 14:39:52 +08:00
Jee Jee Li
ebbb2d55ac
[CI/Build][Bugfix] Fix SD LoRA ( #45941 )
...
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com >
2026-06-18 00:34:15 +00:00
Benjamin Chislett and Giancarlo Delfin
0bae1d3848
[MRV2][Spec Decode] DFlash ( #44586 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com >
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com >
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-06-10 08:47:46 -07:00
Jared Wen and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
32f34d3935
[feature] add index share feature for DSA MTP ( #44420 )
...
Signed-off-by: JaredforReal <w13431838023@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-06 22:04:14 -07:00
Giancarlo Delfin
ceb0111a90
[Model Runner V2][Spec Decode] Add Gemma4 MTP support ( #43241 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-06-04 00:51:06 +00:00
Giancarlo Delfin and Woosuk Kwon
91945b6e4a
[Bug Fix][Model Runner V2][Spec Decode] Warmup & capture with different attention states for speculator prefill ( #44253 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
2026-06-03 13:32:40 -07:00
zhrrr
1edfd09ffd
[Model Runner V2] Use actual batch max_seq_len for attn metadata ( #43991 )
...
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com >
2026-06-02 06:07:56 +00:00
Lucas Wilkinson
4721bb3aa4
[MRV2] Remove Eagle's dedicated CUDA graph pool ( #44078 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
2026-05-31 22:00:33 -07:00
Woosuk Kwon
27fa5aa3b9
[MRV2] Support breakable CUDA graph ( #44050 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
2026-05-30 09:40:52 -07:00
Mengqing Cao and Nick Hill
1223732dda
[ModelRunnerV2][Hybrid model] Support kernel block size in hybrid model ( #38831 )
...
Signed-off-by: MengqingCao <cmq0113@163.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Signed-off-by: Mengqing Cao <cmq0113@163.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-05-28 00:55:55 +00:00
Nicolò Lucchesi and Nick Hill
fba010dd74
[Bugfix][MRV2] Fix KVCache tensor explicit kernel_block_size dim ( #42766 )
...
Signed-off-by: NickLucche <nlucches@redhat.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-05-18 20:25:41 -07:00
PatchyTIS and PatchouliTaisa
0162596603
[Model Runner V2] FP32 gumbel sampling. ( #41775 )
...
Signed-off-by: PatchouliTaisa <patchychen@tencent.com >
Co-authored-by: PatchouliTaisa <patchychen@tencent.com >
2026-05-15 09:20:08 -07:00
Siddharth Bedekar
f51f6844f9
[Bugfix][Spec Decode] Wire draft_probs into probabilistic draft_model rejection ( #40269 )
2026-05-13 21:04:03 -04:00
Nick Hill
a505cf807e
[ModelRunner V2] Share identical MTP weights ( #42538 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-05-13 18:57:04 +00:00
zhrrr
7a08b34fbf
[Model Runner V2] support qwen35 / mamba hybrid model ( #35520 )
...
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com >
2026-05-07 09:31:05 -07:00
Giancarlo Delfin
e1e4646b06
[Model Runner V2] Rebuild attn metadata between draft decode steps ( #41162 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-05-05 00:44:55 +00:00
Giancarlo Delfin
4c7c69b4e0
[Model Runner V2] Skip attention metadata rebuild before draft prefill ( #40410 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-04-27 15:38:05 -07:00
Giancarlo Delfin
f5f9878514
[Model Runner V2] Fix rejection sampling acceptance rate gap vs MRV1 ( #40651 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-04-26 19:12:08 -07:00
+7
4d51588e23
[Feat] DeepSeek V4 Rebased ( #40860 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Signed-off-by: qizixi <zixi@inferact.ai >
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <yongye@inferact.ai >
Co-authored-by: Simon Mo <simon@inferact.ai >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: youkaichao <youkaichao@gmail.com >
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com >
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com >
Co-authored-by: khluu <khluu000@gmail.com >
Co-authored-by: qizixi <zixi@inferact.ai >
Co-authored-by: Zhewen Li <zhewenli@inferact.ai >
2026-04-26 18:31:08 -07:00
Giancarlo Delfin
3bfe55a037
[Model Runner V2] Disable piecewise cudagraph mode fallback for eagle draft decodes ( #39773 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-04-14 17:47:57 -07:00
Giancarlo Delfin
ccf90ba784
[Model Runner V2] Add full cuda graph support for eagle prefill ( #37588 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-04-13 16:01:24 -07:00
zhrrr
8f121f7879
[Model Runner V2] support auto resolve cudagraph mode/sizes based on attn backend ( #32936 )
...
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com >
2026-04-10 08:27:15 -07:00
Giancarlo Delfin
384e4d5f48
[Model Runner V2] Rebuild attention metadata before eagle decode full… ( #38311 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-03-27 13:46:42 -07:00
Giancarlo Delfin
c32e97602d
[Model Runner V2] Enable forcing a specific acceptance rate during rejection sampling ( #38045 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-03-26 13:38:12 -07:00
Giancarlo Delfin and Woosuk Kwon
b3e846017d
[Model Runner V2] Support multi-modal embeddings for spec decode model ( #36097 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
2026-03-22 02:48:43 -07:00
Giancarlo Delfin
dcee9be95a
[Model Runner V2] Fix draft logits not populated during cudagraph replay ( #37639 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-03-20 07:43:47 +00:00
Giancarlo Delfin
39474513f6
[Model Runner V2] fix draft attention metadata generation ( #37364 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-03-19 21:05:15 -07:00
Benjamin Chislett
8b346309a5
[Refactor] Consolidate SupportsEagle ( #36063 )
...
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com >
2026-03-13 23:22:40 +00:00
Giancarlo Delfin
c77181e534
[Model Runner V2] Add probabilistic rejection sampling for spec decoding ( #35461 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-03-11 14:04:32 -07:00
Lucas Wilkinson
483463f735
[MRV2] Extensible CG dispatch rework ( #35959 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
2026-03-09 13:58:45 -07:00
zhrrr and Giancarlo Delfin
97286a20ed
[Model Runner V2] support dp & ep for spec decoding ( #35294 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com >
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-03-03 15:19:45 -08:00
Woosuk Kwon
a0a5178ab4
[Model Runner V2] Use ModelState.prepare_attn() for cuda graph capture [5/N] ( #35774 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
2026-03-02 20:06:27 -08:00
Woosuk Kwon
1a014a0a93
[Model Runner V2] Move MM encoder to Model States [3/N] ( #35564 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
2026-02-27 18:32:38 -08:00