MOVZX and Hinata
5aa2ac64f8
[Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
...
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.
- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
render routes, and vllm/config/speech_to_text.py. GenerationTask is
generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
Multi, Offloading and Example connectors stay; production uses the
CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
arch gate, declarations and op registrations. The CUDA build never
defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
fail with "The model ... does not exist. Valid aliases: ...".
Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 21:06:19 +07:00
zzaebok and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
702ce313f4
[Perf][Multimodal] Sort multimodal encoder inputs by modality ( #60196 )
...
Signed-off-by: Jaebok Lee <jaebok9541@naver.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-09 10:55:36 +08:00
Nick Hill
6c18a54648
[Perf] Avoid h2d copies from non-pinned CPU tensors ( #54299 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-08-29 12:06:30 +08:00
Canlin Guo and Nick Hill
455edc022b
[ModelRunnerV2] Support prompt embeds ( #42963 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
Signed-off-by: Canlin Guo <961750412@qq.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-17 21:58:50 +00:00
Isotr0py and OpenAI Codex
66728feb1f
[MRV2][Multimodal] Enable encoder cuda graph for model runner v2 ( #49852 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-14 09:33:21 +00:00
Li, Jiang
373592ef57
[CPU] Ship triton-cpu wheel and fix several hardcoded pin_memory=True ( #52092 )
...
Signed-off-by: jiang1.li <jiang1.li@intel.com >
2026-08-13 14:40:53 +08:00
Guan-Ming Chiu and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
457a5f34db
[Bugfix][MRV2] Support encoder timing stats in model runner V2 ( #50020 )
...
Signed-off-by: Guan-Ming (Wesley) Chiu <105915352+guan404ming@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-12 00:00:49 +08:00
Wentao Ye
4f819f801b
[Model Runner v2] E/P/D disaggregation support ( #38390 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-08-04 15:52:23 +00:00
Isotr0py
ea0e9c8f2e
[MRV2] Add encoder cache profiling implementation ( #47985 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
2026-07-20 20:18:14 -07:00
wendadawen and Tianyu Guo
69d4f5ef63
[Bugfix][Multimodal] Fix Qwen3-Omni use_audio_in_video with mixed image/video inputs ( #46213 )
...
Signed-off-by: wendadawen <wendadawen@qq.com >
Signed-off-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn >
Co-authored-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn >
2026-07-17 08:31:16 +00:00
Nick Hill
c6dd32a810
[ModelRunner V2] Support realtime embeddings ( #46762 )
2026-06-26 19:42:27 -07:00
3e6529cc0e
[Bugfix][Spec Decode] Fix EAGLE drafter multimodal encoder cache misses ( #46315 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-06-22 18:14:02 +00:00
Nick Hill
7df3d7dada
[Core] Ensure memory is pinned prior to async h2d copy ( #45424 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-06-20 20:02:24 -07:00
Nick Hill
14b438a98b
[ModelRunnerV2] Various model/config compatibility fixes ( #45868 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-06-17 03:23:01 +00:00
Roger Wang
e2bf2b3d84
[Perf] Use bisect for mm feature lookup in model runner v2 ( #45566 )
...
Signed-off-by: Roger Wang <hey@rogerw.io >
2026-06-14 00:22:53 -07:00
Lukas Geiger
4f6eed3bd4
[Core] Simplify multimodal masking ( #34246 )
...
Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com >
2026-04-01 01:18:22 -07:00
Cyrus Leung
6dd302653f
[Misc] Rename group_mm_kwargs_by_modality -> group_and_batch_mm_kwargs ( #36158 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-03-06 12:32:48 +08:00
Woosuk Kwon
da543d1abe
[Model Runner V2] Minor refactoring for EncoderRunner ( #35628 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
2026-03-01 00:15:39 -08:00
Woosuk Kwon
1a014a0a93
[Model Runner V2] Move MM encoder to Model States [3/N] ( #35564 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
2026-02-27 18:32:38 -08:00
Nick Hill
e535d90deb
[ModelRunner V2] Misc minor simplifications and optimizations ( #33467 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-02-01 22:17:14 +00:00
Cyrus Leung
79b6ec6aab
[Bugfix] Fix inconsistent handling of cache reset ( #33481 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-01-31 20:23:41 -08:00
Cyrus Leung
c6e7404cc5
[Multimodal] Simplify MM input definitions ( #33331 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-01-29 13:32:04 +00:00
Woosuk Kwon and Nick Hill
6d86fde09c
[Model Runner V2] Remove UvaBufferPool for cpu->gpu copy ( #33055 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu >
Co-authored-by: Nick Hill <nhill@redhat.com >
2026-01-26 16:47:35 -08:00
Woosuk Kwon
bb1848cd62
[Model Runner V2] Support VLM ( #32546 )
...
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu >
2026-01-18 16:58:51 -08:00