MOVZX and Hinata
5aa2ac64f8
[Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
...
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.
- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
render routes, and vllm/config/speech_to_text.py. GenerationTask is
generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
Multi, Offloading and Example connectors stay; production uses the
CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
arch gate, declarations and op registrations. The CUDA build never
defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
fail with "The model ... does not exist. Valid aliases: ...".
Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 21:06:19 +07:00
MOVZX and Hinata
8dc54f35dd
[Offloader] Stream every multimodal tower parameter to host RAM
...
Qwen3_VisionTransformer derived its compute device from
patch_embed.proj.weight and fed pos_embed.weight straight into the
position-embedding interpolation kernel, so the last two anchor weights
had to stay in VRAM for a tower to be streamable at all.
Record the device the tower was built on before the offloader moves
anything, and fall back to it when no weight is on the accelerator.
Move the interpolation table to that device per encode, which costs one
8.4 MiB host-to-device copy for a 456 MiB tower. Towers now leave no
parameters in VRAM between encodes; only small buffers such as the fused
input norm and rotary inv_freq remain.
Also let a tower restrict streaming through tower_offload_children, and
warn when a tower resolves its device property to another device type,
so a tower that reads an anchor weight reports the cause instead of a
Triton pointer error.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-16 16:07:53 +07:00
Bruno Volpato and Codex
61f084b5f3
[Bugfix][Multimodal] Fix per-video cache option reuse ( #55203 )
...
Signed-off-by: bvolpato <brunocvcunha@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-10-11 06:21:55 +00:00
SeongJun Lee and lesj0610
a98247ab4d
[Model] Drop the 4x M-RoPE cache headroom for Qwen3-VL, Qwen3.5 and Qwen3.8 ( #59853 )
...
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com >
Signed-off-by: lesj0610 <lesj0610@gmail.com >
Co-authored-by: lesj0610 <lesj0610@users.noreply.github.com >
2026-10-10 06:51:52 +00:00
Daniel M. García-Ocaña Hernández
295a0b8d64
[Bugfix][Multimodal] Fix flat/scoped mm_processor_kwargs merge and resolution ( #56372 )
...
Signed-off-by: Daniel M. García-Ocaña Hernández <danielgarciaocana@gmail.com >
2026-09-29 03:58:39 +00:00
cjackal and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
491f44adfa
[Model] Extend device-side mm normalization to Qwen3VL/Qwen3.5/Qwen4Next ( #51289 )
...
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-29 11:44:54 +08:00
Ashraf Bhuiyan
5113cf9d16
[Mypy] Fix mypy typing for Qwen and Qianfan models ( #58046 )
...
Signed-off-by: Ashraf Bhuiyan <mbhuiyan@redhat.com >
2026-09-26 14:31:51 +00:00
Harry Mellor and Claude Opus 5.5
d881812500
[Chore] Use Transformers v5 names and drop redundant processor use_fast ( #58550 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-09-24 14:14:10 +00:00
Harry Mellor and Claude Opus 5
8a5cf54387
[Multimodal] Type dummy options per modality ( #57576 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-18 17:29:11 +01:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Cyrus Leung
0b9e018fb9
[MM] Further cleanup _apply_hf_processor_main ( #53610 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-09-16 10:21:00 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
linitra24
9d4d9aa5bc
[LoRA][Refactor] Unify multimodal LoRA token count hooks ( #55071 )
...
Signed-off-by: linitra24 <renshuang.zhou@daocloud.io >
2026-09-14 13:19:15 +00:00
Alexander Crettenand and Claude
7d8d71e989
[Bugfix] Qwen3-VL(-MoE): pass architectures to with_hf_config for pipeline parallelism ( #43272 )
...
Signed-off-by: Xonder <xonder@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-10 14:49:35 +01:00
Yunzhu Lu and Isotr0py
442d36031c
[MM][CG] Enable encoder CUDA Graph for MiniCPM-V ( #42785 )
...
Signed-off-by: YunzhuLu <lucia.yunzhu@gmail.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-09-10 06:28:48 +00:00
Nick Hill
8277c42e4c
[Perf] Ensure async h2d copies are pinned in more places ( #55202 )
2026-09-04 17:17:01 -07:00
Prudhvi Vuda
7292ee2791
[Bugfix][Multimodal] Honor modality-scoped mm_processor_kwargs ( #53808 )
...
Signed-off-by: Prudhvivuda <prudhvi12042001@gmail.com >
2026-08-31 06:37:12 +00:00
Isotr0py
555ea65e8c
[Frontend] Add video embeds input support ( #54242 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
2026-08-31 03:20:47 +00:00
Devon Krisman and Claude Fable 5
4f78a8fdd0
[Model] Honor cap_pixels_per_frame in Qwen3-VL memory profiling ( #54380 )
...
Signed-off-by: Devon Krisman <git@krisman.dev >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-08-30 08:07:29 +00:00
b016ed8ea3
[Bugfix][Multimodal] Release Qwen2.5-VL and Qwen3-VL RoPE caches with the model ( #54346 )
...
Signed-off-by: waizuichougou <2082431897@qq.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-08-30 11:10:29 +08:00
Taneem Ibrahim and Wentao Ye
21fa2c5a2a
[Mypy] Fix mypy typing for model interfaces and H/I models ( #54079 )
...
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-08-28 16:49:19 -04:00
Cyrus Leung and Qwen Code
ebcd606467
[MM] Simplify prompt updates: replace PromptSeq with list[int] ( #53372 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
Co-authored-by: Qwen Code <noreply@alibaba-inc.com >
2026-08-23 14:50:32 +08:00
Cyrus Leung
0b19ebcacd
[MM] Simplify _apply_hf_processor_main ( #53275 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-08-22 01:49:37 -07:00
linitra24
e85d1b69cf
[Bugfix][LoRA] Use an explicit capability flag for tower connector LoRA ( #53092 )
...
Signed-off-by: linitra24 <Joy25810@foxmail.com >
2026-08-21 09:45:54 +00:00
Cyrus Leung
b00f475f09
[MM] Remove text components from ProcessorInputs ( #53093 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-08-20 20:42:53 -07:00
liuzhenwei
b1d9337e92
[EPD] Allow KV consumers to omit MM embeddings ( #52697 )
...
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com >
2026-08-19 02:16:08 +00:00
Nick Hill and Claude Opus 4.8
615d4cfade
[Core] Check for GPU<->CPU syncs during CI ( #43107 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-08-14 17:42:05 -07:00
Harry Mellor and Claude Opus 5
80d6d557f3
Standardise weight tying on ParallelLMHead.tie_weights ( #52147 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-13 15:10:09 +00:00
Li, Jiang
373592ef57
[CPU] Ship triton-cpu wheel and fix several hardcoded pin_memory=True ( #52092 )
...
Signed-off-by: jiang1.li <jiang1.li@intel.com >
2026-08-13 14:40:53 +08:00
Max Hu and Nick Hill
b1b752042f
Avoid long-blocking H2D copies in ViT ( #51841 )
...
Signed-off-by: Max Hu <hyoung2991@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-12 22:54:04 +08:00
Tianyu Guo
653ebb52df
Add torch compile for qwen3_vl encoder ( #40116 )
...
Signed-off-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn >
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
2026-08-08 08:23:18 +00:00
Tianyu Guo
b30291cf0c
[EPD] Remove duplicate image preprocessing in EPD and enable preprocess on GPU ( #50390 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
2026-08-05 15:59:54 -07:00
Wei-Cheng (Wayne) Chiu and OpenAI Codex
7794b1e08b
[Bugfix] Skip Qwen3 deepstack buffers without vision ( #49397 )
...
Signed-off-by: WEI CHENG CHIU <waynehacking8@gmail.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-08-05 12:46:23 +08:00
nvbfalk
60915c972c
[Feature] Add VidCom2 video token pruning ( #47750 )
...
Signed-off-by: Benedikt Falk <bfalk@nvidia.com >
2026-07-28 03:20:17 +00:00
SYLAR
7c2acd38b7
[Bugfix] Qwen3-VL/Qwen-Omni: honor max_pixels/min_pixels for video prompts ( #49015 )
2026-07-18 10:29:11 -07:00
Harry Mellor
dd94484577
Bump Transformers version to 5.10.4 ( #41359 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-07 05:13:28 -07:00
Harry Mellor
1ab9522935
Remove more unnecessary load_weights methods ( #47058 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-06-30 15:22:16 +01:00
Nick Hill and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
84c62e1cbd
[Model Runner V2][MM] Support EVS ( #46535 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-24 10:18:56 -04:00
Ting SUN
12fe2a9aac
[Bugfix][Qwen3-VL] Fix multi-video crash with list-valued fps/num_frames ( #46305 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
2026-06-21 14:31:23 -07:00
L丶 and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
d272418f45
[Perf] Optimize Qwen3-VL multi-video prompt processing ( #46026 )
...
Signed-off-by: Sirius29 <422058530@qq.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-20 07:09:18 -07:00
ad32608e24
[MM][Perf][CG] Support dual-path ViT full CUDA graph for DeepSeek-OCR ( #43586 )
...
Signed-off-by: shen-shanshan <467638484@qq.com >
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn >
2026-06-16 04:35:20 -07:00
ff5a30cfac
[Bugfix] Replace deprecated Qwen2VLImageProcessorFast with Qwen2VLImageProcessor ( #42700 )
...
Signed-off-by: abinggo <107740309+abinggo@users.noreply.github.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-06-12 21:04:31 -07:00
Rui "Garry" Gao and Cyrus Leung
4b87b3e845
[Bugfix] fix EVS for qwen3-vl ( #44205 )
...
Signed-off-by: Rui "Garry" Gao <garrygaogg@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-06-04 11:06:51 +00:00
Weida Hong
dfe8ba7c80
Adjust design around encoder_cudagraph_forward ( #42288 )
...
Signed-off-by: Weida Hong <wdhongtw@google.com >
2026-05-29 03:02:52 +00:00
Dakai An
5963c19478
Fix Qwen3-VL and Qwen3-omni-thinker accuracy degradation from deepstack inputs under torch.compile ( #43617 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
2026-05-27 15:34:08 -07:00
Isotr0py
f0feb15e7f
[Multimodal] Simplify ViT CUDA graph interfaces ( #41234 )
...
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
2026-05-22 22:31:00 +08:00
Nick Hill
b82e908b4c
[Perf][4/n] Eliminate various GPU<->CPU syncs ( #42347 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-05-19 10:35:54 -04:00
Baorun (Lauren) Mu
a7737cb4f3
[Fix] Misc Fixes in ViT CUDA Graph ( #38040 )
...
Signed-off-by: Baorun Mu <bmu@nvidia.com >
2026-05-14 23:49:06 +08:00
SILONG ZENG
cee6751e54
[Bugfix][Qwen3-VL] Fix pipeline-parallel deepstack initialization ( #42394 )
...
Signed-off-by: MrZ20 <2609716663@qq.com >
2026-05-13 10:58:42 +00:00
Isotr0py
22631f80a0
[Bugfix] Remove invalid deepstack boundary check for Qwen3-VL ( #40932 )
...
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
2026-04-27 07:27:06 +00:00