162 Commits
Author SHA1 Message Date
MOVZXandHinata 5aa2ac64f8 [Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.

- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
  scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
  CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
  CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
  render routes, and vllm/config/speech_to_text.py. GenerationTask is
  generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
  Multi, Offloading and Example connectors stay; production uses the
  CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
  arch gate, declarations and op registrations. The CUDA build never
  defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
  fail with "The model ... does not exist. Valid aliases: ...".

Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-19 21:06:19 +07:00
MOVZXandHinata 8dc54f35dd [Offloader] Stream every multimodal tower parameter to host RAM
Qwen3_VisionTransformer derived its compute device from
patch_embed.proj.weight and fed pos_embed.weight straight into the
position-embedding interpolation kernel, so the last two anchor weights
had to stay in VRAM for a tower to be streamable at all.

Record the device the tower was built on before the offloader moves
anything, and fall back to it when no weight is on the accelerator.
Move the interpolation table to that device per encode, which costs one
8.4 MiB host-to-device copy for a 456 MiB tower. Towers now leave no
parameters in VRAM between encodes; only small buffers such as the fused
input norm and rotary inv_freq remain.

Also let a tower restrict streaming through tower_offload_children, and
warn when a tower resolves its device property to another device type,
so a tower that reads an anchor weight reports the cause instead of a
Triton pointer error.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-16 16:07:53 +07:00
Bruno VolpatoandCodex 61f084b5f3 [Bugfix][Multimodal] Fix per-video cache option reuse (#55203)
Signed-off-by: bvolpato <brunocvcunha@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-11 06:21:55 +00:00
SeongJun Leeandlesj0610 a98247ab4d [Model] Drop the 4x M-RoPE cache headroom for Qwen3-VL, Qwen3.5 and Qwen3.8 (#59853)
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
Signed-off-by: lesj0610 <lesj0610@gmail.com>
Co-authored-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-10-10 06:51:52 +00:00
Daniel M. García-Ocaña Hernández 295a0b8d64 [Bugfix][Multimodal] Fix flat/scoped mm_processor_kwargs merge and resolution (#56372)
Signed-off-by: Daniel M. García-Ocaña Hernández <danielgarciaocana@gmail.com>
2026-09-29 03:58:39 +00:00
cjackalandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 491f44adfa [Model] Extend device-side mm normalization to Qwen3VL/Qwen3.5/Qwen4Next (#51289)
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-29 11:44:54 +08:00
Ashraf Bhuiyan 5113cf9d16 [Mypy] Fix mypy typing for Qwen and Qianfan models (#58046)
Signed-off-by: Ashraf Bhuiyan <mbhuiyan@redhat.com>
2026-09-26 14:31:51 +00:00
Harry MellorandClaude Opus 5.5 d881812500 [Chore] Use Transformers v5 names and drop redundant processor use_fast (#58550)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-24 14:14:10 +00:00
Harry MellorandClaude Opus 5 8a5cf54387 [Multimodal] Type dummy options per modality (#57576)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 17:29:11 +01:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:57:24 -07:00
Cyrus Leung 0b9e018fb9 [MM] Further cleanup _apply_hf_processor_main (#53610)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-09-16 10:21:00 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
linitra24 9d4d9aa5bc [LoRA][Refactor] Unify multimodal LoRA token count hooks (#55071)
Signed-off-by: linitra24 <renshuang.zhou@daocloud.io>
2026-09-14 13:19:15 +00:00
Alexander CrettenandandClaude 7d8d71e989 [Bugfix] Qwen3-VL(-MoE): pass architectures to with_hf_config for pipeline parallelism (#43272)
Signed-off-by: Xonder <xonder@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-10 14:49:35 +01:00
Yunzhu LuandIsotr0py 442d36031c [MM][CG] Enable encoder CUDA Graph for MiniCPM-V (#42785)
Signed-off-by: YunzhuLu <lucia.yunzhu@gmail.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
2026-09-10 06:28:48 +00:00
Nick Hill 8277c42e4c [Perf] Ensure async h2d copies are pinned in more places (#55202) 2026-09-04 17:17:01 -07:00
Prudhvi Vuda 7292ee2791 [Bugfix][Multimodal] Honor modality-scoped mm_processor_kwargs (#53808)
Signed-off-by: Prudhvivuda <prudhvi12042001@gmail.com>
2026-08-31 06:37:12 +00:00
Isotr0py 555ea65e8c [Frontend] Add video embeds input support (#54242)
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
2026-08-31 03:20:47 +00:00
Devon KrismanandClaude Fable 5 4f78a8fdd0 [Model] Honor cap_pixels_per_frame in Qwen3-VL memory profiling (#54380)
Signed-off-by: Devon Krisman <git@krisman.dev>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-30 08:07:29 +00:00
b016ed8ea3 [Bugfix][Multimodal] Release Qwen2.5-VL and Qwen3-VL RoPE caches with the model (#54346)
Signed-off-by: waizuichougou <2082431897@qq.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-08-30 11:10:29 +08:00
Taneem IbrahimandWentao Ye 21fa2c5a2a [Mypy] Fix mypy typing for model interfaces and H/I models (#54079)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-08-28 16:49:19 -04:00
Cyrus LeungandQwen Code ebcd606467 [MM] Simplify prompt updates: replace PromptSeq with list[int] (#53372)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Qwen Code <noreply@alibaba-inc.com>
2026-08-23 14:50:32 +08:00
Cyrus Leung 0b19ebcacd [MM] Simplify _apply_hf_processor_main (#53275)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-08-22 01:49:37 -07:00
linitra24 e85d1b69cf [Bugfix][LoRA] Use an explicit capability flag for tower connector LoRA (#53092)
Signed-off-by: linitra24 <Joy25810@foxmail.com>
2026-08-21 09:45:54 +00:00
Cyrus Leung b00f475f09 [MM] Remove text components from ProcessorInputs (#53093)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-08-20 20:42:53 -07:00
liuzhenwei b1d9337e92 [EPD] Allow KV consumers to omit MM embeddings (#52697)
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
2026-08-19 02:16:08 +00:00
Nick HillandClaude Opus 4.8 615d4cfade [Core] Check for GPU<->CPU syncs during CI (#43107)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-14 17:42:05 -07:00
Harry MellorandClaude Opus 5 80d6d557f3 Standardise weight tying on ParallelLMHead.tie_weights (#52147)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-13 15:10:09 +00:00
Li, Jiang 373592ef57 [CPU] Ship triton-cpu wheel and fix several hardcoded pin_memory=True (#52092)
Signed-off-by: jiang1.li <jiang1.li@intel.com>
2026-08-13 14:40:53 +08:00
Max HuandNick Hill b1b752042f Avoid long-blocking H2D copies in ViT (#51841)
Signed-off-by: Max Hu <hyoung2991@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-08-12 22:54:04 +08:00
Tianyu Guo 653ebb52df Add torch compile for qwen3_vl encoder (#40116)
Signed-off-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn>
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
2026-08-08 08:23:18 +00:00
Tianyu Guo b30291cf0c [EPD] Remove duplicate image preprocessing in EPD and enable preprocess on GPU (#50390)
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
2026-08-05 15:59:54 -07:00
Wei-Cheng (Wayne) ChiuandOpenAI Codex 7794b1e08b [Bugfix] Skip Qwen3 deepstack buffers without vision (#49397)
Signed-off-by: WEI CHENG CHIU <waynehacking8@gmail.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-08-05 12:46:23 +08:00
nvbfalk 60915c972c [Feature] Add VidCom2 video token pruning (#47750)
Signed-off-by: Benedikt Falk <bfalk@nvidia.com>
2026-07-28 03:20:17 +00:00
SYLAR 7c2acd38b7 [Bugfix] Qwen3-VL/Qwen-Omni: honor max_pixels/min_pixels for video prompts (#49015) 2026-07-18 10:29:11 -07:00
Harry Mellor dd94484577 Bump Transformers version to 5.10.4 (#41359)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-07-07 05:13:28 -07:00
Harry Mellor 1ab9522935 Remove more unnecessary load_weights methods (#47058)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-06-30 15:22:16 +01:00
Nick Hillandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 84c62e1cbd [Model Runner V2][MM] Support EVS (#46535)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-24 10:18:56 -04:00
Ting SUN 12fe2a9aac [Bugfix][Qwen3-VL] Fix multi-video crash with list-valued fps/num_frames (#46305)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
2026-06-21 14:31:23 -07:00
L丶andmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> d272418f45 [Perf] Optimize Qwen3-VL multi-video prompt processing (#46026)
Signed-off-by: Sirius29 <422058530@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-20 07:09:18 -07:00
ad32608e24 [MM][Perf][CG] Support dual-path ViT full CUDA graph for DeepSeek-OCR (#43586)
Signed-off-by: shen-shanshan <467638484@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
2026-06-16 04:35:20 -07:00
ff5a30cfac [Bugfix] Replace deprecated Qwen2VLImageProcessorFast with Qwen2VLImageProcessor (#42700)
Signed-off-by: abinggo <107740309+abinggo@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
2026-06-12 21:04:31 -07:00
Rui "Garry" GaoandCyrus Leung 4b87b3e845 [Bugfix] fix EVS for qwen3-vl (#44205)
Signed-off-by: Rui "Garry" Gao <garrygaogg@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-06-04 11:06:51 +00:00
Weida Hong dfe8ba7c80 Adjust design around encoder_cudagraph_forward (#42288)
Signed-off-by: Weida Hong <wdhongtw@google.com>
2026-05-29 03:02:52 +00:00
Dakai An 5963c19478 Fix Qwen3-VL and Qwen3-omni-thinker accuracy degradation from deepstack inputs under torch.compile (#43617)
Signed-off-by: Dakai An <dakaian108@gmail.com>
2026-05-27 15:34:08 -07:00
Isotr0py f0feb15e7f [Multimodal] Simplify ViT CUDA graph interfaces (#41234)
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
2026-05-22 22:31:00 +08:00
Nick Hill b82e908b4c [Perf][4/n] Eliminate various GPU<->CPU syncs (#42347)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-05-19 10:35:54 -04:00
Baorun (Lauren) Mu a7737cb4f3 [Fix] Misc Fixes in ViT CUDA Graph (#38040)
Signed-off-by: Baorun Mu <bmu@nvidia.com>
2026-05-14 23:49:06 +08:00
SILONG ZENG cee6751e54 [Bugfix][Qwen3-VL] Fix pipeline-parallel deepstack initialization (#42394)
Signed-off-by: MrZ20 <2609716663@qq.com>
2026-05-13 10:58:42 +00:00
Isotr0py 22631f80a0 [Bugfix] Remove invalid deepstack boundary check for Qwen3-VL (#40932)
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
2026-04-27 07:27:06 +00:00