58 Commits
Author SHA1 Message Date
MOVZXandHinata 5aa2ac64f8 [Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.

- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
  scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
  CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
  CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
  render routes, and vllm/config/speech_to_text.py. GenerationTask is
  generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
  Multi, Offloading and Example connectors stay; production uses the
  CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
  arch gate, declarations and op registrations. The CUDA build never
  defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
  fail with "The model ... does not exist. Valid aliases: ...".

Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-19 21:06:19 +07:00
MOVZXandHinata 6a95f62e2f [Bugfix] Load checkpoint KV-cache scales in the Qwen3.5 family
Checkpoints that ship calibrated k_scale/v_scale tensors (e.g.
unsloth/Qwen3.8-27B-NVFP4, which carries one pair per full-attention
layer) name them ...self_attn.{k,v}_scale, but the runtime parameters
live on the inner attention layer (...self_attn.attn.{k,v}_scale).
Qwen3_5Model.load_weights hands the iterable straight to
AutoWeightsLoader, which performs no scale-name remapping, so the
scales are silently dropped and --kv-cache-dtype fp8 runs at scale 1.0
with only a warning_once that fires even on success (the MTP draft
layer legitimately has no scales).

Route scale-suffixed names through maybe_remap_kv_scale_name first,
mirroring gemma4/gpt_oss. No behavior change for checkpoints without
scales.

Measured on RTX 5090 / Qwen3.8-27B NVFP4 / fp8 KV / MTP k=3 (GSM8K
greedy, n=300): 92.3% at scale 1.0 vs 93.7% with scales loaded
(bf16-KV control 94.5%).

Fixes #54623

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-18 17:19:34 +00:00
MOVZXandHinata 0076adc78b [Qwen3.5] Project the refusal direction out of the residual stream at runtime
orcarouter/Qwen3.8-27B-Uncensored removes refusal by editing weights. The
Swift release we serve is AutoRound W8A16, so re-editing weights means
re-quantizing. Project one unit direction r out of every residual-stream
writer output instead: x <- x - r (r . x), which equals the weight edit
W' = W - r(r^T W) on every writer feeding that stream (Arditi et al. 2024).

Sites: attention output for both the full-attention and GDN branches, the
MLP output, and token embeddings, in Qwen3NextDecoderLayer.forward and
Qwen3NextModel.embed_input_ids, which Qwen3_5DecoderLayer and Qwen3_5Model
inherit but whose base constructors they skip, so both register there too.
The MTP drafter reuses Qwen3_5DecoderLayer and calls its own
embed_input_ids, so the draft path is covered at the same width, and
_maybe_share_embeddings at PP=1 hands the draft the target embedding
module. Sequence parallelism shards dim 0 only, so the last dim is
hidden_size at every site.

VLLM_ABLITERATION names a JSON file with key refusal_direction, a
safetensors tensor, or a bare name resolved under
/opt/qwen38/abliteration then /data/abliteration. Empty or 0 is off.
register() always creates the buffer _abliteration_r, zeros when off, so
one compiled graph serves ON and OFF and toggling the switch
(declared in envs.py, hence in compile_factors) invalidates the compile
cache. project_() is an in-place rank-1 update with no full-size
temporary, and skips tensors whose width is not hidden_size.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-15 21:40:16 +07:00
MOVZX 76df77c04e Merge branch 'main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-15 21:16:39 +07:00
SeongJun Leeandlesj0610 a98247ab4d [Model] Drop the 4x M-RoPE cache headroom for Qwen3-VL, Qwen3.5 and Qwen3.8 (#59853)
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
Signed-off-by: lesj0610 <lesj0610@gmail.com>
Co-authored-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-10-10 06:51:52 +00:00
MOVZX ac305c44cb Merge branch 'main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 17:41:13 +00:00
Harry MellorandClaude Opus 5.5 2c99ee9333 [Chore] Remove vendored code made redundant by Transformers 5.16.1 (#60623)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:29:03 +00:00
MOVZX 2877121b2d Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-03 21:59:03 +07:00
cb6458c5ce [Core] Remove deprecated mamba_cache_mode "all" (#58997)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Thomas Parnell <tpa@zurich.ibm.com>
2026-10-01 02:59:37 -07:00
MOVZX a63dca4afb Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-29 12:54:18 +07:00
cjackalandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 491f44adfa [Model] Extend device-side mm normalization to Qwen3VL/Qwen3.5/Qwen4Next (#51289)
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-29 11:44:54 +08:00
MOVZXandHinata 758cc4f9e1 [Qwen3.5] Pass quant_config and prefix to embed_tokens in Qwen3_5Model
Apply the qwen3_5_quantized_embedding patch: construct the embedding
through the model's quant config so quantized (e.g. fp8/int8)
embeddings are supported, and give it a stable prefix for weight
loading.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 09:53:21 +07:00
Ashraf Bhuiyan 5113cf9d16 [Mypy] Fix mypy typing for Qwen and Qianfan models (#58046)
Signed-off-by: Ashraf Bhuiyan <mbhuiyan@redhat.com>
2026-09-26 14:31:51 +00:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
Giancarlo Delfin d154d90d6c [Model Runner V2] batch-sharded sample (#50465)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
2026-08-24 15:49:57 -07:00
Harry Mellor 1fe3a1571a Reduce AutoWeightsLoader kwargs (#53106)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-08-20 15:40:11 +00:00
5a4c8d9924 [Bugfix][LoRA] Add embedding_modules for Qwen3.5 CausalLM (#48850)
Signed-off-by: m0_68169237 <m0_68169237@noreply.gitcode.com>
Co-authored-by: m0_68169237 <m0_68169237@noreply.gitcode.com>
Co-authored-by: Cursor Composer <composer@cursor.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
2026-08-19 04:55:20 +00:00
fxmarty-amd 88b2bff2c6 [MOE] Standardize and abstract fused shared expert optimization selection (#51695)
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
2026-08-18 12:56:56 -04:00
Harry MellorandClaude Opus 5 80d6d557f3 Standardise weight tying on ParallelLMHead.tie_weights (#52147)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-13 15:10:09 +00:00
haic0 f2bfad9167 [Model] Enable Qwen3.8 for AMD Rocm (#50068)
Signed-off-by: haic0 <haic0@users.noreply.github.com>
Signed-off-by: haic0 <haichzha@amd.com>
2026-08-07 21:19:07 +08:00
zofiaandopencode febea17f6a [Model] Fix weight prefix mapping for native Qwen3.5 text-only checkp… (#50355)
Signed-off-by: Zhu, Zufang <zufang.zhu@intel.com>
Co-authored-by: opencode <opencode@anomaly.co>
2026-08-06 07:40:44 +00:00
Rafli Surya Wijayaandloulanyue 0601850791 [Bugfix][Models] Accept Qwen3_5MoeTextConfig in Qwen3_5MoeProcessingInfo for transformers 5.x compatibility (#50704)
Signed-off-by: loulanyue <loulanyue@users.noreply.github.com>
Co-authored-by: loulanyue <loulanyue@users.noreply.github.com>
2026-08-02 05:50:40 +00:00
Perkz ZhengandPerkz Zheng ad5d29db70 [Model] Support Qwen3.5 text-only dense and MoE models (#50210)
Signed-off-by: Perkz Zheng <PerkzZheng@users.noreply.github.com>
Co-authored-by: Perkz Zheng <PerkzZheng@users.noreply.github.com>
2026-07-29 08:21:57 +00:00
Rui "Garry" Gao dbccc5ae32 [Model] Enable EVS for Qwen3.5 (#48912)
Signed-off-by: Rui "Garry" Gao <garrygaogg@gmail.com>
2026-07-27 13:42:35 +00:00
Harry Mellor c233d90aa8 Remove even more unnecessary load_weights methods (#48496)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-07-18 08:40:27 +00:00
Canlin Guoandyewentao256 a02984ed47 [Perf][Qwen] Replace MOE all-reduce with reduce-scatter (#47006)
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: yewentao256 <zhyanwentao@126.com>
2026-07-12 06:14:49 +00:00
Harry Mellor 1ab9522935 Remove more unnecessary load_weights methods (#47058)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-06-30 15:22:16 +01:00
Harry Mellor d8eb734d94 Fix Transformers backend FP8 MoE and remove some boilerplate (#46820)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-06-27 00:16:05 +01:00
Nico Holmberg 80e511772f [ROCm][Bugfix][Perf] enable shared expert fusion for Qwen3.5 (#44434)
Signed-off-by: Nico Holmberg <nico.holmberg@amd.com>
2026-06-23 23:19:51 +00:00
Jiangyun Zhu ca4cfd8731 [Bugfix] fix qwen3.5 ep weight loading (#45002)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
2026-06-09 13:55:30 -07:00
Wei-Ming Chen 6f5b533241 Add LM head quantization support for ModelOpt (#42124)
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
2026-05-26 09:21:05 -07:00
wangxiyuan 7e1b45a092 [Attention] Mamba attention module refactor (#41126)
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
2026-05-22 17:13:12 +08:00
Harsh ShahandHarsh Shah ca60a4e84f [Fix] Weight loading for qwen3_5 using runai_streamer (#42521)
Signed-off-by: Harsh Shah <iharsh@google.com>
Co-authored-by: Harsh Shah <iharsh@google.com>
2026-05-14 10:36:20 +08:00
Shanshan Shen 92def124bc [MM][Perf][CG] Support ViT full CUDA graph for Qwen3.5 (#42151)
Signed-off-by: shen-shanshan <467638484@qq.com>
2026-05-13 16:00:32 +08:00
Isotr0py 48698b1b9b [Bugfix] Fuse Qwen3.5 in_qkvz_proj forwarding with LoRA enabled (#37912)
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
2026-05-10 10:59:02 +00:00
ℍ𝕠𝕝𝕝𝕠𝕨 𝕄𝕒𝕟 908a713488 [Bugfix] LoRA: extend expert base_layer loading to Qwen3.5 and Step3.x (#37114)
Signed-off-by: Hollow Man <hollowman@opensuse.org>
2026-04-21 14:17:03 +00:00
elenalil-aws 81994e1d0e [Bugfix][LoRA] Fix missing in_proj_z in Qwen3_5ForConditionalGenerati… (#38927)
Signed-off-by: elenalil-aws <elenalil@amazon.com>
2026-04-03 23:30:09 +00:00
Xiaoshuang Wang a8eab8f30d [Model] Extract GatedDeltaNetAttention into shared layer for Qwen3Next and Qwen3.5 (#37975)
Signed-off-by: wxsIcey <1790571317@qq.com>
Signed-off-by: Icey <1790571317@qq.com>
2026-03-27 14:13:21 +08:00
Xin Yang 9704a5c310 Disable dual stream execution of input projection for Qwen3 (#38152)
Signed-off-by: Xin Yang <xyangx@amazon.com>
2026-03-26 01:20:39 +00:00
Jee Jee Li 8fbe3f303f [Bugfix][LoRA] Fix Qwen35 LoRA (#36976)
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
2026-03-20 11:09:32 +08:00
4120a05ff1 Fix AttributeError in Qwen3.5 GDN layers with quantized models (#37448)
Signed-off-by: Jim Smith <jim@joshua8.ai>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Xin Yang <105740670+xyang16@users.noreply.github.com>
2026-03-19 19:21:14 -04:00
Xin Yang f1740006e4 [Perf] Enable dual stream execution of input projection for Qwen3 (#36795)
Signed-off-by: Xin Yang <xyangx@amazon.com>
2026-03-18 11:13:27 +08:00
Rahul Tuli 9d07a3d6e4 Add: Eagle3 support for Qwen3.5 (#36658)
Signed-off-by: Rahul-Tuli <rtuli@redhat.com>
2026-03-11 03:07:42 -07:00
Ajay Anubolu 4e95ec111c [Bugfix] Fix Qwen3-Next in_proj_ba weight sharding with TP > 1 (#36242)
Signed-off-by: AjAnubolu <anuboluajay@gmail.com>
2026-03-09 19:16:26 -07:00
Alex Brooks bd2659a566 Increase Flexibility for OOV Multimodal Token Handling (#34858)
Signed-off-by: Alex Brooks <albrooks@redhat.com>
2026-03-08 20:30:49 -07:00
William Zhang 70c73df69e [Bugfix] Fix EVS implementation for Qwen3 VL (#33607)
Signed-off-by: 2ez4bz <133824995+2ez4bz@users.noreply.github.com>
2026-03-04 02:18:11 +00:00
lailoo afd089f231 [Bugfix][Model] Fix Qwen3.5/Qwen3Next ignoring --dtype flag on older GPUs (#35617) 2026-03-01 03:27:37 +00:00
Linda 6fff24f30f [Bugfix] Qwen3.5 kv-scale weight remapping (#34719)
Signed-off-by: Linda-Stadter <57756729+Linda-Stadter@users.noreply.github.com>
2026-02-19 04:13:37 -08:00
Isotr0pyandJJJYmmm c0bd8b13da [Bugfix] Redo Qwen3.5/Qwen3-Next GDN projector fusion (#34697)
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: JJJYmmm <92386084+JJJYmmm@users.noreply.github.com>
2026-02-18 09:46:53 -08:00
Michael Goin 909b147197 [Bugfix] Fix prefix creation for Qwen3.5 (#34723)
Signed-off-by: mgoin <mgoin64@gmail.com>
2026-02-17 23:39:15 -08:00