MOVZX and Hinata
5aa2ac64f8
[Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
...
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.
- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
render routes, and vllm/config/speech_to_text.py. GenerationTask is
generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
Multi, Offloading and Example connectors stay; production uses the
CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
arch gate, declarations and op registrations. The CUDA build never
defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
fail with "The model ... does not exist. Valid aliases: ...".
Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 21:06:19 +07:00
MOVZX and Hinata
0076adc78b
[Qwen3.5] Project the refusal direction out of the residual stream at runtime
...
orcarouter/Qwen3.8-27B-Uncensored removes refusal by editing weights. The
Swift release we serve is AutoRound W8A16, so re-editing weights means
re-quantizing. Project one unit direction r out of every residual-stream
writer output instead: x <- x - r (r . x), which equals the weight edit
W' = W - r(r^T W) on every writer feeding that stream (Arditi et al. 2024).
Sites: attention output for both the full-attention and GDN branches, the
MLP output, and token embeddings, in Qwen3NextDecoderLayer.forward and
Qwen3NextModel.embed_input_ids, which Qwen3_5DecoderLayer and Qwen3_5Model
inherit but whose base constructors they skip, so both register there too.
The MTP drafter reuses Qwen3_5DecoderLayer and calls its own
embed_input_ids, so the draft path is covered at the same width, and
_maybe_share_embeddings at PP=1 hands the draft the target embedding
module. Sequence parallelism shards dim 0 only, so the last dim is
hidden_size at every site.
VLLM_ABLITERATION names a JSON file with key refusal_direction, a
safetensors tensor, or a bare name resolved under
/opt/qwen38/abliteration then /data/abliteration. Empty or 0 is off.
register() always creates the buffer _abliteration_r, zeros when off, so
one compiled graph serves ON and OFF and toggling the switch
(declared in envs.py, hence in compile_factors) invalidates the compile
cache. project_() is an in-place rank-1 update with no full-size
temporary, and skips tensors whose width is not hidden_size.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:40:16 +07:00
Harry Huang and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
c75cb63cd2
[Qwen4Exp] support Sequence Parallelism for Qwen3.8-Flash-Next ( #56322 )
...
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-12 20:56:44 +08:00
SeongJun Lee and lesj0610
a98247ab4d
[Model] Drop the 4x M-RoPE cache headroom for Qwen3-VL, Qwen3.5 and Qwen3.8 ( #59853 )
...
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com >
Signed-off-by: lesj0610 <lesj0610@gmail.com >
Co-authored-by: lesj0610 <lesj0610@users.noreply.github.com >
2026-10-10 06:51:52 +00:00
Harry Mellor and Claude Opus 5.5
2c99ee9333
[Chore] Remove vendored code made redundant by Transformers 5.16.1 ( #60623 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 09:29:03 +00:00
9f3d70e081
[Quantization] Enable shared expert fusion compatibility with online shared_expert quantization (showcase: along Quark MXFP4 routed experts) ( #55686 )
...
Signed-off-by: Felix Marty <Felix.Marty@amd.com >
Signed-off-by: fxmarty-amd <felmarty@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-10-08 12:04:46 -04:00
cb6458c5ce
[Core] Remove deprecated mamba_cache_mode "all" ( #58997 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Thomas Parnell <tpa@zurich.ibm.com >
2026-10-01 02:59:37 -07:00
Ashraf Bhuiyan
5113cf9d16
[Mypy] Fix mypy typing for Qwen and Qianfan models ( #58046 )
...
Signed-off-by: Ashraf Bhuiyan <mbhuiyan@redhat.com >
2026-09-26 14:31:51 +00:00
Jee Jee Li
de6674b002
[MoE] Use GateLinear for all MoE models ( #58234 )
...
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
2026-09-23 05:40:13 +00:00
Marceli Fylcek and Kunshang Ji
ccbaed5b91
[XPU] Route to fused_qk_rmsnorm_rope_gate triton kernel ( #56096 )
...
Signed-off-by: Marceli Fylcek <marceli.fylcek@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-09-16 11:05:54 +08:00
Colin Z
588a813a60
[ROCm] [BugFix] Fix AITER MXFP4 ASM-GEMM crash on unfused shared experts ( #55213 )
...
Signed-off-by: Colin Zeng <Colin.Zeng@amd.com >
2026-09-10 05:23:36 +00:00
+5
e126687a9a
[Model] Support Qwen3.8-Flash-Next ( #53896 )
...
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com >
Signed-off-by: andyluo7 <andy.luo@amd.com >
Signed-off-by: lishunyang12 <lishunyang03@gmail.com >
Signed-off-by: Harry Huang <huanghaoyan.hhy@alibaba-inc.com >
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: KungYork <firelikest@163.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: andyluo7 <43718156+andyluo7@users.noreply.github.com >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: andyluo7 <andy.luo@amd.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: lishunyang12 <lishunyang03@gmail.com >
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-31 05:57:56 +00:00
kiroxu and OpenAI Codex
07ef21bc69
[Kernel][Perf] Enable fused QK-norm + partial MRoPE + gate for Qwen3.6 ( #52676 )
...
Signed-off-by: kiroxu <148877251+BabyDrangoner@users.noreply.github.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-24 18:28:49 -07:00
Harry Mellor
1fe3a1571a
Reduce AutoWeightsLoader kwargs ( #53106 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-08-20 15:40:11 +00:00
fxmarty-amd
88b2bff2c6
[MOE] Standardize and abstract fused shared expert optimization selection ( #51695 )
...
Signed-off-by: Felix Marty <Felix.Marty@amd.com >
2026-08-18 12:56:56 -04:00
c05d75aaa9
[Bugfix][Refactor] Keep Qwen3Next layer boundaries sequence parallel ( #50685 )
...
Signed-off-by: kunzh <RimeK20@outlook.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
2026-08-14 11:58:32 +08:00
Wentao Ye
fd04edef3e
[Refactor] Delete dead code in models ( #51838 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-08-11 15:44:26 -04:00
bnellnm and Harry Mellor
6e311c6e20
[MoE Refactor] Rename FusedMoE to FusedMoEFactory ( #44941 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
Signed-off-by: bnellnm <49004751+bnellnm@users.noreply.github.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-31 11:28:18 +01:00
Harry Mellor
c233d90aa8
Remove even more unnecessary load_weights methods ( #48496 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-18 08:40:27 +00:00
Canlin Guo and yewentao256
a02984ed47
[Perf][Qwen] Replace MOE all-reduce with reduce-scatter ( #47006 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Co-authored-by: yewentao256 <zhyanwentao@126.com >
2026-07-12 06:14:49 +00:00
Jiangyun Zhu
300e33797f
[Perf] fuse more rmsnorm and all-reduce in qwen3.5 ( #46998 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
2026-07-10 15:37:51 +08:00
Harry Mellor
1ab9522935
Remove more unnecessary load_weights methods ( #47058 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-06-30 15:22:16 +01:00
Harry Mellor
d8eb734d94
Fix Transformers backend FP8 MoE and remove some boilerplate ( #46820 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-06-27 00:16:05 +01:00
Nico Holmberg
80e511772f
[ROCm][Bugfix][Perf] enable shared expert fusion for Qwen3.5 ( #44434 )
...
Signed-off-by: Nico Holmberg <nico.holmberg@amd.com >
2026-06-23 23:19:51 +00:00
Athrael Soju and Claude Opus 4.8
3c8e49596c
[Model] ColQwen3.5: fix retrieval correctness (bias + bidirectional) ( #46108 )
...
Signed-off-by: Athrael Soju <athrael.soju@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-06-22 17:25:54 +08:00
Jonas I. Liechti
d6fd7ce8da
[Model][Dflash] Enable Dflash support for Qwen3NextForCausalLM targets ( #45319 )
...
Signed-off-by: Jonas I. Liechti <j-i-l@t4d.ch >
2026-06-12 10:30:09 -07:00
Jiangyun Zhu
7a89b72564
[Perf] fuse qk rmsnorm rope gate for qwen3.5 ( #44176 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
2026-06-09 22:12:17 +08:00
wangxiyuan
7e1b45a092
[Attention] Mamba attention module refactor ( #41126 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2026-05-22 17:13:12 +08:00
2c6b59b807
[ROCm][Perf] Add Fused Shared Expert (FSE) support for Qwen3-Next ( #39280 )
...
Signed-off-by: nholmber <nholmber@users.noreply.github.com >
Signed-off-by: Tres Popp <tres.popp@amd.com >
Signed-off-by: Doug Lehr <douglehr@amd.com >
Co-authored-by: nholmber <nholmber@users.noreply.github.com >
Co-authored-by: Tres <tpopp@users.noreply.github.com >
Co-authored-by: Tres Popp <tres.popp@amd.com >
Co-authored-by: Doug Lehr <douglehr@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
2026-05-08 15:38:00 -04:00
bnellnm
1c2c1eb8b9
[MoE Refactor] Rename FusedMoE.make_expert_params_mapping to fused_moe_make_expert_params_mapping ( #40671 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
2026-04-23 11:22:34 -04:00
bnellnm
5e584ce9ec
[MoE Refactor] Remove SharedFusedMoE class ( #35782 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
2026-04-21 18:12:12 -04:00
bnellnm and Robert Shaw
726efe177b
[MoE Refactor] Move the shared/fused expert output sum into MoERunnerBase ( #35949 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-04-20 12:28:46 -04:00
Kai Song
f3c7941ec8
[Bugfix]Fix EP precision for Qwen3.5, Qwen3-Next ( #39181 )
...
Signed-off-by: Song Kai <songkai05@baidu.com >
2026-04-09 01:47:48 +04:00
Benjamin Chislett
494636b29d
[Feat][Spec Decode] DFlash ( #36847 )
...
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com >
2026-03-30 15:03:15 -04:00
Xiaoshuang Wang
a8eab8f30d
[Model] Extract GatedDeltaNetAttention into shared layer for Qwen3Next and Qwen3.5 ( #37975 )
...
Signed-off-by: wxsIcey <1790571317@qq.com >
Signed-off-by: Icey <1790571317@qq.com >
2026-03-27 14:13:21 +08:00
Xin Yang
9704a5c310
Disable dual stream execution of input projection for Qwen3 ( #38152 )
...
Signed-off-by: Xin Yang <xyangx@amazon.com >
2026-03-26 01:20:39 +00:00
Wentao Ye
c59a132f96
[V0 Deprecation] Refactor kv cache from list to element ( #37487 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-03-23 20:10:11 -07:00
Artem Perevedentsev
a16133a0f1
[Perf] [Bugfix] Fix Triton autotuning in inference for Qwen3.5 ( #37338 )
...
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com >
2026-03-23 00:37:58 -07:00
Hojin Yang and Roger Wang
54ab804e87
[Bugfix] Store Qwen3Next A_log in fp32 ( #37810 )
...
Signed-off-by: effortprogrammer <yhjhoward7@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-03-23 15:36:57 +08:00
Jee Jee Li
8fbe3f303f
[Bugfix][LoRA] Fix Qwen35 LoRA ( #36976 )
...
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com >
2026-03-20 11:09:32 +08:00
4120a05ff1
Fix AttributeError in Qwen3.5 GDN layers with quantized models ( #37448 )
...
Signed-off-by: Jim Smith <jim@joshua8.ai >
Signed-off-by: mgoin <mgoin64@gmail.com >
Signed-off-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Xin Yang <105740670+xyang16@users.noreply.github.com >
2026-03-19 19:21:14 -04:00
Cyrus Leung
657855ab41
[Misc] Cleanup more configs and processors ( #37560 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-03-19 15:45:23 +00:00
Wentao Ye
0d81a1fe61
[V0 Deprecation] Deprecate virtual engine ( #37195 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-03-18 14:30:14 -07:00
JartX
a913b612d8
[Bugfix] Fix ROCm crash in qwen3_next multi-stream events ( #36795 ) ( #37427 )
...
Signed-off-by: JartX <sagformas@epdcenter.es >
2026-03-18 16:06:31 -04:00
Wentao Ye
c373b5c00d
[Log] Reduce duplicate log ( #37313 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-03-18 10:57:44 -04:00
Xin Yang
f1740006e4
[Perf] Enable dual stream execution of input projection for Qwen3 ( #36795 )
...
Signed-off-by: Xin Yang <xyangx@amazon.com >
2026-03-18 11:13:27 +08:00
Jiangyun Zhu and Roger Wang
697e4ff352
[GDN] add a config for gdn kernel selection ( #36647 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-03-16 00:40:17 +08:00
Thomas Parnell
f296a1966d
[Bugfix] Fix FlashInfer GDN warmup ValueError on SM90 GPUs ( #36876 )
2026-03-13 07:09:39 +01:00
caozuoba and Roger Wang
9e19f8338b
[Perf] add packed recurrent fast path for decode ( #36596 )
...
Signed-off-by: hdj <1293066020@qq.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-03-12 04:01:57 -07:00
Xu Jinyang
3e64fe4a18
[Bugfix] Warm up Triton autotuner for GDN layers during V1 profiling ( #36599 )
...
Signed-off-by: AuYang <459461160@qq.com >
2026-03-12 00:51:09 -07:00