MOVZX and Hinata
6cf86cd95f
[Bugfix] Load checkpoint KV-cache scales in the Qwen3.5 family
...
Checkpoints that ship calibrated k_scale/v_scale tensors (e.g.
unsloth/Qwen3.8-27B-NVFP4, which carries one pair per full-attention
layer) name them ...self_attn.{k,v}_scale, but the runtime parameters
live on the inner attention layer (...self_attn.attn.{k,v}_scale).
Qwen3_5Model.load_weights hands the iterable straight to
AutoWeightsLoader, which performs no scale-name remapping, so the
scales are silently dropped and --kv-cache-dtype fp8 runs at scale 1.0
with only a warning_once that fires even on success (the MTP draft
layer legitimately has no scales).
Route scale-suffixed names through maybe_remap_kv_scale_name first,
mirroring gemma4/gpt_oss. No behavior change for checkpoints without
scales.
Measured on RTX 5090 / Qwen3.8-27B NVFP4 / fp8 KV / MTP k=3 (GSM8K
greedy, n=300): 92.3% at scale 1.0 vs 93.7% with scales loaded
(bf16-KV control 94.5%).
Fixes #54623
Co-Authored-By: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-18 17:18:47 +00:00
MOVZX and Hinata
0076adc78b
[Qwen3.5] Project the refusal direction out of the residual stream at runtime
...
orcarouter/Qwen3.8-27B-Uncensored removes refusal by editing weights. The
Swift release we serve is AutoRound W8A16, so re-editing weights means
re-quantizing. Project one unit direction r out of every residual-stream
writer output instead: x <- x - r (r . x), which equals the weight edit
W' = W - r(r^T W) on every writer feeding that stream (Arditi et al. 2024).
Sites: attention output for both the full-attention and GDN branches, the
MLP output, and token embeddings, in Qwen3NextDecoderLayer.forward and
Qwen3NextModel.embed_input_ids, which Qwen3_5DecoderLayer and Qwen3_5Model
inherit but whose base constructors they skip, so both register there too.
The MTP drafter reuses Qwen3_5DecoderLayer and calls its own
embed_input_ids, so the draft path is covered at the same width, and
_maybe_share_embeddings at PP=1 hands the draft the target embedding
module. Sequence parallelism shards dim 0 only, so the last dim is
hidden_size at every site.
VLLM_ABLITERATION names a JSON file with key refusal_direction, a
safetensors tensor, or a bare name resolved under
/opt/qwen38/abliteration then /data/abliteration. Empty or 0 is off.
register() always creates the buffer _abliteration_r, zeros when off, so
one compiled graph serves ON and OFF and toggling the switch
(declared in envs.py, hence in compile_factors) invalidates the compile
cache. project_() is an in-place rank-1 update with no full-size
temporary, and skips tensors whose width is not hidden_size.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:40:16 +07:00
MOVZX
76df77c04e
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:16:39 +07:00
SeongJun Lee and lesj0610
a98247ab4d
[Model] Drop the 4x M-RoPE cache headroom for Qwen3-VL, Qwen3.5 and Qwen3.8 ( #59853 )
...
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com >
Signed-off-by: lesj0610 <lesj0610@gmail.com >
Co-authored-by: lesj0610 <lesj0610@users.noreply.github.com >
2026-10-10 06:51:52 +00:00
MOVZX
ac305c44cb
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-09 17:41:13 +00:00
Harry Mellor and Claude Opus 5.5
2c99ee9333
[Chore] Remove vendored code made redundant by Transformers 5.16.1 ( #60623 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 09:29:03 +00:00
MOVZX
2877121b2d
Merge remote-tracking branch 'upstream/main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-03 21:59:03 +07:00
cb6458c5ce
[Core] Remove deprecated mamba_cache_mode "all" ( #58997 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Thomas Parnell <tpa@zurich.ibm.com >
2026-10-01 02:59:37 -07:00
MOVZX
a63dca4afb
Merge remote-tracking branch 'upstream/main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-09-29 12:54:18 +07:00
cjackal and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
491f44adfa
[Model] Extend device-side mm normalization to Qwen3VL/Qwen3.5/Qwen4Next ( #51289 )
...
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-29 11:44:54 +08:00
MOVZX and Hinata
758cc4f9e1
[Qwen3.5] Pass quant_config and prefix to embed_tokens in Qwen3_5Model
...
Apply the qwen3_5_quantized_embedding patch: construct the embedding
through the model's quant config so quantized (e.g. fp8/int8)
embeddings are supported, and give it a stable prefix for weight
loading.
Co-Authored-By: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-09-27 09:53:21 +07:00
Ashraf Bhuiyan
5113cf9d16
[Mypy] Fix mypy typing for Qwen and Qianfan models ( #58046 )
...
Signed-off-by: Ashraf Bhuiyan <mbhuiyan@redhat.com >
2026-09-26 14:31:51 +00:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
Giancarlo Delfin
d154d90d6c
[Model Runner V2] batch-sharded sample ( #50465 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-08-24 15:49:57 -07:00
Harry Mellor
1fe3a1571a
Reduce AutoWeightsLoader kwargs ( #53106 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-08-20 15:40:11 +00:00
5a4c8d9924
[Bugfix][LoRA] Add embedding_modules for Qwen3.5 CausalLM ( #48850 )
...
Signed-off-by: m0_68169237 <m0_68169237@noreply.gitcode.com >
Co-authored-by: m0_68169237 <m0_68169237@noreply.gitcode.com >
Co-authored-by: Cursor Composer <composer@cursor.com >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
2026-08-19 04:55:20 +00:00
fxmarty-amd
88b2bff2c6
[MOE] Standardize and abstract fused shared expert optimization selection ( #51695 )
...
Signed-off-by: Felix Marty <Felix.Marty@amd.com >
2026-08-18 12:56:56 -04:00
Harry Mellor and Claude Opus 5
80d6d557f3
Standardise weight tying on ParallelLMHead.tie_weights ( #52147 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-13 15:10:09 +00:00
haic0
f2bfad9167
[Model] Enable Qwen3.8 for AMD Rocm ( #50068 )
...
Signed-off-by: haic0 <haic0@users.noreply.github.com >
Signed-off-by: haic0 <haichzha@amd.com >
2026-08-07 21:19:07 +08:00
zofia and opencode
febea17f6a
[Model] Fix weight prefix mapping for native Qwen3.5 text-only checkp… ( #50355 )
...
Signed-off-by: Zhu, Zufang <zufang.zhu@intel.com >
Co-authored-by: opencode <opencode@anomaly.co >
2026-08-06 07:40:44 +00:00
Rafli Surya Wijaya and loulanyue
0601850791
[Bugfix][Models] Accept Qwen3_5MoeTextConfig in Qwen3_5MoeProcessingInfo for transformers 5.x compatibility ( #50704 )
...
Signed-off-by: loulanyue <loulanyue@users.noreply.github.com >
Co-authored-by: loulanyue <loulanyue@users.noreply.github.com >
2026-08-02 05:50:40 +00:00
Perkz Zheng and Perkz Zheng
ad5d29db70
[Model] Support Qwen3.5 text-only dense and MoE models ( #50210 )
...
Signed-off-by: Perkz Zheng <PerkzZheng@users.noreply.github.com >
Co-authored-by: Perkz Zheng <PerkzZheng@users.noreply.github.com >
2026-07-29 08:21:57 +00:00
Rui "Garry" Gao
dbccc5ae32
[Model] Enable EVS for Qwen3.5 ( #48912 )
...
Signed-off-by: Rui "Garry" Gao <garrygaogg@gmail.com >
2026-07-27 13:42:35 +00:00
Harry Mellor
c233d90aa8
Remove even more unnecessary load_weights methods ( #48496 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-18 08:40:27 +00:00
Canlin Guo and yewentao256
a02984ed47
[Perf][Qwen] Replace MOE all-reduce with reduce-scatter ( #47006 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Co-authored-by: yewentao256 <zhyanwentao@126.com >
2026-07-12 06:14:49 +00:00
Harry Mellor
1ab9522935
Remove more unnecessary load_weights methods ( #47058 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-06-30 15:22:16 +01:00
Harry Mellor
d8eb734d94
Fix Transformers backend FP8 MoE and remove some boilerplate ( #46820 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-06-27 00:16:05 +01:00
Nico Holmberg
80e511772f
[ROCm][Bugfix][Perf] enable shared expert fusion for Qwen3.5 ( #44434 )
...
Signed-off-by: Nico Holmberg <nico.holmberg@amd.com >
2026-06-23 23:19:51 +00:00
Jiangyun Zhu
ca4cfd8731
[Bugfix] fix qwen3.5 ep weight loading ( #45002 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
2026-06-09 13:55:30 -07:00
Wei-Ming Chen
6f5b533241
Add LM head quantization support for ModelOpt ( #42124 )
...
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com >
2026-05-26 09:21:05 -07:00
wangxiyuan
7e1b45a092
[Attention] Mamba attention module refactor ( #41126 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2026-05-22 17:13:12 +08:00
Harsh Shah and Harsh Shah
ca60a4e84f
[Fix] Weight loading for qwen3_5 using runai_streamer ( #42521 )
...
Signed-off-by: Harsh Shah <iharsh@google.com >
Co-authored-by: Harsh Shah <iharsh@google.com >
2026-05-14 10:36:20 +08:00
Shanshan Shen
92def124bc
[MM][Perf][CG] Support ViT full CUDA graph for Qwen3.5 ( #42151 )
...
Signed-off-by: shen-shanshan <467638484@qq.com >
2026-05-13 16:00:32 +08:00
Isotr0py
48698b1b9b
[Bugfix] Fuse Qwen3.5 in_qkvz_proj forwarding with LoRA enabled ( #37912 )
...
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
2026-05-10 10:59:02 +00:00
ℍ𝕠𝕝𝕝𝕠𝕨 𝕄𝕒𝕟
908a713488
[Bugfix] LoRA: extend expert base_layer loading to Qwen3.5 and Step3.x ( #37114 )
...
Signed-off-by: Hollow Man <hollowman@opensuse.org >
2026-04-21 14:17:03 +00:00
elenalil-aws
81994e1d0e
[Bugfix][LoRA] Fix missing in_proj_z in Qwen3_5ForConditionalGenerati… ( #38927 )
...
Signed-off-by: elenalil-aws <elenalil@amazon.com >
2026-04-03 23:30:09 +00:00
Xiaoshuang Wang
a8eab8f30d
[Model] Extract GatedDeltaNetAttention into shared layer for Qwen3Next and Qwen3.5 ( #37975 )
...
Signed-off-by: wxsIcey <1790571317@qq.com >
Signed-off-by: Icey <1790571317@qq.com >
2026-03-27 14:13:21 +08:00
Xin Yang
9704a5c310
Disable dual stream execution of input projection for Qwen3 ( #38152 )
...
Signed-off-by: Xin Yang <xyangx@amazon.com >
2026-03-26 01:20:39 +00:00
Jee Jee Li
8fbe3f303f
[Bugfix][LoRA] Fix Qwen35 LoRA ( #36976 )
...
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com >
2026-03-20 11:09:32 +08:00
4120a05ff1
Fix AttributeError in Qwen3.5 GDN layers with quantized models ( #37448 )
...
Signed-off-by: Jim Smith <jim@joshua8.ai >
Signed-off-by: mgoin <mgoin64@gmail.com >
Signed-off-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Xin Yang <105740670+xyang16@users.noreply.github.com >
2026-03-19 19:21:14 -04:00
Xin Yang
f1740006e4
[Perf] Enable dual stream execution of input projection for Qwen3 ( #36795 )
...
Signed-off-by: Xin Yang <xyangx@amazon.com >
2026-03-18 11:13:27 +08:00
Rahul Tuli
9d07a3d6e4
Add: Eagle3 support for Qwen3.5 ( #36658 )
...
Signed-off-by: Rahul-Tuli <rtuli@redhat.com >
2026-03-11 03:07:42 -07:00
Ajay Anubolu
4e95ec111c
[Bugfix] Fix Qwen3-Next in_proj_ba weight sharding with TP > 1 ( #36242 )
...
Signed-off-by: AjAnubolu <anuboluajay@gmail.com >
2026-03-09 19:16:26 -07:00
Alex Brooks
bd2659a566
Increase Flexibility for OOV Multimodal Token Handling ( #34858 )
...
Signed-off-by: Alex Brooks <albrooks@redhat.com >
2026-03-08 20:30:49 -07:00
William Zhang
70c73df69e
[Bugfix] Fix EVS implementation for Qwen3 VL ( #33607 )
...
Signed-off-by: 2ez4bz <133824995+2ez4bz@users.noreply.github.com >
2026-03-04 02:18:11 +00:00
lailoo
afd089f231
[Bugfix][Model] Fix Qwen3.5/Qwen3Next ignoring --dtype flag on older GPUs ( #35617 )
2026-03-01 03:27:37 +00:00
Linda
6fff24f30f
[Bugfix] Qwen3.5 kv-scale weight remapping ( #34719 )
...
Signed-off-by: Linda-Stadter <57756729+Linda-Stadter@users.noreply.github.com >
2026-02-19 04:13:37 -08:00
Isotr0py and JJJYmmm
c0bd8b13da
[Bugfix] Redo Qwen3.5/Qwen3-Next GDN projector fusion ( #34697 )
...
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Co-authored-by: JJJYmmm <92386084+JJJYmmm@users.noreply.github.com >
2026-02-18 09:46:53 -08:00
Michael Goin
909b147197
[Bugfix] Fix prefix creation for Qwen3.5 ( #34723 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
2026-02-17 23:39:15 -08:00
Jiangyun Zhu
1d65283e95
Revert "[Models] Fuse Qwen3.5 GDN's qkvz_proj and ba_proj" ( #34683 )
2026-02-17 01:29:27 -08:00