27 Commits
Author SHA1 Message Date
MOVZXandHinata 0076adc78b [Qwen3.5] Project the refusal direction out of the residual stream at runtime
orcarouter/Qwen3.8-27B-Uncensored removes refusal by editing weights. The
Swift release we serve is AutoRound W8A16, so re-editing weights means
re-quantizing. Project one unit direction r out of every residual-stream
writer output instead: x <- x - r (r . x), which equals the weight edit
W' = W - r(r^T W) on every writer feeding that stream (Arditi et al. 2024).

Sites: attention output for both the full-attention and GDN branches, the
MLP output, and token embeddings, in Qwen3NextDecoderLayer.forward and
Qwen3NextModel.embed_input_ids, which Qwen3_5DecoderLayer and Qwen3_5Model
inherit but whose base constructors they skip, so both register there too.
The MTP drafter reuses Qwen3_5DecoderLayer and calls its own
embed_input_ids, so the draft path is covered at the same width, and
_maybe_share_embeddings at PP=1 hands the draft the target embedding
module. Sequence parallelism shards dim 0 only, so the last dim is
hidden_size at every site.

VLLM_ABLITERATION names a JSON file with key refusal_direction, a
safetensors tensor, or a bare name resolved under
/opt/qwen38/abliteration then /data/abliteration. Empty or 0 is off.
register() always creates the buffer _abliteration_r, zeros when off, so
one compiled graph serves ON and OFF and toggling the switch
(declared in envs.py, hence in compile_factors) invalidates the compile
cache. project_() is an in-place rank-1 update with no full-size
temporary, and skips tensors whose width is not hidden_size.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-15 21:40:16 +07:00
MOVZX ac305c44cb Merge branch 'main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 17:41:13 +00:00
Harry MellorandClaude Opus 5.5 2c99ee9333 [Chore] Remove vendored code made redundant by Transformers 5.16.1 (#60623)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 09:29:03 +00:00
e572e02d21 [UX] Skip checkpoint shards an MTP head does not load (#60152)
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 15:16:19 -04:00
MOVZX 2877121b2d Merge remote-tracking branch 'upstream/main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-03 21:59:03 +07:00
cb6458c5ce [Core] Remove deprecated mamba_cache_mode "all" (#58997)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Thomas Parnell <tpa@zurich.ibm.com>
2026-10-01 02:59:37 -07:00
MOVZXandHinata 5af48b498c [Qwen3.5] Build MTP head unquantized when checkpoint ships it dense (PR #58813)
Fixes #58807. A compressed-tensors pack-quantized checkpoint that ships
its MTP head dense (plain mtp.*.weight names in a separate bf16 shard)
crashes at weight load: Qwen3_5MultiTokenPredictor builds its Linears
from the target quant config, so the loader expects packed names
(weight_packed) the checkpoint never had, and the weight-map walk dies
with "no module or parameter named fc.weight".

Detect the dense MTP head from the checkpoint itself: read the
safetensors index via the same repo-file channel used for recipe.yaml
(local cache first, Hub download as fallback). If any mtp.* tensor
carries a quantization artifact suffix the head is properly quantized
and nothing changes; if every mtp.* entry is a plain .weight the whole
predictor builds unquantized (fc and layers), mirroring the existing
modelopt_fp4 and GPTQ dynamic carve-outs. Single-file checkpoints have
no index and keep current behavior; NVFP4 checkpoints are unaffected
(their mtp.layers.* carry artifacts).

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-09-27 09:54:24 +07:00
Ashraf Bhuiyan 5113cf9d16 [Mypy] Fix mypy typing for Qwen and Qianfan models (#58046)
Signed-off-by: Ashraf Bhuiyan <mbhuiyan@redhat.com>
2026-09-26 14:31:51 +00:00
Chris EastwoodNick Hillmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
6fe67cbbf3 [Spec][V2] Support MTP speculative decoding under pipeline parallelism (#46994)
Signed-off-by: Chris Eastwood <chris.eastwood@pwn4g3.dev>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-11 17:55:21 +00:00
fxmarty-amd 88b2bff2c6 [MOE] Standardize and abstract fused shared expert optimization selection (#51695)
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
2026-08-18 12:56:56 -04:00
c05d75aaa9 [Bugfix][Refactor] Keep Qwen3Next layer boundaries sequence parallel (#50685)
Signed-off-by: kunzh <RimeK20@outlook.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
2026-08-14 11:58:32 +08:00
Harry MellorandClaude Opus 5 80d6d557f3 Standardise weight tying on ParallelLMHead.tie_weights (#52147)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-13 15:10:09 +00:00
c8db00b16c Fix GPTQ quantized Qwen3.5 MTP weight loading with spec decode (#48816)
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
Co-authored-by: noobHappylife <64898326+noobHappylife@users.noreply.github.com>
2026-07-23 06:59:48 -07:00
Harry Mellor c233d90aa8 Remove even more unnecessary load_weights methods (#48496)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-07-18 08:40:27 +00:00
Canlin Guo 36484e464a [BugFix] Restore full tokens for Qwen MTP When MoE SP (#48429)
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
2026-07-13 13:29:41 +08:00
Harry Mellor 1ab9522935 Remove more unnecessary load_weights methods (#47058)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-06-30 15:22:16 +01:00
yiheng bd2d83ff31 [SpecDecode] Reduce TP communication for large-vocab draft models speculative decoding (#39419)
Signed-off-by: EanWang211123 <wangyiheng@sangfor.com.cn>
2026-06-10 07:59:24 +00:00
Jiangyun Zhu ca4cfd8731 [Bugfix] fix qwen3.5 ep weight loading (#45002)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
2026-06-09 13:55:30 -07:00
Wei-Ming Chen 6f5b533241 Add LM head quantization support for ModelOpt (#42124)
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
2026-05-26 09:21:05 -07:00
weizhoublue a94189295b Fix Weight loading for Qwen3.5-MTP and Qwen3-VL using runai_streamer (#42716)
Signed-off-by: weizhoublue <weizhou.lan@daocloud.io>
2026-05-16 17:54:27 -07:00
bnellnm 1c2c1eb8b9 [MoE Refactor] Rename FusedMoE.make_expert_params_mapping to fused_moe_make_expert_params_mapping (#40671)
Signed-off-by: Bill Nell <bnell@redhat.com>
2026-04-23 11:22:34 -04:00
ℍ𝕠𝕝𝕝𝕠𝕨 𝕄𝕒𝕟 908a713488 [Bugfix] LoRA: extend expert base_layer loading to Qwen3.5 and Step3.x (#37114)
Signed-off-by: Hollow Man <hollowman@opensuse.org>
2026-04-21 14:17:03 +00:00
Vadim Gimpelson 771913e4a0 [Bugfix] Fix NVFP4+MTP crash: force unquantized mtp.fc for Qwen3.5 (#38832)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
2026-04-03 04:45:57 +04:00
Alex Brooks bd2659a566 Increase Flexibility for OOV Multimodal Token Handling (#34858)
Signed-off-by: Alex Brooks <albrooks@redhat.com>
2026-03-08 20:30:49 -07:00
cwazai 63d7972f13 Fix Qwen3_5MTP packed_modules_mapping for gate_up_proj (#35581) 2026-02-28 14:50:55 +00:00
Roger Wang 5885e330ef [Misc] Port Qwen3.5 Configs (#34512)
Signed-off-by: Roger Wang <hey@rogerw.io>
2026-02-13 05:24:25 -08:00
9562912cea [MODEL] Adding Support for Qwen3.5 Models (#34110)
Signed-off-by: JJJYmmm <1650675829@qq.com>
Signed-off-by: JJJYmmm <92386084+JJJYmmm@users.noreply.github.com>
Signed-off-by: Roger Wang <hey@rogerw.io>
Co-authored-by: wulipc <wulipc@users.noreply.github.com>
Co-authored-by: ywang96 <ywang96@users.noreply.github.com>
Co-authored-by: Isotr0py <Isotr0py@users.noreply.github.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
2026-02-09 21:12:58 +08:00