MOVZX
be95edb7f6
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-16 17:35:42 +07:00
MOVZX and Hinata
8dc54f35dd
[Offloader] Stream every multimodal tower parameter to host RAM
...
Qwen3_VisionTransformer derived its compute device from
patch_embed.proj.weight and fed pos_embed.weight straight into the
position-embedding interpolation kernel, so the last two anchor weights
had to stay in VRAM for a tower to be streamable at all.
Record the device the tower was built on before the offloader moves
anything, and fall back to it when no weight is on the accelerator.
Move the interpolation table to that device per encode, which costs one
8.4 MiB host-to-device copy for a 456 MiB tower. Towers now leave no
parameters in VRAM between encodes; only small buffers such as the fused
input norm and rotary inv_freq remain.
Also let a tower restrict streaming through tower_offload_children, and
warn when a tower resolves its device property to another device type,
so a tower that reads an anchor weight reports the cause instead of a
Triton pointer error.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-16 16:07:53 +07:00
Andreas Karatzas and Claude
29345f42a6
[ROCm][Qwen4Exp] Support HC and MoE sequence parallelism on ROCm ( #61471 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-15 20:11:17 -05:00
MOVZX and Hinata
5d6754ff6b
fix: keep the refusal-direction projection dynamo-safe
...
With VLLM_ABLITERATION set, boot crashed: dynamo traces
abliteration.register from Qwen3NextGatedDeltaNet.__init__, and
buf.copy_(r) fails on the meta device during the compile-time model
construction pass. Disable tracing for register and guard the copy so
an unmaterializable buffer turns projection off with a warning.
Log activation at registration instead of inside project_: project_
runs in the compiled region, and a logger call there aborts dynamo.
The width guard uses the same escape hatch.
Also project the embedding that Qwen3NextMultiTokenPredictor builds
for itself, which the first pass missed, and add tests for the
2560-value Flash-Next direction against a 5120 model and for a
half-width activation.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 22:56:06 +07:00
MOVZX and Hinata
0076adc78b
[Qwen3.5] Project the refusal direction out of the residual stream at runtime
...
orcarouter/Qwen3.8-27B-Uncensored removes refusal by editing weights. The
Swift release we serve is AutoRound W8A16, so re-editing weights means
re-quantizing. Project one unit direction r out of every residual-stream
writer output instead: x <- x - r (r . x), which equals the weight edit
W' = W - r(r^T W) on every writer feeding that stream (Arditi et al. 2024).
Sites: attention output for both the full-attention and GDN branches, the
MLP output, and token embeddings, in Qwen3NextDecoderLayer.forward and
Qwen3NextModel.embed_input_ids, which Qwen3_5DecoderLayer and Qwen3_5Model
inherit but whose base constructors they skip, so both register there too.
The MTP drafter reuses Qwen3_5DecoderLayer and calls its own
embed_input_ids, so the draft path is covered at the same width, and
_maybe_share_embeddings at PP=1 hands the draft the target embedding
module. Sequence parallelism shards dim 0 only, so the last dim is
hidden_size at every site.
VLLM_ABLITERATION names a JSON file with key refusal_direction, a
safetensors tensor, or a bare name resolved under
/opt/qwen38/abliteration then /data/abliteration. Empty or 0 is off.
register() always creates the buffer _abliteration_r, zeros when off, so
one compiled graph serves ON and OFF and toggling the switch
(declared in envs.py, hence in compile_factors) invalidates the compile
cache. project_() is an in-place rank-1 update with no full-size
temporary, and skips tensors whose width is not hidden_size.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:40:16 +07:00
MOVZX
76df77c04e
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:16:39 +07:00
littlecircle0730 and Shanshan Shen
71536bd5f8
[MM][CG] Support ViT full CUDA graph for DeepSeek-VL2 ( #46005 )
...
Signed-off-by: littlecircle0730 <littlecircle0730@gmail.com >
Signed-off-by: littlecircle0730 <43994952+littlecircle0730@users.noreply.github.com >
Co-authored-by: Shanshan Shen <467638484@qq.com >
2026-10-15 21:38:52 +08:00
Xunzhuo
766c837bd5
[Model] Support every Decision 3.0 size (d3-flash, d3-mini, d3-nano, d3-lite, d3-edge) ( #61778 )
...
Signed-off-by: Xunzhuo Liu <xunzhuo.liu@amd.com >
2026-10-15 13:29:24 +00:00
Timofey and Codex
716c5225ae
[Model] Add AliceAI support ( #57963 )
...
Signed-off-by: timka <timka.byzov@yandex.ru >
Co-authored-by: Codex <noreply@openai.com >
2026-10-15 10:15:03 +00:00
Vivaan Sharma and Kevin H. Luu
96b5b0ee4b
[Model] Enable torch.compile and CUDA graphs for EmbeddingGemma2 ( #61572 )
...
Signed-off-by: Vivaan Sharma <99265792+Vivaansharma2003@users.noreply.github.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 12:32:39 +08:00
Haiyu Zuo and Kevin H. Luu
ff5a91a456
[Feature][Model Loader] Apply model-owned MTP shard selection to RunAI ModelStreamer ( #60759 )
...
Signed-off-by: 浩韵 <zuohaiyu.zhy@alibaba-inc.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 12:23:11 +08:00
Woosuk Kwon and Claude Opus 5.5
8530aed2b7
[Spec Decode] Remove Medusa and MLP speculator support ( #61701 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 17:29:23 -07:00
d1d8e6c096
[Perf][Nemotron-Parse] Do image rescale/normalize on the device ( #60500 )
...
Signed-off-by: DiegoCao <DiegoCao@users.noreply.github.com >
Co-authored-by: DiegoCao <DiegoCao@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 18:33:22 +00:00
Taimys and OpenAI Codex
2c74fd1873
[Model Runner V2] Enable batch-sharded sampling for GPT-OSS ( #61257 )
...
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-10-14 10:11:11 -07:00
Xunzhuo
89cab55086
[Model] Support Decision 3.0 decision models (vllm-sr/d3) ( #61565 )
...
Signed-off-by: Xunzhuo Liu <xunzhuo.liu@amd.com >
2026-10-14 17:10:55 +00:00
zexplorerhj and Codex
3895a8babe
[Bugfix] Fix TP loading of small block-FP8 MLA gates in BailingMoeV3 ( #61642 )
...
Signed-off-by: zexplorerhj <zhjoneson@163.com >
Co-authored-by: Codex <noreply@openai.com >
2026-10-14 10:09:23 -07:00
Nan Zhang and Robert Shaw
9e2b579aab
[Model] Support EAGLE3/DFlash aux hidden states for GraniteMoe and GraniteMoeShared ( #61110 )
...
Signed-off-by: ZN1010 <neilzhang1010@126.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-10-14 12:01:14 -04:00
3723a1ea96
[Bugfix][Transformers backend] Fix CUDA graph capture for dynamic-rope models ( #58064 )
...
Signed-off-by: YongfengX <erkang040804@gmail.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-14 05:56:54 -07:00
ssingh231219
3a5a6b11a3
[Perf][Model] Batch PaddleOCR-VL vision encoder and projector ( #61458 )
...
Signed-off-by: ssingh231219 <simran.singh@artificialy.com >
2026-10-14 19:59:44 +08:00
d39c2fb05a
[Bugfix][Gemma4] Quantize audio towers independently ( #59935 )
...
Signed-off-by: sudhanshu shukla <sudhanshu112233shukla@users.noreply.github.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: sudhanshu shukla <sudhanshu112233shukla@users.noreply.github.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-10-14 11:27:44 +00:00
Lazurite
23fbc2ec2f
[Model] Support precomputed image embeddings for LightOnOCR ( #61585 )
...
Signed-off-by: Rapisurazurite <43494437+Rapisurazurite@users.noreply.github.com >
2026-10-14 10:54:09 +00:00
Ronald and aoshen02
19aba3e4d5
[Bugfix][RL] Load attention sinks through weight_loader for layerwise reload ( #57797 )
...
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-14 10:17:58 +00:00
698e79f3bd
[Model] Kolibri 1 via the Transformers modeling backend ( #60827 )
...
Signed-off-by: Luca Motz <luca.motz@icloud.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-10-13 22:17:52 +01:00
Wentao Ye
2f18d13cf0
[GLM5.3 Perf] Enable fp8 kv cache for GLM5.3 Flash by default, 9.8% E2E Throughput Improvement ( #61228 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-10-13 15:29:17 -04:00
c068db0e1d
[Bugfix][Compile] Replace Transformers AOT caching with graph caching ( #58918 )
...
Signed-off-by: fangyao fan <2281693949@qq.com >
Signed-off-by: Taimys <ffy2281693949@gmail.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Taimys <ffy2281693949@gmail.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-10-13 19:17:29 +00:00
6849ca696a
[Quantization] Add ModelOpt IQ2_XS, IQ2_XXS, and Q8_0 support via b12x ( #59604 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Signed-off-by: Luke Alonso <lalonso@gmail.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
Co-authored-by: Luke Alonso <lalonso@gmail.com >
2026-10-13 09:06:33 -07:00
Rishabh Agarwal
f395a6b1ad
[ROCM][Bugfix] Deferring to TRITON_ATTN backend for Prefix lm models instead of ROCM_ATTN and fixed minor bug in Gemma3 Multimodal code ( #46754 )
...
Signed-off-by: GodRishUniverse <rishabh.agarwal@amd.com >
Signed-off-by: Rishabh Agarwal <risagarw@amd.com >
Signed-off-by: GodRishUniverse <risagarw@amd.com >
2026-10-13 10:03:48 -05:00
Nathan Sala and Shaun Kotek - Nvidia
0838cd6593
[Bugfix] Load and apply Nemotron Super final vision LayerNorm ( #61401 )
...
Signed-off-by: Shaun Kotek <skotek@nvidia.com >
Co-authored-by: Shaun Kotek - Nvidia <skotek@nvidia.com >
2026-10-13 12:42:29 +00:00
noone-zhhh and Isotr0py
426f3bd36b
[Bugfix][Model][Rust Frontend] Qwen3_ASR: Handle ragged Qwen audio batches from Rust frontend ( #60979 )
...
Signed-off-by: zh98 <workallday.zhh@gmail.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-10-13 10:49:39 +00:00
Harry Mellor and Claude Opus 5
6f11c220b5
[Transformers backend] Fuse MoE routers that score after the top-k ( #57410 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-10-13 08:41:23 +00:00
2cc2e68d1c
[Core][Mamba] Remove leftovers of mamba_cache_mode="all" ( #60043 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Jiangyun Zhu <zhujiangyun@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-13 00:58:59 -07:00
380638afbb
[Model] Add Clef-Flash support ( #61308 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-10-13 05:46:37 +00:00
Mook and cjackal
5555a2d8f7
[Bugfix][Model] Fix EXAONE-4.5 image inputs: ViT received un-normalised pixels ( #61104 )
...
Signed-off-by: godmook <cmoh4135@naver.com >
Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com >
2026-10-13 13:02:12 +08:00
189bbfb9f4
Add MoNe experts pruning support ( #50415 )
...
Signed-off-by: Ilya Boytsov <ieboytsov@nebius.com >
Signed-off-by: Misha Goin <mgoin64@gmail.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Ilya Boytsov <ieboytsov@nebius.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 23:18:43 +00:00
SeongJun Lee and Ning Xie
56e3b081ae
[Bugfix][weight loader] Stop warning when a parameter only widens the weight ( #60020 )
...
Signed-off-by: lesj0610 <lesj0610@gmail.com >
Co-authored-by: Ning Xie <andy.xning@gmail.com >
2026-10-12 14:33:31 -04:00
Woosuk Kwon and Claude Opus 5.5
e1a5086256
[Attention] Remove cascade attention ( #61126 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 10:41:07 -07:00
Hoai Vo and Claude Opus 5.5
f3cb10b690
[Model][LoRA] Enable LoRA support for BERT and RoBERTa embedding models ( #61033 )
...
Signed-off-by: hoaivoke <vokehoai@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 14:09:53 +00:00
092c35dd15
[Model] Support Cloudflare/clef-omni on /v1/systemone ( #61064 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-10-12 21:27:34 +08:00
Harry Huang and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
c75cb63cd2
[Qwen4Exp] support Sequence Parallelism for Qwen3.8-Flash-Next ( #56322 )
...
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-12 20:56:44 +08:00
MatthewCYM
f7a3a7265f
[Bugfix][Multimodal] Fix Qwen3-ASR audio preprocessing parity ( #57146 )
...
Signed-off-by: MatthewCYM <chenyiming0v0@gmail.com >
2026-10-12 10:42:51 +00:00
Michał Ganczarenko and Claude Opus 5.5
03091e28b8
[Bugfix] Read Ling quantization_config from top-level config for Bailing V3 VL ( #59026 )
...
Signed-off-by: Michal Ganczarenko <michal.ganczarenko@intel.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 09:25:29 +08:00
tc-mb and Cyrus Leung
c85616efc6
[Model] Support MiniCPM-V 4.7 ( #58674 )
...
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-11 10:21:10 +00:00
Bruno Volpato and Codex
61f084b5f3
[Bugfix][Multimodal] Fix per-video cache option reuse ( #55203 )
...
Signed-off-by: bvolpato <brunocvcunha@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-10-11 06:21:55 +00:00
Wentao Ye
3709632ff2
[Refactor] Remove dead code multiple places ( #60120 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-10-10 15:41:10 +00:00
Mook
bf16c9c139
[Bugfix][Model] K-EXAONE: restore loading of pre-2.0 configs and apply RoPE to the MTP layer ( #60187 )
...
Signed-off-by: godmook <cmoh4135@naver.com >
Signed-off-by: Mook <68294499+Godmook@users.noreply.github.com >
2026-10-10 15:05:55 +00:00
SeongJun Lee and lesj0610
a98247ab4d
[Model] Drop the 4x M-RoPE cache headroom for Qwen3-VL, Qwen3.5 and Qwen3.8 ( #59853 )
...
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com >
Signed-off-by: lesj0610 <lesj0610@gmail.com >
Co-authored-by: lesj0610 <lesj0610@users.noreply.github.com >
2026-10-10 06:51:52 +00:00
Andreas Karatzas and Claude
a205fb3177
[Bugfix][Model] Restore AXK1 config defaults dropped with the vendored config ( #60954 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-10 05:54:52 +00:00
devpramod and Claude Opus 5.5
46fb84c2d4
[Model] Support EAGLE3/DFlash aux hidden states for Granite ( #60855 )
...
Signed-off-by: Pai, Pramod <pramod.pai@intel.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-10 13:34:10 +08:00
MOVZX
ac305c44cb
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-09 17:41:13 +00:00
Juan Pérez de Algaba and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
3d52578e13
[Security] Do not retain Phi-4 MM audio attention matrices ( #60614 )
...
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-09 22:23:35 +08:00
MOVZX
