MOVZX and Hinata
0076adc78b
[Qwen3.5] Project the refusal direction out of the residual stream at runtime
...
orcarouter/Qwen3.8-27B-Uncensored removes refusal by editing weights. The
Swift release we serve is AutoRound W8A16, so re-editing weights means
re-quantizing. Project one unit direction r out of every residual-stream
writer output instead: x <- x - r (r . x), which equals the weight edit
W' = W - r(r^T W) on every writer feeding that stream (Arditi et al. 2024).
Sites: attention output for both the full-attention and GDN branches, the
MLP output, and token embeddings, in Qwen3NextDecoderLayer.forward and
Qwen3NextModel.embed_input_ids, which Qwen3_5DecoderLayer and Qwen3_5Model
inherit but whose base constructors they skip, so both register there too.
The MTP drafter reuses Qwen3_5DecoderLayer and calls its own
embed_input_ids, so the draft path is covered at the same width, and
_maybe_share_embeddings at PP=1 hands the draft the target embedding
module. Sequence parallelism shards dim 0 only, so the last dim is
hidden_size at every site.
VLLM_ABLITERATION names a JSON file with key refusal_direction, a
safetensors tensor, or a bare name resolved under
/opt/qwen38/abliteration then /data/abliteration. Empty or 0 is off.
register() always creates the buffer _abliteration_r, zeros when off, so
one compiled graph serves ON and OFF and toggling the switch
(declared in envs.py, hence in compile_factors) invalidates the compile
cache. project_() is an in-place rank-1 update with no full-size
temporary, and skips tensors whose width is not hidden_size.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:40:16 +07:00
MOVZX
76df77c04e
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:16:39 +07:00
Laith Sakka and Nick Hill
9cede34a67
[Bugfix] Create cuBLAS handles before memory profiling to avoid OOM ( #61044 )
...
Signed-off-by: Laith Sakka <lsakka@meta.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-10-15 08:59:22 -04:00
Martin Hickey
252be28c18
[Frontend] Replace return_mm_kwargs with an mm_payload enum on render requests ( #61427 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-10-15 12:41:46 +00:00
Martin Hickey
c6b772cf6b
[Bugfix][Frontend] Emit kv/ec transfer params at the top level of render output ( #61241 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-10-15 12:34:15 +00:00
Micah Williamson
e8e1e96196
[ROCm][DSv4] AITER MegaMoE Fixed-Slot Dispatch For DeepSeek-V4 ( #61262 )
...
Signed-off-by: Micah Williamson <micah.williamson@amd.com >
2026-10-15 19:10:59 +08:00
8b27d27402
[Bugfix][KV Offload] Always attempt NIXL agent removal during peer teardown ( #60279 )
...
Signed-off-by: nej1gotnochill <divyanshlalotra2@gmail.com >
Co-authored-by: Codebuff <noreply@codebuff.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 14:07:23 +03:00
Talor Abramovich
81e42c9987
[Feature][Frontend] Control the log level of request JSON body using env var ( #60630 )
...
Signed-off-by: talora <talora@nvidia.com >
2026-10-15 11:07:19 +00:00
jimmy-adams
365aaa77bd
[Bugfix][DSv4] Remap Quark MTP layer_quant_config onto draft prefixes ( #56054 )
...
Signed-off-by: jimmy-adams <41593649+jimmy-adams@users.noreply.github.com >
2026-10-15 18:51:57 +08:00
e10c10a644
[Core] Add PROMOTION_LATENCY histogram metric for tiering offload ( #53910 )
...
Signed-off-by: Wen Zhou <wenzhou@redhat.com >
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-10-15 13:47:41 +03:00
Pei Li and Kevin H. Luu
6fe7401fac
[Perf] Read VLLM_COMPUTE_NANS_IN_LOGITS once per IterationStats ( #61710 )
...
Signed-off-by: Pei Li <peili@nvidia.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 18:35:54 +08:00
Timofey and Codex
716c5225ae
[Model] Add AliceAI support ( #57963 )
...
Signed-off-by: timka <timka.byzov@yandex.ru >
Co-authored-by: Codex <noreply@openai.com >
2026-10-15 10:15:03 +00:00
7020600b1d
[KV Connector][NIXL] Register packed MLA KV rows as a single region ( #61074 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
2026-10-15 09:18:15 +00:00
Flora Feng and Chauncey
80dd705c1d
[Frontend] Make finish_reason and Responses item status truthful ( #59324 )
...
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-10-15 10:12:02 +01:00
Denis Khachyan and Claude Code
427915bed2
[Build][CPU] Install Run:ai Model Streamer in vllm-openai-cpu image ( #61382 )
...
Signed-off-by: Denis Khachyan <khachyanda@gmail.com >
Co-authored-by: Claude Code <noreply@anthropic.com >
2026-10-15 09:11:51 +00:00
Kevin H. Luu and Claude Opus 5.5
7f730b4cea
[CI] Run 4xH100 steps nightly only ( #61782 )
...
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 01:32:43 -07:00
wang.yuqi
0f0bc4dbb1
[Test] Split the chat_utils tests. ( #61765 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-10-15 08:26:22 +00:00
b965ee564d
[Chore] Use upstream Transformers DeepseekV4Config ( #61392 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 01:08:08 -07:00
Misha Goin and Claude Opus 5.5
1c4c319792
[Frontend] Advertise --enable-x/--disable-x for boolean CLI flags ( #61479 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 01:08:02 -07:00
48f17b77bb
[Bugfix] Reject mismatched float16/bfloat16 KV cache dtype ( #60885 )
...
Signed-off-by: cipheraxat <cipheraxat@users.noreply.github.com >
Co-authored-by: cipheraxat <cipheraxat@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 08:06:59 +00:00
fc8e08a2a6
[Bugfix][KV Offload] Do not let mark_miss() flip a pending or in-flight async lookup probe ( #59096 )
...
Signed-off-by: Sohom Chakraborty <16609933+sohom-cs@users.noreply.github.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 10:57:46 +03:00
Bugen Zhao and Claude Opus 5.5
8fa5ae844a
[Frontend] Let an unset --tool-strict-level fall back to the tool parser's default ( #61328 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 15:48:38 +08:00
91ed2859c7
[CPU][Zen] Update zentorch to 2.14.0.1 & enable float16 on Venice and later ( #61411 )
...
Signed-off-by: Ganesh R <ganesh.r@amd.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-15 00:23:50 -07:00
85c94c1706
[Tests][XPU] Skip modelopt_fp4 online MoE re-quantization test on XPU ( #61723 )
...
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-15 00:23:45 -07:00
Kevin H. Luu and Claude Opus 5.5
8f111b22ee
[CI] Run 4xB200 steps nightly only ( #61541 )
...
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 00:08:48 -07:00
Kevin H. Luu and Claude Opus 5.5
32feecd91f
[CI] Pin DeepSeek-V4.1-Flash-NVFP4 eval to its FP8-Engram revision ( #61763 )
...
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 00:05:18 -07:00
Chaojun Zhang and Kevin H. Luu
ab593678b5
[CI][XPU] Ignore test_reload.py on Intel CI ( #61740 )
...
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 15:03:34 +08:00
a9fe30e9f6
[Bugfix][Multimodal] Recover PyNvVideoCodec decoder slots after failures ( #59597 )
...
Signed-off-by: Daniel Afrimi <dafrimi@nvidia.com >
Signed-off-by: danielafrimi <45691845+danielafrimi@users.noreply.github.com >
Signed-off-by: Daniel Afrimi <dafrimi@oci-hsg-cs-001-login-02.cm.cluster >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Daniel Afrimi <dafrimi@oci-hsg-cs-001-login-02.cm.cluster >
Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com >
2026-10-15 09:17:19 +03:00
liranschour and Kevin H. Luu
c5f8adc0b1
[KV Offload] Expose OffloadingManager.lock()/unlock() and service the tiering control plane off the scheduler thread ( #58168 )
...
Signed-off-by: Liran Schour <lirans@il.ibm.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 08:50:56 +03:00
Harshil and Cyrus Leung
c54dca96bd
[Bugfix][Benchmark] Fix probe latency and duration in bench serve ( #60511 )
...
Signed-off-by: QHarshil <harshil_c@hotmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-15 05:50:30 +00:00
5341b5ba7d
[KV Offload] Pin only each worker's own offload slots ( #59544 )
...
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-10-15 08:49:42 +03:00
6605802904
[Perf][KV Connector][Mooncake] Send the KV layout once per peer ( #59825 )
...
Signed-off-by: staryxchen <staryxchen@tencent.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com >
2026-10-15 05:36:31 +00:00
a400804ce4
[CI] Relax MTP sharded sampling cross-boot noise bound ( #57026 )
...
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com >
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 22:31:45 -07:00
3f1bcfd82b
[Bugfix][KV Offload] Bound unanswered P2P lookup waits ( #56836 )
...
Signed-off-by: jacklin78911-collab <jacklin78911@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 08:23:34 +03:00
81e1c3f183
[XPU][MLA] Integrate XPU MLA support into the existing FLASH_ATTN_MLA backend ( #58297 )
...
Signed-off-by: baodii <di.bao@intel.com >
Signed-off-by: Zhouxiaoya25 <xiaoya.zhou@intel.com >
Signed-off-by: ZhouXiaoya25 <xiaoya.zhou@intel.com >
Co-authored-by: baodii <di.bao@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-10-15 12:47:37 +08:00
Vivaan Sharma and Kevin H. Luu
96b5b0ee4b
[Model] Enable torch.compile and CUDA graphs for EmbeddingGemma2 ( #61572 )
...
Signed-off-by: Vivaan Sharma <99265792+Vivaansharma2003@users.noreply.github.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 12:32:39 +08:00
Haiyu Zuo and Kevin H. Luu
ff5a91a456
[Feature][Model Loader] Apply model-owned MTP shard selection to RunAI ModelStreamer ( #60759 )
...
Signed-off-by: 浩韵 <zuohaiyu.zhy@alibaba-inc.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 12:23:11 +08:00
Chaitanya Sri Krishna Lolla and Cursor
833665cded
[ROCm][DI] Enable MoRI FP4 dispatch for mxfp4 w4a4 moe ( #59596 )
...
Signed-off-by: lcskrishna <lollachaitanya@gmail.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-10-14 23:20:14 -05:00
Tuukka Sarvi and Kevin H. Luu
10d2c6083d
[ROCm][DSv4.1] Enable DSpark adaptive verification ( #58151 )
...
Signed-off-by: Tuukka Sarvi <tuukka.sarvi@amd.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 12:05:43 +08:00
Shanshan Shen
f46021a7fa
[ROCm][Perf] Enable DPA+ETP prefill compute/comm dual-batch-overlap for DeepSeek-V4 ( #57773 )
...
Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com >
Signed-off-by: shen-shanshan <467638484@qq.com >
2026-10-15 11:16:48 +08:00
Feanor and Taneem Ibrahim
5d5c3d7c43
[Bugfix] Preserve padding token types for cross encoders ( #58143 )
...
Signed-off-by: zhaoye <772971548@qq.com >
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
2026-10-15 03:09:24 +00:00
429a3da665
[Bugfix][ROCm][Disagg] Allow MoRIIO HMA per-layer block_len ( #58585 )
...
Signed-off-by: avininjamay8 <Avinash.Paul@amd.com >
Signed-off-by: avininjamay8 <avpaul@amd.com >
Co-authored-by: avininjamay8 <avpaul@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Shanshan Shen <467638484@qq.com >
2026-10-15 09:35:31 +08:00
Kai and Kunshang Ji
de2b869ee9
[Bugfix][Kernel] Register int8 scaled_mm as a custom op ( #57331 )
...
Signed-off-by: Kai <limkc.reply@gmail.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-10-15 09:27:49 +08:00
Aarushi Jain
2abd479aee
[ROCm][CI] Add AMD job for DeepSeek-V4.1 Mono Decode Numerics ( #61720 )
...
Signed-off-by: aarushjain29 <aarushi.jain2@amd.com >
2026-10-15 01:23:25 +00:00
Woosuk Kwon and Claude Opus 5.5
e57bb4ace7
[Core] Remove the legacy Ray compiled-graph executor ( #61712 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 18:13:27 -07:00
bfdbfa1632
[Bugfix][PCP] Align MoE routing padding with gathered tokens ( #60946 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: wangyicong <wangyicong@bytedance.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-10-14 21:04:00 -04:00
Woosuk Kwon and Claude Opus 5.5
8530aed2b7
[Spec Decode] Remove Medusa and MLP speculator support ( #61701 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 17:29:23 -07:00
josh-schell-amd and Claude Opus 5.5
273a04964e
[ROCm] Bump AITER to v0.1.25 and relax the DSv4.1 mono MoE tolerance ( #61697 )
...
Signed-off-by: josh schell <josh.schell@amd.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-14 23:50:24 +00:00
Aarushi Jain and Andreas Karatzas
7840bede98
[ROCm][CI] Add AMD jobs for test groups that only exist in tes… ( #59678 )
...
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-10-14 23:00:44 +00:00
Kevin H. Luu
4b61f300c0
[CI] Fix /ci run reporting a failed merge on fork PRs ( #61684 )
...
Signed-off-by: khluu <khluu000@gmail.com >
2026-10-14 15:26:15 -07:00