MOVZX and Hinata
18c06ed406
[Offloader] Stream multimodal towers from host RAM with bulk-copy
...
Add --offload-towers: place multimodal tower weights in pinned host
memory and bulk-copy each unit to the GPU only during its forward,
keeping towers out of VRAM between encodes. Unlike UVA zero-copy,
whole-tensor H2D transfers avoid re-reading operand tiles over PCIe
inside the GEMM inner loop. Tower streaming is independent of layer
group scheduling, so group_size may be 0 for tower-only offload.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-16 14:32:14 +07:00
MOVZX and Hinata
5d6754ff6b
fix: keep the refusal-direction projection dynamo-safe
...
With VLLM_ABLITERATION set, boot crashed: dynamo traces
abliteration.register from Qwen3NextGatedDeltaNet.__init__, and
buf.copy_(r) fails on the meta device during the compile-time model
construction pass. Disable tracing for register and guard the copy so
an unmaterializable buffer turns projection off with a warning.
Log activation at registration instead of inside project_: project_
runs in the compiled region, and a logger call there aborts dynamo.
The width guard uses the same escape hatch.
Also project the embedding that Qwen3NextMultiTokenPredictor builds
for itself, which the first pass missed, and add tests for the
2560-value Flash-Next direction against a 5120 model and for a
half-width activation.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 22:56:06 +07:00
MOVZX and Hinata
0076adc78b
[Qwen3.5] Project the refusal direction out of the residual stream at runtime
...
orcarouter/Qwen3.8-27B-Uncensored removes refusal by editing weights. The
Swift release we serve is AutoRound W8A16, so re-editing weights means
re-quantizing. Project one unit direction r out of every residual-stream
writer output instead: x <- x - r (r . x), which equals the weight edit
W' = W - r(r^T W) on every writer feeding that stream (Arditi et al. 2024).
Sites: attention output for both the full-attention and GDN branches, the
MLP output, and token embeddings, in Qwen3NextDecoderLayer.forward and
Qwen3NextModel.embed_input_ids, which Qwen3_5DecoderLayer and Qwen3_5Model
inherit but whose base constructors they skip, so both register there too.
The MTP drafter reuses Qwen3_5DecoderLayer and calls its own
embed_input_ids, so the draft path is covered at the same width, and
_maybe_share_embeddings at PP=1 hands the draft the target embedding
module. Sequence parallelism shards dim 0 only, so the last dim is
hidden_size at every site.
VLLM_ABLITERATION names a JSON file with key refusal_direction, a
safetensors tensor, or a bare name resolved under
/opt/qwen38/abliteration then /data/abliteration. Empty or 0 is off.
register() always creates the buffer _abliteration_r, zeros when off, so
one compiled graph serves ON and OFF and toggling the switch
(declared in envs.py, hence in compile_factors) invalidates the compile
cache. project_() is an in-place rank-1 update with no full-size
temporary, and skips tensors whose width is not hidden_size.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:40:16 +07:00
MOVZX
76df77c04e
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:16:39 +07:00
Laith Sakka and Nick Hill
9cede34a67
[Bugfix] Create cuBLAS handles before memory profiling to avoid OOM ( #61044 )
...
Signed-off-by: Laith Sakka <lsakka@meta.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-10-15 08:59:22 -04:00
Martin Hickey
252be28c18
[Frontend] Replace return_mm_kwargs with an mm_payload enum on render requests ( #61427 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-10-15 12:41:46 +00:00
Martin Hickey
c6b772cf6b
[Bugfix][Frontend] Emit kv/ec transfer params at the top level of render output ( #61241 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-10-15 12:34:15 +00:00
Micah Williamson
e8e1e96196
[ROCm][DSv4] AITER MegaMoE Fixed-Slot Dispatch For DeepSeek-V4 ( #61262 )
...
Signed-off-by: Micah Williamson <micah.williamson@amd.com >
2026-10-15 19:10:59 +08:00
8b27d27402
[Bugfix][KV Offload] Always attempt NIXL agent removal during peer teardown ( #60279 )
...
Signed-off-by: nej1gotnochill <divyanshlalotra2@gmail.com >
Co-authored-by: Codebuff <noreply@codebuff.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 14:07:23 +03:00
Talor Abramovich
81e42c9987
[Feature][Frontend] Control the log level of request JSON body using env var ( #60630 )
...
Signed-off-by: talora <talora@nvidia.com >
2026-10-15 11:07:19 +00:00
jimmy-adams
365aaa77bd
[Bugfix][DSv4] Remap Quark MTP layer_quant_config onto draft prefixes ( #56054 )
...
Signed-off-by: jimmy-adams <41593649+jimmy-adams@users.noreply.github.com >
2026-10-15 18:51:57 +08:00
e10c10a644
[Core] Add PROMOTION_LATENCY histogram metric for tiering offload ( #53910 )
...
Signed-off-by: Wen Zhou <wenzhou@redhat.com >
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-10-15 13:47:41 +03:00
Pei Li and Kevin H. Luu
6fe7401fac
[Perf] Read VLLM_COMPUTE_NANS_IN_LOGITS once per IterationStats ( #61710 )
...
Signed-off-by: Pei Li <peili@nvidia.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 18:35:54 +08:00
Timofey and Codex
716c5225ae
[Model] Add AliceAI support ( #57963 )
...
Signed-off-by: timka <timka.byzov@yandex.ru >
Co-authored-by: Codex <noreply@openai.com >
2026-10-15 10:15:03 +00:00
7020600b1d
[KV Connector][NIXL] Register packed MLA KV rows as a single region ( #61074 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
2026-10-15 09:18:15 +00:00
Flora Feng and Chauncey
80dd705c1d
[Frontend] Make finish_reason and Responses item status truthful ( #59324 )
...
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-10-15 10:12:02 +01:00
Denis Khachyan and Claude Code
427915bed2
[Build][CPU] Install Run:ai Model Streamer in vllm-openai-cpu image ( #61382 )
...
Signed-off-by: Denis Khachyan <khachyanda@gmail.com >
Co-authored-by: Claude Code <noreply@anthropic.com >
2026-10-15 09:11:51 +00:00
Kevin H. Luu and Claude Opus 5.5
7f730b4cea
[CI] Run 4xH100 steps nightly only ( #61782 )
...
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 01:32:43 -07:00
wang.yuqi
0f0bc4dbb1
[Test] Split the chat_utils tests. ( #61765 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-10-15 08:26:22 +00:00
b965ee564d
[Chore] Use upstream Transformers DeepseekV4Config ( #61392 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 01:08:08 -07:00
Misha Goin and Claude Opus 5.5
1c4c319792
[Frontend] Advertise --enable-x/--disable-x for boolean CLI flags ( #61479 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 01:08:02 -07:00
48f17b77bb
[Bugfix] Reject mismatched float16/bfloat16 KV cache dtype ( #60885 )
...
Signed-off-by: cipheraxat <cipheraxat@users.noreply.github.com >
Co-authored-by: cipheraxat <cipheraxat@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 08:06:59 +00:00
fc8e08a2a6
[Bugfix][KV Offload] Do not let mark_miss() flip a pending or in-flight async lookup probe ( #59096 )
...
Signed-off-by: Sohom Chakraborty <16609933+sohom-cs@users.noreply.github.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 10:57:46 +03:00
Bugen Zhao and Claude Opus 5.5
8fa5ae844a
[Frontend] Let an unset --tool-strict-level fall back to the tool parser's default ( #61328 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 15:48:38 +08:00
91ed2859c7
[CPU][Zen] Update zentorch to 2.14.0.1 & enable float16 on Venice and later ( #61411 )
...
Signed-off-by: Ganesh R <ganesh.r@amd.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-15 00:23:50 -07:00
85c94c1706
[Tests][XPU] Skip modelopt_fp4 online MoE re-quantization test on XPU ( #61723 )
...
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-15 00:23:45 -07:00
Kevin H. Luu and Claude Opus 5.5
8f111b22ee
[CI] Run 4xB200 steps nightly only ( #61541 )
...
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 00:08:48 -07:00
Kevin H. Luu and Claude Opus 5.5
32feecd91f
[CI] Pin DeepSeek-V4.1-Flash-NVFP4 eval to its FP8-Engram revision ( #61763 )
...
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 00:05:18 -07:00
Chaojun Zhang and Kevin H. Luu
ab593678b5
[CI][XPU] Ignore test_reload.py on Intel CI ( #61740 )
...
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 15:03:34 +08:00
a9fe30e9f6
[Bugfix][Multimodal] Recover PyNvVideoCodec decoder slots after failures ( #59597 )
...
Signed-off-by: Daniel Afrimi <dafrimi@nvidia.com >
Signed-off-by: danielafrimi <45691845+danielafrimi@users.noreply.github.com >
Signed-off-by: Daniel Afrimi <dafrimi@oci-hsg-cs-001-login-02.cm.cluster >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Daniel Afrimi <dafrimi@oci-hsg-cs-001-login-02.cm.cluster >
Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com >
2026-10-15 09:17:19 +03:00
liranschour and Kevin H. Luu
c5f8adc0b1
[KV Offload] Expose OffloadingManager.lock()/unlock() and service the tiering control plane off the scheduler thread ( #58168 )
...
Signed-off-by: Liran Schour <lirans@il.ibm.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 08:50:56 +03:00
Harshil and Cyrus Leung
c54dca96bd
[Bugfix][Benchmark] Fix probe latency and duration in bench serve ( #60511 )
...
Signed-off-by: QHarshil <harshil_c@hotmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-15 05:50:30 +00:00
5341b5ba7d
[KV Offload] Pin only each worker's own offload slots ( #59544 )
...
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-10-15 08:49:42 +03:00
6605802904
[Perf][KV Connector][Mooncake] Send the KV layout once per peer ( #59825 )
...
Signed-off-by: staryxchen <staryxchen@tencent.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com >
2026-10-15 05:36:31 +00:00
a400804ce4
[CI] Relax MTP sharded sampling cross-boot noise bound ( #57026 )
...
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com >
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 22:31:45 -07:00
3f1bcfd82b
[Bugfix][KV Offload] Bound unanswered P2P lookup waits ( #56836 )
...
Signed-off-by: jacklin78911-collab <jacklin78911@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 08:23:34 +03:00
81e1c3f183
[XPU][MLA] Integrate XPU MLA support into the existing FLASH_ATTN_MLA backend ( #58297 )
...
Signed-off-by: baodii <di.bao@intel.com >
Signed-off-by: Zhouxiaoya25 <xiaoya.zhou@intel.com >
Signed-off-by: ZhouXiaoya25 <xiaoya.zhou@intel.com >
Co-authored-by: baodii <di.bao@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-10-15 12:47:37 +08:00
Vivaan Sharma and Kevin H. Luu
96b5b0ee4b
[Model] Enable torch.compile and CUDA graphs for EmbeddingGemma2 ( #61572 )
...
Signed-off-by: Vivaan Sharma <99265792+Vivaansharma2003@users.noreply.github.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 12:32:39 +08:00
Haiyu Zuo and Kevin H. Luu
ff5a91a456
[Feature][Model Loader] Apply model-owned MTP shard selection to RunAI ModelStreamer ( #60759 )
...
Signed-off-by: 浩韵 <zuohaiyu.zhy@alibaba-inc.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 12:23:11 +08:00
Chaitanya Sri Krishna Lolla and Cursor
833665cded
[ROCm][DI] Enable MoRI FP4 dispatch for mxfp4 w4a4 moe ( #59596 )
...
Signed-off-by: lcskrishna <lollachaitanya@gmail.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-10-14 23:20:14 -05:00
Tuukka Sarvi and Kevin H. Luu
10d2c6083d
[ROCm][DSv4.1] Enable DSpark adaptive verification ( #58151 )
...
Signed-off-by: Tuukka Sarvi <tuukka.sarvi@amd.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 12:05:43 +08:00
Shanshan Shen
f46021a7fa
[ROCm][Perf] Enable DPA+ETP prefill compute/comm dual-batch-overlap for DeepSeek-V4 ( #57773 )
...
Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com >
Signed-off-by: shen-shanshan <467638484@qq.com >
2026-10-15 11:16:48 +08:00
Feanor and Taneem Ibrahim
5d5c3d7c43
[Bugfix] Preserve padding token types for cross encoders ( #58143 )
...
Signed-off-by: zhaoye <772971548@qq.com >
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
2026-10-15 03:09:24 +00:00
429a3da665
[Bugfix][ROCm][Disagg] Allow MoRIIO HMA per-layer block_len ( #58585 )
...
Signed-off-by: avininjamay8 <Avinash.Paul@amd.com >
Signed-off-by: avininjamay8 <avpaul@amd.com >
Co-authored-by: avininjamay8 <avpaul@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Shanshan Shen <467638484@qq.com >
2026-10-15 09:35:31 +08:00
Kai and Kunshang Ji
de2b869ee9
[Bugfix][Kernel] Register int8 scaled_mm as a custom op ( #57331 )
...
Signed-off-by: Kai <limkc.reply@gmail.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-10-15 09:27:49 +08:00
Aarushi Jain
2abd479aee
[ROCm][CI] Add AMD job for DeepSeek-V4.1 Mono Decode Numerics ( #61720 )
...
Signed-off-by: aarushjain29 <aarushi.jain2@amd.com >
2026-10-15 01:23:25 +00:00
Woosuk Kwon and Claude Opus 5.5
e57bb4ace7
[Core] Remove the legacy Ray compiled-graph executor ( #61712 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 18:13:27 -07:00
bfdbfa1632
[Bugfix][PCP] Align MoE routing padding with gathered tokens ( #60946 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: wangyicong <wangyicong@bytedance.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-10-14 21:04:00 -04:00
Woosuk Kwon and Claude Opus 5.5
8530aed2b7
[Spec Decode] Remove Medusa and MLP speculator support ( #61701 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 17:29:23 -07:00
josh-schell-amd and Claude Opus 5.5
273a04964e
[ROCm] Bump AITER to v0.1.25 and relax the DSv4.1 mono MoE tolerance ( #61697 )
...
Signed-off-by: josh schell <josh.schell@amd.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-14 23:50:24 +00:00