MOVZX
c5ab42bc0f
Merge branch 'main' into patches
2026-10-18 16:04:36 +07:00
2067297b2f
[ROCm][Perf] Use AITER tuned configs to select a4w4 gemm kernel instead of env var ( #61278 )
...
Signed-off-by: Andy Friedrich <afriedri@amd.com >
Signed-off-by: afriedri <afriedri@amd.com >
Co-authored-by: Simon Danielsson <70206058+simondanielsson@users.noreply.github.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-10-17 04:00:28 -05:00
MOVZX
e0fe5df731
Merge branch 'main' into patches
2026-10-17 08:06:56 +00:00
153a17166e
[Frontend] Add the Beam reasoning and tool parsers ( #62052 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Yoni Gozlan <74535834+yonigozlan@users.noreply.github.com >
Co-authored-by: neilkg <33635204+neilkg@users.noreply.github.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-16 14:08:29 -07:00
Dan Blanaru and Claude Opus 5.5
9731f30a03
[Bugfix] Keep VLLM_RAY_BUNDLE_INDICES out of the compile cache key ( #61211 )
...
Signed-off-by: DanBlanaru <48605845+DanBlanaru@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-16 20:59:48 +00:00
MOVZX and Hinata
0076adc78b
[Qwen3.5] Project the refusal direction out of the residual stream at runtime
...
orcarouter/Qwen3.8-27B-Uncensored removes refusal by editing weights. The
Swift release we serve is AutoRound W8A16, so re-editing weights means
re-quantizing. Project one unit direction r out of every residual-stream
writer output instead: x <- x - r (r . x), which equals the weight edit
W' = W - r(r^T W) on every writer feeding that stream (Arditi et al. 2024).
Sites: attention output for both the full-attention and GDN branches, the
MLP output, and token embeddings, in Qwen3NextDecoderLayer.forward and
Qwen3NextModel.embed_input_ids, which Qwen3_5DecoderLayer and Qwen3_5Model
inherit but whose base constructors they skip, so both register there too.
The MTP drafter reuses Qwen3_5DecoderLayer and calls its own
embed_input_ids, so the draft path is covered at the same width, and
_maybe_share_embeddings at PP=1 hands the draft the target embedding
module. Sequence parallelism shards dim 0 only, so the last dim is
hidden_size at every site.
VLLM_ABLITERATION names a JSON file with key refusal_direction, a
safetensors tensor, or a bare name resolved under
/opt/qwen38/abliteration then /data/abliteration. Empty or 0 is off.
register() always creates the buffer _abliteration_r, zeros when off, so
one compiled graph serves ON and OFF and toggling the switch
(declared in envs.py, hence in compile_factors) invalidates the compile
cache. project_() is an in-place rank-1 update with no full-size
temporary, and skips tensors whose width is not hidden_size.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:40:16 +07:00
Talor Abramovich
81e42c9987
[Feature][Frontend] Control the log level of request JSON body using env var ( #60630 )
...
Signed-off-by: talora <talora@nvidia.com >
2026-10-15 11:07:19 +00:00
Woosuk Kwon and Claude Opus 5.5
e57bb4ace7
[Core] Remove the legacy Ray compiled-graph executor ( #61712 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 18:13:27 -07:00
xuebwang
81503550f3
[ROCm] Disable AITER FP8/FP4 BMM for MLA by default ( #53123 )
...
Signed-off-by: xuebwang-amd <xuebwang@amd.com >
2026-10-14 10:46:02 +00:00
Juan Pérez de Algaba and OpenAI Codex
7173c64e2e
[Security] Bound decoded and downloaded media per request ( #61226 )
...
Signed-off-by: juanpas97 <juan.perezalgaba@gmail.com >
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-10-13 10:17:14 +00:00
Fangzhou Ai and Claude Opus 5.5
cc2a6eea1a
[ROCm][Perf][DSv4.1] Mono decode on every backbone layer ( #61082 )
...
Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-13 10:16:50 +08:00
AMD-yanfeiwang and aoshen02
e3bbaf2369
[ROCm][DSv4] Fuse the shared expert into AITER MegaMoE ( #61004 )
...
Signed-off-by: Yanfei Wang <yanfei.wang@amd.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-12 15:53:16 +08:00
93d2cab35e
[Security] Fix P2P ZMQ socket exhaustion via unbounded peer sessions ( #51504 )
...
Signed-off-by: jperezde <jperezde@redhat.com >
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
2026-10-11 11:39:11 -04:00
Fangzhou Ai and Claude Opus 5.5
193922d6ff
[ROCm][Perf][DSv4.1] Mono decode layer for DeepSeek-V4.1-Flash on MI355X ( #60397 )
...
Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 20:53:18 -05:00
Hyukjoon Lee and Claude Opus 5.5
2342343746
[ROCm][DCP] Default MLA DCP verify to the round-robin asm decode where it applies ( #59965 )
...
Signed-off-by: hyukjlee <hyukjlee@amd.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-07 03:10:54 +00:00
Ayaan Nadamal
84593955fa
[Perf][MoE] Support fp8 combine in FlashInfer one-sided MoE all2all ( #57995 )
...
Signed-off-by: Ayaan Nadamal <anadamal@nvidia.com >
2026-10-02 16:29:16 -04:00
Wentao Ye
f700178f90
[Refactor] Remove dead env and config ( #59781 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-10-02 18:07:20 +00:00
e12291d733
[CPU][Zen] Add DA8W4 (W4A8) int4 support for dense and MoE layers ( #54024 )
...
Signed-off-by: R <Ganesh.R@amd.com >
Signed-off-by: Ganesh R <Ganesh.R@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-01 15:18:25 +08:00
Juan Pérez de Algaba and Cursor
cff08b461e
[Security] Fix chat template resource-exhaustion DoS (GHSA-4hhp-h66f-… ( #50300 )
...
Signed-off-by: jperezde <jperezde@redhat.com >
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-30 08:32:00 -07:00
Fangzhou Ai and Cursor Agent
77e8b02c8d
[ROCm][Perf] Add opt-in a4w4 (FP4 activation) MoE for DeepSeek V4.1 on AITER ( #58819 )
...
Signed-off-by: fai <fangzhouai@gmail.com >
Signed-off-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com >
Co-authored-by: Cursor Agent (Claude Sonnet 5) <cursoragent@cursor.com >
2026-09-30 15:07:44 +08:00
Tyler Michael Smith and Codex
50239dfc4d
[WideEP] Change DeepEPv2 to auto select hybrid mode by default ( #57991 )
...
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-29 18:36:30 -04:00
Mehmet Cagri
70ae0b7435
[ROCm][MLA] Enable sparse MLA Gluon kernel from Aiter ( #53492 )
...
Signed-off-by: Mehmet Cagri Kaymak <mehmet.kaymak@amd.com >
Signed-off-by: Mehmet Cagri <mehmet.kaymak@amd.com >
2026-09-29 09:08:18 +08:00
25624f65cf
[ROCm][MLA] Add an AITER ASM round-robin decode route for DCP multi-token verify ( #56861 )
...
Signed-off-by: Xiaohu Guo <Xiaohu.Guo@amd.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: seungrokj <144636725+seungrokj@users.noreply.github.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-28 14:52:21 -04:00
Chao Lei
386ac2573e
[Bugfix][KV Connector] Retry Mooncake bootstrap registration on timeout ( reopens #55763 ) ( #58919 )
2026-09-27 19:47:36 +00:00
Hongxia Yang
3137ff0773
[ROCm][Kimi-K3] Make VLLM_ROCM_USE_AITER_MOE_SITUV2 select a4w4/a8w4/a16w4 ( #58201 )
...
Signed-off-by: Hongxia Yang <hongxia.yang@amd.com >
2026-09-27 02:26:49 +00:00
YiSheng5 and Kunshang Ji
b05f7d5fbc
[PP][XPU]Add the flag to control microbatch feature on MRV2+PP ( #55145 )
...
Signed-off-by: yisheng <yi.sheng@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-09-25 12:05:35 +08:00
Misha Goin and Codex
71891bdb94
[Perf] Parallelize registered CUDA Triton kernel warmup at startup ( #58582 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-24 19:52:00 -04:00
liuzhenwei
dcfc17e0b1
[XPU] enable XPU GRAPH by default ( #51600 )
...
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com >
2026-09-24 19:17:25 +08:00
f92b78f6ef
[Kernel][DSV4.1] Fuse MXFP8 wo_b GEMM with sequence-parallel reduce-scatter ( #57428 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
Signed-off-by: Canlin <canlinguosdu@gmail.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
2026-09-22 01:07:17 -07:00
yinfengLiu and Shanshan Shen
2af05512cf
[Kernel] Add opt-in load-time MXFP4 dequantization ( #50814 )
...
Signed-off-by: Liuyinfeng01 <yinfeliu@amd.com >
Co-authored-by: Shanshan Shen <467638484@qq.com >
2026-09-22 14:51:12 +08:00
Nicolò Lucchesi
54020c3c3e
[Engram] Drop redundant VLLM_PLE_CPU_OFFLOAD env var ( #57937 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-09-21 20:16:02 +02:00
7d99c2c4fd
[Feature][Humming] Humming feature integration ( #56685 )
...
Signed-off-by: jinzhen.ljz <jinzhen.ljz@antgroup.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
2026-09-20 08:11:01 -07:00
shaohuaxi and Codex
67a8a3f926
[Bugfix][CPU] Fix macOS multimodal SHM cache initialization ( #57142 )
...
Signed-off-by: 子华 <huaxi.shx@alibaba-inc.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-18 09:05:13 +00:00
afriedri
a9a7e45f31
[ROCm] Restore VLLM_ROCM_USE_AITER_FP4_ASM_GEMM and default w4a4 ASM GEMM back to off ( #57055 )
...
Signed-off-by: Andy Friedrich <afriedri@amd.com >
2026-09-17 13:39:52 -05:00
Nick Hill
5efbf0886a
[CI] Support FORCE_COLOR env var; enable colors in CPU CI output ( #56913 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-09-16 22:40:21 -04:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
wangxiyuan and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
118e17f5ff
[Platform] Move env check function to platform interface ( #48599 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-16 10:08:36 +00:00
wangxiyuan
b0898a4937
[Misc] Allow install empty package from pip ( #41074 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2026-09-16 02:02:37 +00:00
a4d2d9d95e
[ROCm][Kimi-K3] Enable a4w4 flydsl kernels for KimiK3 ( #53940 )
...
Signed-off-by: ppalanga <ppalanga@amd.com >
Signed-off-by: Poovaiah Palangappa <poovaiah.palangappa@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Poovaiah Palangappa <poovaiah.palangappa@amd.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com >
2026-09-16 01:38:02 +00:00
Martin Hickey
934b1fcbcb
[Frontend] Replace VLLM_ENABLE_SCALE_OUT_ENDPOINTS with --enable-scale-out ( #55176 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-09-14 10:02:14 +00:00
d2d649e674
[DS V4.1][Engram] Support async prefetch for offloaded engram lookups and engram DP sharding ( #56512 )
...
Signed-off-by: Juntian Liu <Juntianl777@gmail.com >
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: ywang96 <>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
2026-09-13 02:44:23 -07:00
Itay Alroy
658c8131c7
[Elastic EP] Reuse CUDA graphs across reconfiguration ( #54985 )
2026-09-12 10:26:39 -04:00
Wentao Ye
5fe77aecfc
[Deprecation] Deprecate items scheduled for 0.29 ( #55353 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-11 08:17:02 -07:00
Simon Danielsson
ae48466cf3
[Perf][ROCm] Add AITER custom AG/RS (DP only) ( #48247 )
...
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com >
2026-09-11 13:19:57 +08:00
Harry Huang and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
3116c5d06b
[Qwen4Exp] Support UVA PLE-offload and Engram tensor parallelism ( #54371 )
...
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-09 07:32:34 -07:00
Nick Hill
4aadb0f14d
[Core] Enhance cpu<->gpu sync checking to include paged async copies ( #53491 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-09-08 17:26:21 -07:00
Francisco Javier Arceo
c00091e026
[Frontend] Gate scale-out endpoints behind opt-in flag ( #54579 )
...
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com >
2026-09-02 14:31:14 +08:00
lzhan011 and Cyrus Leung
ce7391712b
[Bugfix][Security] Bound embedding densification before to_dense() ( #54632 )
...
Signed-off-by: lzhan011 <35493221+lzhan011@users.noreply.github.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-09-01 08:45:14 +00:00
446c769482
[Distributed] Add opt-in FlashInfer PCIe IPC all-reduce backend ( #53576 )
...
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-01 03:34:56 +00:00