MOVZX
be95edb7f6
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-16 17:35:42 +07:00
MOVZX and Hinata
18c06ed406
[Offloader] Stream multimodal towers from host RAM with bulk-copy
...
Add --offload-towers: place multimodal tower weights in pinned host
memory and bulk-copy each unit to the GPU only during its forward,
keeping towers out of VRAM between encodes. Unlike UVA zero-copy,
whole-tensor H2D transfers avoid re-reading operand tiles over PCIe
inside the GEMM inner loop. Tower streaming is independent of layer
group scheduling, so group_size may be 0 for tower-only offload.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-16 14:32:14 +07:00
fa75d3456d
[Feature][PCP] Add DeepSeek-V4.1-Flash support ( #59857 )
...
Signed-off-by: wangyicong <wangyicong@bytedance.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 23:17:51 -07:00
dskwe
49cb2bbf71
[Bugfix] Honor code_revision in config-based tokenizer loading ( #61611 )
...
Signed-off-by: Yuan Li <dskwelmcy@163.com >
2026-10-16 05:21:12 +00:00
4ac1f8d2ac
[Bugfix][Structured Output] Mask grammar-invalid implicit reasoning ends ( #61574 )
...
Signed-off-by: Maroon Ayoub <mayoub@redhat.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-10-16 12:53:41 +08:00
maithilijoshi20 and Cyrus Leung
92dc3dd393
[Bugfix] Honor OPT token embedding width for prompt embeddings ( #61736 )
...
Signed-off-by: maithilijoshi20 <maithilij2003@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-16 04:41:50 +00:00
Andreas Karatzas and Claude
b466fbacd2
[ROCm][Perf] Any-shape low-latency MoE router GEMM on MI355 ( #61895 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-15 23:22:31 -05:00
dd9e07a25f
[Feature] Add External LB Mode Support for Elastic EP Scaling ( #43202 )
...
Signed-off-by: lqh-hh <liuqihang6@huawei.com >
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: lqh-hh <liuqihang6@huawei.com >
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: 2017403603 <705328646@qq.com >
Co-authored-by: Julien Mancuso <jmancuso@nvidia.com >
Co-authored-by: Itay Alroy <75032521+itayalroy@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-15 21:29:02 -04:00
Andreas Karatzas and Claude
29345f42a6
[ROCm][Qwen4Exp] Support HC and MoE sequence parallelism on ROCm ( #61471 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-15 20:11:17 -05:00
a8be1500ba
[Bugfix][MoE] Restore TP-sharded routed-expert capture for DP=1 ( #61388 )
...
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: Ao Shen <281812371+aoshen02@users.noreply.github.com >
Signed-off-by: Ao Shen <aoshen@inferact.ai >
Co-authored-by: Ao Shen <281812371+aoshen02@users.noreply.github.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Ao Shen <aoshen@inferact.ai >
2026-10-16 00:21:24 +00:00
Jack Hu and Douglas Lehr
ef38e8d006
[Model][Hardware][AMD][Kernel] Part 2/2 -> Enable e2e QK Norm + RoPE + KV Cache runtime fusion for Qwen3-30B-A3B on ROCM_ATTN ( #42755 )
...
Signed-off-by: Jack Hu <Jack.Hu@amd.com >
Signed-off-by: Jack Hu <jingxujaster@hotmail.com >
Signed-off-by: Jack Hu <jack.hu@amd.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
2026-10-15 17:10:14 -07:00
Juntian Liu and Claude Opus 5.5
19b0efce30
[Perf][Core] Skip the null head of sliding-window block tables ( #61724 )
...
Signed-off-by: Juntian Liu <Juntianl777@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 16:14:51 -07:00
Woosuk Kwon and Claude Opus 5.5
4a89316efa
[Core] Drop the Ray driver-node GPU pin left over from the legacy executor ( #61713 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 16:00:18 -07:00
3f1e57225a
[ROCm][Perf][Kimi-K3] Use AITER Gluon KDA kernels for decode and spec decode ( #57392 )
...
Signed-off-by: omuhamma <omar.muhammad@amd.com >
Signed-off-by: Nicholas Susanto <nicholas.susanto@amd.com >
Co-authored-by: Nicholas Susanto <nicholas.susanto@amd.com >
Co-authored-by: Simon Danielsson <70206058+simondanielsson@users.noreply.github.com >
2026-10-15 17:54:24 -05:00
de7d70676d
[Perf] Low-latency CPU all-reduce for DP metadata syncs ( #61767 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: hjjq <50634613+hjjq@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-15 15:29:13 -07:00
Stefano Castagnetta and Kevin H. Luu
784d1fff0c
[MRV2][Spec Decode] Quantized draft lm_head for drafters that share the target head ( #59973 )
...
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 22:06:40 +00:00
d92a4f25f2
[watermarking] Add DFlash support to dual-key gumbel-max watermarking ( #59554 )
...
Signed-off-by: Shern Shiou Tan <shernshiou@gmail.com >
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Raphael Rialland <raphael.rialland@mistral.ai >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-10-15 22:06:06 +00:00
e711723352
[ROCm] gfx1250 optimizations and model enablement ( #58800 )
...
Signed-off-by: jpvillam <juan.villamizar@amd.com >
Signed-off-by: Jaden Mathias <jaden.mathias@amd.com >
Signed-off-by: Juan Villamizar <jpvillam@amd.com >
Signed-off-by: jpvillam <jpvillam@amd.com >
Signed-off-by: Omar Muhammad <omar.muhammad@amd.com >
Signed-off-by: Daniel <danichan@amd.com >
Signed-off-by: Juan Villamizar <100237675+jpvillam-amd@users.noreply.github.com >
Co-authored-by: Jaden Mathias <jaden.mathias@amd.com >
Co-authored-by: jpvillam <jpvillam@amd.com >
Co-authored-by: Omar Muhammad <omar.muhammad@amd.com >
Co-authored-by: Daniel <danichan@amd.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Daniel Chan <Daniel.Chan@amd.com >
2026-10-15 14:51:57 -07:00
DimensionSTP and Kevin H. Luu
9edcc7fa7e
[Bugfix][Spec Decode] Avoid reusing random draws across block verification rounds ( #61161 )
...
Signed-off-by: DimensionSTP <65501090+DimensionSTP@users.noreply.github.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 14:46:38 -07:00
Mikko Tukiainen
0c25849292
[ROCm][Perf][Bugfix] Probe the preshuffled FP4 GEMM config by logical K ( #59406 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
2026-10-15 21:11:33 +00:00
f8460ef236
[Kernel][DSv4.1] Bump FlashMLA to the 2026.09.30 release and its interleaved V4.1 KV cache ( #61371 )
...
Signed-off-by: juntianl <juntianl@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Kimi Code <noreply@moonshot.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-10-15 13:58:25 -07:00
33212db2c2
[KV Connector][NIXL] Reject NVFP4 KV cache with heterogeneous TP head split ( #61865 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 13:57:37 -07:00
SeongJun Lee and Wei Zhao
ee8c47961b
[CI] Use an ungated model as the attention test config default ( #61732 )
...
Signed-off-by: lesj0610 <lesj0610@gmail.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-10-15 20:19:09 +00:00
21f4b6dc84
[KV Offload] Split the CPU offload region across shm files ( #59546 )
...
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 20:17:40 +00:00
d12ea8e309
[Bugfix][MRV2] Keep input_ids on later PP stages for raw-token models ( #61016 )
...
Signed-off-by: booth-algo <kevinlauofficial01@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 12:54:21 -07:00
d486219fc2
[Bugfix][Core] Uncount stop-trimmed spec tokens from num_computed_tokens ( #58668 )
...
Signed-off-by: David Cheung <d7cheung@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-10-15 12:50:06 -07:00
49f944be3e
[Perf][HiSparse] Reuse pinned host pool after graph profiling ( #61668 )
...
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 12:33:03 -07:00
Rohan Potdar and Claude Opus 5.5
105605b8e9
[ROCm] Support DeepSeek-V4 hash routing in AITER Triton W4A16 MoE ( #60282 )
...
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 14:10:12 -05:00
Thien Tran
82b9ccaa79
[Perf][MoE] Skip the TRTLLM fp8-block activation-scale transpose copy ( #61773 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-10-15 12:09:07 -07:00
4d58fa254a
[MoE] Unify DeepGEMM MegaMoE experts behind one feature-matrix layer; add NVFP4 ( #61538 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 19:02:01 +00:00
Misha Goin and Claude Opus 5.5
4d9a8d6a41
[Bugfix] Make Humming DeviceInfo lookups traceable under torch.compile ( #61858 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 11:12:21 -07:00
Mikko Tukiainen and Cursor
47fd76e4ce
[ROCm][Perf] Route softmax MoE top-k, including fused shared-expert sigmoid, through AITER topk_gating ( #56226 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-10-15 17:53:18 +00:00
4f49b55c1f
[MoE] Shape-based padding for FlashInfer TRT-LLM MXFP8 experts (gated and non-gated) ( #61293 )
...
Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-10-15 13:44:11 -04:00
7cd7982dfd
[Spec Decode] Don't reserve DFlash/DSpark draft slots in the scheduler token budget ( #59468 )
...
Signed-off-by: zixi-qi <zixi@inferact.ai >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-16 01:23:58 +08:00
bzsuni and Flora Feng
338dab077c
[Bugfix] Fix Apertus streaming scientific notation ( #61812 )
...
Signed-off-by: bzsuni <bingzhe.sun@daocloud.io >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-15 16:49:57 +00:00
957d512e96
[Bugfix][Frontend] Preserve streaming chunk boundaries in RequestOutputCollector ( #58941 )
...
Signed-off-by: David Cheung <d7cheung@gmail.com >
Co-authored-by: Benedikt Falk <bfalk@nvidia.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-10-15 16:38:28 +00:00
Jovan
a9e90b9d1d
[Bugfix][Frontend] Reject Cohere REQUIRED tool choice without tools ( #61825 )
...
Signed-off-by: Kevin <jia__kuan@163.com >
2026-10-15 16:37:08 +00:00
d6f0b7a416
[CI] Skip exited descendants when finding the worker in the FT e2e test ( #61648 )
...
Signed-off-by: Oxana Korzh <okorzh@amd.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-10-15 16:08:14 +00:00
ed3050add4
[Core] Apply prefill_schedule_interval outside data parallelism ( #54627 )
...
Signed-off-by: lifulu <fululi12@amd.com >
Signed-off-by: lifulu <lifulu@users.noreply.github.com >
Co-authored-by: Cursor Agent <cursoragent@cursor.com >
Co-authored-by: lifulu <lifulu@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-10-15 11:03:13 -05:00
c825e1d235
[Perf] Fuse the mamba align block-table gather into one Triton kernel, 7 kernels -> 1, 6x speedup on all shapes for this kernel ( #58737 )
...
Signed-off-by: lifulu <fululi12@amd.com >
Signed-off-by: lifulu <lifulu@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: lifulu <lifulu@users.noreply.github.com >
2026-10-15 11:02:53 -05:00
MOVZX and Hinata
5d6754ff6b
fix: keep the refusal-direction projection dynamo-safe
...
With VLLM_ABLITERATION set, boot crashed: dynamo traces
abliteration.register from Qwen3NextGatedDeltaNet.__init__, and
buf.copy_(r) fails on the meta device during the compile-time model
construction pass. Disable tracing for register and guard the copy so
an unmaterializable buffer turns projection off with a warning.
Log activation at registration instead of inside project_: project_
runs in the compiled region, and a logger call there aborts dynamo.
The width guard uses the same escape hatch.
Also project the embedding that Qwen3NextMultiTokenPredictor builds
for itself, which the first pass missed, and add tests for the
2560-value Flash-Next direction against a 5120 model and for a
half-width activation.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 22:56:06 +07:00
3dcfaeb4e8
[Frontend] Apply each endpoint's top_logprobs cut in derender ( #59518 )
...
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com >
Co-authored-by: Yin Li <kevinli.ai.work@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-15 15:04:04 +00:00
MOVZX and Hinata
0076adc78b
[Qwen3.5] Project the refusal direction out of the residual stream at runtime
...
orcarouter/Qwen3.8-27B-Uncensored removes refusal by editing weights. The
Swift release we serve is AutoRound W8A16, so re-editing weights means
re-quantizing. Project one unit direction r out of every residual-stream
writer output instead: x <- x - r (r . x), which equals the weight edit
W' = W - r(r^T W) on every writer feeding that stream (Arditi et al. 2024).
Sites: attention output for both the full-attention and GDN branches, the
MLP output, and token embeddings, in Qwen3NextDecoderLayer.forward and
Qwen3NextModel.embed_input_ids, which Qwen3_5DecoderLayer and Qwen3_5Model
inherit but whose base constructors they skip, so both register there too.
The MTP drafter reuses Qwen3_5DecoderLayer and calls its own
embed_input_ids, so the draft path is covered at the same width, and
_maybe_share_embeddings at PP=1 hands the draft the target embedding
module. Sequence parallelism shards dim 0 only, so the last dim is
hidden_size at every site.
VLLM_ABLITERATION names a JSON file with key refusal_direction, a
safetensors tensor, or a bare name resolved under
/opt/qwen38/abliteration then /data/abliteration. Empty or 0 is off.
register() always creates the buffer _abliteration_r, zeros when off, so
one compiled graph serves ON and OFF and toggling the switch
(declared in envs.py, hence in compile_factors) invalidates the compile
cache. project_() is an in-place rank-1 update with no full-size
temporary, and skips tensors whose width is not hidden_size.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:40:16 +07:00
Matthias Gehre
011fa68ca9
[ROCm][Perf] Pad fp16/bf16 GEMM weight row stride for L2 cache ( #55090 )
...
Signed-off-by: Matthias Gehre <matthias.gehre@amd.com >
2026-10-15 14:26:32 +00:00
MOVZX
76df77c04e
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:16:39 +07:00
Ronald and aoshen02
d12d5460a5
[Bugfix][MoE] Keep routing rows during DeepSeek V4.1 decoder replay ( #61643 )
...
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-15 14:02:06 +00:00
24f632ca7e
[Bugfix] Engage indexer TP row sharding on total rows, not per-rank rows ( #61647 )
...
Signed-off-by: Zheng Cai <caizheng1993@gmail.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
2026-10-15 13:54:14 +00:00
mosafariuk
69425846f8
[Bugfix][CPU] Report the real cause when the worker outgrows its memory reservation ( #59472 )
...
Signed-off-by: Mo <mosafariuk@gmail.com >
2026-10-15 21:52:05 +08:00
b7b20b1b8f
[Bugfix][Frontend] Leave the final stop token out of chat derender text ( #61757 )
...
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-15 21:51:34 +08:00
littlecircle0730 and Shanshan Shen
71536bd5f8
[MM][CG] Support ViT full CUDA graph for DeepSeek-VL2 ( #46005 )
...
Signed-off-by: littlecircle0730 <littlecircle0730@gmail.com >
Signed-off-by: littlecircle0730 <43994952+littlecircle0730@users.noreply.github.com >
Co-authored-by: Shanshan Shen <467638484@qq.com >
2026-10-15 21:38:52 +08:00
MOVZX
