33212db2c2
[KV Connector][NIXL] Reject NVFP4 KV cache with heterogeneous TP head split ( #61865 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 13:57:37 -07:00
Wentao Ye
54d9b1cb48
[GLM5.3 Perf] Inplace update draft decode metadata, 1.7% E2E Throughput Improvement ( #61873 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-10-15 20:20:12 +00:00
21f4b6dc84
[KV Offload] Split the CPU offload region across shm files ( #59546 )
...
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 20:17:40 +00:00
d12ea8e309
[Bugfix][MRV2] Keep input_ids on later PP stages for raw-token models ( #61016 )
...
Signed-off-by: booth-algo <kevinlauofficial01@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 12:54:21 -07:00
d486219fc2
[Bugfix][Core] Uncount stop-trimmed spec tokens from num_computed_tokens ( #58668 )
...
Signed-off-by: David Cheung <d7cheung@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-10-15 12:50:06 -07:00
49f944be3e
[Perf][HiSparse] Reuse pinned host pool after graph profiling ( #61668 )
...
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 12:33:03 -07:00
Rohan Potdar and Claude Opus 5.5
105605b8e9
[ROCm] Support DeepSeek-V4 hash routing in AITER Triton W4A16 MoE ( #60282 )
...
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 14:10:12 -05:00
Thien Tran
82b9ccaa79
[Perf][MoE] Skip the TRTLLM fp8-block activation-scale transpose copy ( #61773 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-10-15 12:09:07 -07:00
4d58fa254a
[MoE] Unify DeepGEMM MegaMoE experts behind one feature-matrix layer; add NVFP4 ( #61538 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 19:02:01 +00:00
Misha Goin and Claude Opus 5.5
4d9a8d6a41
[Bugfix] Make Humming DeviceInfo lookups traceable under torch.compile ( #61858 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 11:12:21 -07:00
Mikko Tukiainen and Cursor
47fd76e4ce
[ROCm][Perf] Route softmax MoE top-k, including fused shared-expert sigmoid, through AITER topk_gating ( #56226 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-10-15 17:53:18 +00:00
4f49b55c1f
[MoE] Shape-based padding for FlashInfer TRT-LLM MXFP8 experts (gated and non-gated) ( #61293 )
...
Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-10-15 13:44:11 -04:00
7cd7982dfd
[Spec Decode] Don't reserve DFlash/DSpark draft slots in the scheduler token budget ( #59468 )
...
Signed-off-by: zixi-qi <zixi@inferact.ai >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-16 01:23:58 +08:00
bzsuni and Flora Feng
338dab077c
[Bugfix] Fix Apertus streaming scientific notation ( #61812 )
...
Signed-off-by: bzsuni <bingzhe.sun@daocloud.io >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-15 16:49:57 +00:00
957d512e96
[Bugfix][Frontend] Preserve streaming chunk boundaries in RequestOutputCollector ( #58941 )
...
Signed-off-by: David Cheung <d7cheung@gmail.com >
Co-authored-by: Benedikt Falk <bfalk@nvidia.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-10-15 16:38:28 +00:00
Jovan
a9e90b9d1d
[Bugfix][Frontend] Reject Cohere REQUIRED tool choice without tools ( #61825 )
...
Signed-off-by: Kevin <jia__kuan@163.com >
2026-10-15 16:37:08 +00:00
ed3050add4
[Core] Apply prefill_schedule_interval outside data parallelism ( #54627 )
...
Signed-off-by: lifulu <fululi12@amd.com >
Signed-off-by: lifulu <lifulu@users.noreply.github.com >
Co-authored-by: Cursor Agent <cursoragent@cursor.com >
Co-authored-by: lifulu <lifulu@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-10-15 11:03:13 -05:00
c825e1d235
[Perf] Fuse the mamba align block-table gather into one Triton kernel, 7 kernels -> 1, 6x speedup on all shapes for this kernel ( #58737 )
...
Signed-off-by: lifulu <fululi12@amd.com >
Signed-off-by: lifulu <lifulu@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: lifulu <lifulu@users.noreply.github.com >
2026-10-15 11:02:53 -05:00
MOVZX and Hinata
5d6754ff6b
fix: keep the refusal-direction projection dynamo-safe
...
With VLLM_ABLITERATION set, boot crashed: dynamo traces
abliteration.register from Qwen3NextGatedDeltaNet.__init__, and
buf.copy_(r) fails on the meta device during the compile-time model
construction pass. Disable tracing for register and guard the copy so
an unmaterializable buffer turns projection off with a warning.
Log activation at registration instead of inside project_: project_
runs in the compiled region, and a logger call there aborts dynamo.
The width guard uses the same escape hatch.
Also project the embedding that Qwen3NextMultiTokenPredictor builds
for itself, which the first pass missed, and add tests for the
2560-value Flash-Next direction against a 5120 model and for a
half-width activation.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 22:56:06 +07:00
CharlesWu and Yichao Zhu
a97d19d792
[UMBP][1/10] Add shared connector, runtime contracts, and request lifecycle ( #61789 )
...
Signed-off-by: CharlesWu <Charles.Wu@amd.com >
Co-authored-by: Yichao Zhu <Yichao.Zhu@amd.com >
2026-10-15 15:12:39 +00:00
3dcfaeb4e8
[Frontend] Apply each endpoint's top_logprobs cut in derender ( #59518 )
...
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com >
Co-authored-by: Yin Li <kevinli.ai.work@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-15 15:04:04 +00:00
MOVZX and Hinata
0076adc78b
[Qwen3.5] Project the refusal direction out of the residual stream at runtime
...
orcarouter/Qwen3.8-27B-Uncensored removes refusal by editing weights. The
Swift release we serve is AutoRound W8A16, so re-editing weights means
re-quantizing. Project one unit direction r out of every residual-stream
writer output instead: x <- x - r (r . x), which equals the weight edit
W' = W - r(r^T W) on every writer feeding that stream (Arditi et al. 2024).
Sites: attention output for both the full-attention and GDN branches, the
MLP output, and token embeddings, in Qwen3NextDecoderLayer.forward and
Qwen3NextModel.embed_input_ids, which Qwen3_5DecoderLayer and Qwen3_5Model
inherit but whose base constructors they skip, so both register there too.
The MTP drafter reuses Qwen3_5DecoderLayer and calls its own
embed_input_ids, so the draft path is covered at the same width, and
_maybe_share_embeddings at PP=1 hands the draft the target embedding
module. Sequence parallelism shards dim 0 only, so the last dim is
hidden_size at every site.
VLLM_ABLITERATION names a JSON file with key refusal_direction, a
safetensors tensor, or a bare name resolved under
/opt/qwen38/abliteration then /data/abliteration. Empty or 0 is off.
register() always creates the buffer _abliteration_r, zeros when off, so
one compiled graph serves ON and OFF and toggling the switch
(declared in envs.py, hence in compile_factors) invalidates the compile
cache. project_() is an in-place rank-1 update with no full-size
temporary, and skips tensors whose width is not hidden_size.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:40:16 +07:00
Matthias Gehre
011fa68ca9
[ROCm][Perf] Pad fp16/bf16 GEMM weight row stride for L2 cache ( #55090 )
...
Signed-off-by: Matthias Gehre <matthias.gehre@amd.com >
2026-10-15 14:26:32 +00:00
MOVZX
76df77c04e
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:16:39 +07:00
Ronald and aoshen02
d12d5460a5
[Bugfix][MoE] Keep routing rows during DeepSeek V4.1 decoder replay ( #61643 )
...
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-15 14:02:06 +00:00
24f632ca7e
[Bugfix] Engage indexer TP row sharding on total rows, not per-rank rows ( #61647 )
...
Signed-off-by: Zheng Cai <caizheng1993@gmail.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
2026-10-15 13:54:14 +00:00
mosafariuk
69425846f8
[Bugfix][CPU] Report the real cause when the worker outgrows its memory reservation ( #59472 )
...
Signed-off-by: Mo <mosafariuk@gmail.com >
2026-10-15 21:52:05 +08:00
b7b20b1b8f
[Bugfix][Frontend] Leave the final stop token out of chat derender text ( #61757 )
...
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-15 21:51:34 +08:00
littlecircle0730 and Shanshan Shen
71536bd5f8
[MM][CG] Support ViT full CUDA graph for DeepSeek-VL2 ( #46005 )
...
Signed-off-by: littlecircle0730 <littlecircle0730@gmail.com >
Signed-off-by: littlecircle0730 <43994952+littlecircle0730@users.noreply.github.com >
Co-authored-by: Shanshan Shen <467638484@qq.com >
2026-10-15 21:38:52 +08:00
0951fc183c
[ROCm][DSv4.1] Disambiguate ROCm attention class names ( #58689 )
...
Signed-off-by: Tuukka Sarvi <tuukka.sarvi@amd.com >
Co-authored-by: Shanshan Shen <467638484@qq.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 21:38:39 +08:00
1463501553
[Bugfix] Terminate streaming sessions on ABORT/ERROR and prevent zombie sessions ( #58942 )
...
Signed-off-by: David Cheung <d7cheung@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Benedikt Falk <bfalk@nvidia.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-10-15 13:32:19 +00:00
Xunzhuo
766c837bd5
[Model] Support every Decision 3.0 size (d3-flash, d3-mini, d3-nano, d3-lite, d3-edge) ( #61778 )
...
Signed-off-by: Xunzhuo Liu <xunzhuo.liu@amd.com >
2026-10-15 13:29:24 +00:00
Laith Sakka and Nick Hill
9cede34a67
[Bugfix] Create cuBLAS handles before memory profiling to avoid OOM ( #61044 )
...
Signed-off-by: Laith Sakka <lsakka@meta.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-10-15 08:59:22 -04:00
Martin Hickey
252be28c18
[Frontend] Replace return_mm_kwargs with an mm_payload enum on render requests ( #61427 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-10-15 12:41:46 +00:00
Martin Hickey
c6b772cf6b
[Bugfix][Frontend] Emit kv/ec transfer params at the top level of render output ( #61241 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-10-15 12:34:15 +00:00
Micah Williamson
e8e1e96196
[ROCm][DSv4] AITER MegaMoE Fixed-Slot Dispatch For DeepSeek-V4 ( #61262 )
...
Signed-off-by: Micah Williamson <micah.williamson@amd.com >
2026-10-15 19:10:59 +08:00
8b27d27402
[Bugfix][KV Offload] Always attempt NIXL agent removal during peer teardown ( #60279 )
...
Signed-off-by: nej1gotnochill <divyanshlalotra2@gmail.com >
Co-authored-by: Codebuff <noreply@codebuff.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 14:07:23 +03:00
Talor Abramovich
81e42c9987
[Feature][Frontend] Control the log level of request JSON body using env var ( #60630 )
...
Signed-off-by: talora <talora@nvidia.com >
2026-10-15 11:07:19 +00:00
jimmy-adams
365aaa77bd
[Bugfix][DSv4] Remap Quark MTP layer_quant_config onto draft prefixes ( #56054 )
...
Signed-off-by: jimmy-adams <41593649+jimmy-adams@users.noreply.github.com >
2026-10-15 18:51:57 +08:00
e10c10a644
[Core] Add PROMOTION_LATENCY histogram metric for tiering offload ( #53910 )
...
Signed-off-by: Wen Zhou <wenzhou@redhat.com >
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-10-15 13:47:41 +03:00
Pei Li and Kevin H. Luu
6fe7401fac
[Perf] Read VLLM_COMPUTE_NANS_IN_LOGITS once per IterationStats ( #61710 )
...
Signed-off-by: Pei Li <peili@nvidia.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 18:35:54 +08:00
Timofey and Codex
716c5225ae
[Model] Add AliceAI support ( #57963 )
...
Signed-off-by: timka <timka.byzov@yandex.ru >
Co-authored-by: Codex <noreply@openai.com >
2026-10-15 10:15:03 +00:00
7020600b1d
[KV Connector][NIXL] Register packed MLA KV rows as a single region ( #61074 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
2026-10-15 09:18:15 +00:00
Flora Feng and Chauncey
80dd705c1d
[Frontend] Make finish_reason and Responses item status truthful ( #59324 )
...
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-10-15 10:12:02 +01:00
b965ee564d
[Chore] Use upstream Transformers DeepseekV4Config ( #61392 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 01:08:08 -07:00
Misha Goin and Claude Opus 5.5
1c4c319792
[Frontend] Advertise --enable-x/--disable-x for boolean CLI flags ( #61479 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 01:08:02 -07:00
48f17b77bb
[Bugfix] Reject mismatched float16/bfloat16 KV cache dtype ( #60885 )
...
Signed-off-by: cipheraxat <cipheraxat@users.noreply.github.com >
Co-authored-by: cipheraxat <cipheraxat@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 08:06:59 +00:00
fc8e08a2a6
[Bugfix][KV Offload] Do not let mark_miss() flip a pending or in-flight async lookup probe ( #59096 )
...
Signed-off-by: Sohom Chakraborty <16609933+sohom-cs@users.noreply.github.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 10:57:46 +03:00
Bugen Zhao and Claude Opus 5.5
8fa5ae844a
[Frontend] Let an unset --tool-strict-level fall back to the tool parser's default ( #61328 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 15:48:38 +08:00
91ed2859c7
[CPU][Zen] Update zentorch to 2.14.0.1 & enable float16 on Venice and later ( #61411 )
...
Signed-off-by: Ganesh R <ganesh.r@amd.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-15 00:23:50 -07:00
MOVZXandHinata