MOVZX
49d708e158
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-18 18:09:11 +00:00
MOVZX and Hinata
6cf86cd95f
[Bugfix] Load checkpoint KV-cache scales in the Qwen3.5 family
...
Checkpoints that ship calibrated k_scale/v_scale tensors (e.g.
unsloth/Qwen3.8-27B-NVFP4, which carries one pair per full-attention
layer) name them ...self_attn.{k,v}_scale, but the runtime parameters
live on the inner attention layer (...self_attn.attn.{k,v}_scale).
Qwen3_5Model.load_weights hands the iterable straight to
AutoWeightsLoader, which performs no scale-name remapping, so the
scales are silently dropped and --kv-cache-dtype fp8 runs at scale 1.0
with only a warning_once that fires even on success (the MTP draft
layer legitimately has no scales).
Route scale-suffixed names through maybe_remap_kv_scale_name first,
mirroring gemma4/gpt_oss. No behavior change for checkpoints without
scales.
Measured on RTX 5090 / Qwen3.8-27B NVFP4 / fp8 KV / MTP k=3 (GSM8K
greedy, n=300): 92.3% at scale 1.0 vs 93.7% with scales loaded
(bf16-KV control 94.5%).
Fixes #54623
Co-Authored-By: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-18 17:18:47 +00:00
alp and mottopanikeiku
ff01a6b14f
[Bugfix][Frontend] Stop beam search on generation config EOS tokens ( #62072 )
...
pre-commit / pre-run-check (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Signed-off-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com >
Co-authored-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com >
2026-10-18 16:49:42 +00:00
e80b6c642f
[ROCm][DSv4] Overlap the unfused shared expert with AITER MegaMoE ( #61819 )
...
Signed-off-by: Taoyu Zhu <zhutaoyu97@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: vllm-agent <claw@inferact.ai >
2026-10-18 22:39:38 +08:00
Harry Huang and Thien Tran
494663940b
[Qwen4Exp] Clean up HC and PLE implementations for Qwen3.8-Flash-Next ( #61454 )
...
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-10-18 19:38:21 +08:00
Rémi Delhaye
3c13653738
[Frontend] Score with the chat template declared by Sentence Transformers CrossEncoders ( #60877 )
...
Signed-off-by: rdlhio <contact@rdlh.io >
2026-10-18 11:30:12 +00:00
Isotr0py
3f1b04e045
[Bugfix] Fix Opencv video backend frames count missing for WebM video inputs ( #61759 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
2026-10-18 10:37:39 +00:00
MOVZX
c5ab42bc0f
Merge branch 'main' into patches
2026-10-18 16:04:36 +07:00
7c9042c05a
[Bugfix][MRV2] Fix speculative warmup deadlock with pipeline parallelism ( #62129 )
...
Signed-off-by: chenjr97 <chenjr97@mail2.sysu.edu.cn >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: vllm-agent <claw@inferact.ai >
2026-10-17 23:51:32 -07:00
b2456c622b
[ROCm][CI] Make ROCm 10.0 the default image; keep ROCm 7.2 as -rocm72 ( #58761 )
...
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com >
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com >
Signed-off-by: Micah Williamson <micah.williamson@amd.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Micah Williamson <micah.williamson@amd.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-10-17 21:43:59 -05:00
djramic and Andreas Karatzas
9c778e9ff8
[Bugfix][EPLB] Wait for pending expert buffer copies before NIXL READs ( #62081 )
...
Signed-off-by: Djordje Ramic <djoramic@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-10-17 18:18:02 -05:00
7879f3687b
[NIXL] Drop unreachable producer-side notif counting in push mode ( #62210 )
...
Signed-off-by: David Cheung <d7cheung@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: vllm-agent <claw@inferact.ai >
2026-10-17 21:24:35 +00:00
03e8300d2c
[Bugfix] Enable inductor batch-invariant mode under VLLM_BATCH_INVARIANT ( #61118 )
...
Signed-off-by: Yifan Feng <evanfengff@gmail.com >
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-17 14:53:06 -04:00
Afonso Ribeiro and Flora Feng
5c1c903501
[Perf][Parser] Stream Qwen3 tool argument values incrementally (12 KB call: 2.1 s to 11 ms) ( #62027 )
...
Signed-off-by: Afonso Ribeiro <afonso.diamantino2@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-17 18:49:06 +00:00
Tyrone
bafb263664
[Bugfix][Benchmarks] Name the real cause when the /tokenize alignment probe fails ( #59888 )
2026-10-17 14:47:52 -04:00
Nick Hill and Claude Opus 5.5
0c5dd18e7a
[MRV2] Collect KV transfer results before the drafter runs ( #62041 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-17 10:27:57 -07:00
763e3f7c91
[Bugfix][Spec Decode] Disable dynamic speculative decoding for fixed-K proposers instead of crashing ( #46399 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-10-17 12:41:04 -04:00
c0b6f74bee
[Benchmark] Add per-case kernel profiles to vllm bench forward ( #61692 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: vllm-agent <claw@inferact.ai >
2026-10-17 08:27:11 -07:00
siyu and Xun Sun
c61a1e73c6
[Feature] Skip MLA weight absorption when weights are pre-processed ( #57320 )
...
Co-authored-by: Xun Sun <UNIDY2002@outlook.com >
2026-10-17 23:10:04 +08:00
837e87e1c5
[Feature][LoRA][MoE] Expert LoRA for FlashInfer TRTLLM MXFP4 experts ( #61979 )
...
Signed-off-by: aoshen524 <aoshen524@gmail.com >
Co-authored-by: aoshen524 <aoshen524@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-17 20:17:56 +08:00
liuzhenwei and Kunshang Ji
4e964b2623
[XPU] Fix Event init failure w/ external ( #61743 )
...
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-10-17 19:10:39 +08:00
Vadim Gimpelson and Claude
ec8b20fbf0
[Bugfix] Fix Gemma 4 encoder CUDA graph capture OOM at startup ( #61365 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-17 11:04:49 +00:00
2b85d9cb65
[Perf][Bugfix] Prefer MNNVL for single-node FlashInfer allreduce again ( #61989 )
...
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-10-17 02:55:28 -07:00
026f6ab268
[Model] DeepSeek-V4/V4.1 VL: stream language-model weights instead of sorting the checkpoint ( #58857 )
...
Signed-off-by: git bisector <gitbisector@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-10-17 09:37:56 +00:00
f98ae6a9fd
[Bugfix][Core] Keep resumable sessions that stop while preempted ( #58778 )
...
Signed-off-by: David Cheung <d7cheung@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Yifan Zong <yzong@redhat.com >
2026-10-17 09:08:14 +00:00
2067297b2f
[ROCm][Perf] Use AITER tuned configs to select a4w4 gemm kernel instead of env var ( #61278 )
...
Signed-off-by: Andy Friedrich <afriedri@amd.com >
Signed-off-by: afriedri <afriedri@amd.com >
Co-authored-by: Simon Danielsson <70206058+simondanielsson@users.noreply.github.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-10-17 04:00:28 -05:00
Jee Jee Li
e56b1603af
[Kernel]Add norm fused kernel for Step-5 Preview and Muse Spark Open ( #62132 )
...
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
2026-10-17 01:53:00 -07:00
Ting SUN
eb8de8d1a1
[Bugfix][OpenAI] Respect allowed_token_ids in beam search ( #47630 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
2026-10-17 08:50:40 +00:00
dskwe
b05ede5616
[Bugfix][Frontend] Preserve token IDs in completion derender responses ( #62114 )
...
Signed-off-by: Yuan Li <dskwelmcy@163.com >
2026-10-17 08:27:48 +00:00
61bb590877
[Bugfix][Spec Decode] Clear stale shared block-table rows ( #61174 )
...
Signed-off-by: Cheng Rui <286040359@qq.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com >
2026-10-17 01:25:02 -07:00
c3de0b37e2
[Core][Spec Decode] Replace EAGLE group heuristics with explicit draft-layer tags ( #61455 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-17 08:22:33 +00:00
MOVZX
e0fe5df731
Merge branch 'main' into patches
2026-10-17 08:06:56 +00:00
znzryb and Woosuk Kwon
ad2688ab30
[Bugfix] Validate draft-token capacity in forward benchmark ( #61809 )
...
Signed-off-by: znzryb <znzryb34@gmail.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
2026-10-17 00:51:36 -07:00
Jovan and OpenAI Codex
5be4266f5c
[Bugfix][Frontend] Preserve Cohere and Anthropic streaming errors ( #61947 )
...
Signed-off-by: Kevin <jia__kuan@163.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-10-17 06:10:16 +00:00
Wei Zhao
90d6d871d6
[Attention] Add CuTe DSL backend selection for MLA context prefill ( #62069 )
...
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
2026-10-17 01:21:55 -04:00
Cy Cheng and Cy
d74f4180cd
[Bugfix][Frontend] Detokenize echoed prompts after truncation ( #61876 )
...
Signed-off-by: Cy <15118312+cycancode@users.noreply.github.com >
Co-authored-by: Cy <15118312+cycancode@users.noreply.github.com >
2026-10-17 05:20:38 +00:00
Jee Jee Li
39983a5e7a
[Kernel] Add FP32 router GEMM support for Step-5 Preview and Muse Spark Open ( #62126 )
...
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
2026-10-16 22:17:55 -07:00
41b8c15809
[Feature][MoRIIO] Support hybrid WRITE transfers with DSpark ( #60544 )
...
Signed-off-by: Yichao Zhu <Yichao.Zhu@amd.com >
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-17 04:57:58 +00:00
Micah Williamson and Cursor
a0e7137c4b
[CI][ROCm][Bugfix] DeepSeek V4.1: don't pass expert intermediate padding to AITER ( #61960 )
...
Signed-off-by: Micah Williamson <micah.williamson@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-10-17 04:25:39 +00:00
7b6310c109
[Perf][Sampler] Speed up the small-batch split top-p pipeline ( #61996 )
...
Signed-off-by: Bowen Wang <bowen@inferact.ai >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: vllm-agent <claw@inferact.ai >
2026-10-16 21:22:11 -07:00
Fadi Arafeh and vllm-agent
556ac15eb9
[Bugfix] Remove extra args from CPU's compute slot mapping kernel ( #61992 )
...
Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com >
Co-authored-by: vllm-agent <claw@inferact.ai >
2026-10-17 12:11:33 +08:00
Gabriel Wu and Claude Opus 5.5
dcd07967ca
[CPU] Keep the hybrid block size across the engine core's config rerun ( #61805 )
...
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-17 12:06:03 +08:00
ce49c47cbb
[Bugfix][Sampler] Carry the boundary tie count across blocks in the split top-p mask kernel ( #62046 )
...
Signed-off-by: cipheraxat <cipheraxat@users.noreply.github.com >
Co-authored-by: cipheraxat <cipheraxat@users.noreply.github.com >
Co-authored-by: Bowen Wang <bowen@inferact.ai >
2026-10-17 02:26:57 +00:00
Sherif Waly and vllm-agent
ce8853d27a
[Bugfix][KV Offload] Allow explicit SimpleCPUOffload block counts ( #61674 )
...
Signed-off-by: Sherif Waly <sherif.waly@mistral.ai >
Co-authored-by: vllm-agent <claw@inferact.ai >
2026-10-17 02:11:12 +00:00
71c20f73f2
[FlashInfer] Migrate autotune warmup to flashinfer.autotune_v2 managed cache ( #55671 )
...
Signed-off-by: Yang Xu <yanxu@nvidia.com >
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
Signed-off-by: YangXu <yanxu@nvidia.com >
Co-authored-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
Co-authored-by: Kimi Code <noreply@moonshot.cn >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
Co-authored-by: vllm-agent <claw@inferact.ai >
2026-10-17 02:08:28 +00:00
Shuolei Wang and aoshen02
b3a7e2ff6e
[RL][Frontend] Return weight-version spans on OpenAI completions and chat ( #61347 )
...
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-17 02:08:07 +00:00
6b6a6c5c98
[RL] Support per-parameter source layouts for NCCL M2N ( #60934 )
...
Signed-off-by: pkousha <pkousha@nvidia.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-16 21:41:17 -04:00
Harry Mellor and Claude Opus 5.5
f8112704d5
[Chore] Use upstream Transformers InklingProcessor ( #61883 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-17 01:04:54 +00:00
f019e3e105
[ROCm] attn res + sigmoid_mul + conv fusions ( #52968 )
...
Signed-off-by: Omar Muhammad <omar.muhammad@amd.com >
Signed-off-by: Nicholas Susanto <nicholas.susanto@amd.com >
Co-authored-by: Nicholas Susanto <nicholas.susanto@amd.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-17 00:54:54 +00:00
Vadim Gimpelson and Claude
783380af39
[Bugfix] Fix NaN output of the Mamba2 chunk scan kernel on sm90 ( #61718 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-16 17:24:51 -07:00
MOVZX
