iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
23,341 Commits 3 Branches 0 Tags
patches
Commit Graph
23341 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
David CheungBenedikt FalkNick Hill
 		957d512e96 [Bugfix][Frontend] Preserve streaming chunk boundaries in RequestOutputCollector (#58941) 
Signed-off-by: David Cheung <d7cheung@gmail.com>
Co-authored-by: Benedikt Falk <bfalk@nvidia.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-10-15 16:38:28 +00:00
Jovan a9e90b9d1d [Bugfix][Frontend] Reject Cohere REQUIRED tool choice without tools (#61825) 
Signed-off-by: Kevin <jia__kuan@163.com>
 2026-10-15 16:37:08 +00:00
Oxana KorzhClaude Opus 5.5Andreas Karatzas
 		d6f0b7a416 [CI] Skip exited descendants when finding the worker in the FT e2e test (#61648) 
Signed-off-by: Oxana Korzh <okorzh@amd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-10-15 16:08:14 +00:00
fululi12Cursor AgentlifuluClaude Opus 5
 		ed3050add4 [Core] Apply prefill_schedule_interval outside data parallelism (#54627) 
Signed-off-by: lifulu <fululi12@amd.com>
Signed-off-by: lifulu <lifulu@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: lifulu <lifulu@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-10-15 11:03:13 -05:00
fululi12Claude Opus 5lifulu
 		c825e1d235 [Perf] Fuse the mamba align block-table gather into one Triton kernel, 7 kernels -> 1, 6x speedup on all shapes for this kernel (#58737) 
Signed-off-by: lifulu <fululi12@amd.com>
Signed-off-by: lifulu <lifulu@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: lifulu <lifulu@users.noreply.github.com>
 2026-10-15 11:02:53 -05:00
MOVZXandHinata 5d6754ff6b fix: keep the refusal-direction projection dynamo-safe 
With VLLM_ABLITERATION set, boot crashed: dynamo traces
abliteration.register from Qwen3NextGatedDeltaNet.__init__, and
buf.copy_(r) fails on the meta device during the compile-time model
construction pass. Disable tracing for register and guard the copy so
an unmaterializable buffer turns projection off with a warning.

Log activation at registration instead of inside project_: project_
runs in the compiled region, and a logger call there aborts dynamo.
The width guard uses the same escape hatch.

Also project the embedding that Qwen3NextMultiTokenPredictor builds
for itself, which the first pass missed, and add tests for the
2560-value Flash-Next direction against a 5120 model and for a
half-width activation.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-15 22:56:06 +07:00
CharlesWuandYichao Zhu a97d19d792 [UMBP][1/10] Add shared connector, runtime contracts, and request lifecycle (#61789) 
Signed-off-by: CharlesWu <Charles.Wu@amd.com>
Co-authored-by: Yichao Zhu <Yichao.Zhu@amd.com>
 2026-10-15 15:12:39 +00:00
Yin LiYin LiCyrus Leung
 		3dcfaeb4e8 [Frontend] Apply each endpoint's top_logprobs cut in derender (#59518) 
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com>
Co-authored-by: Yin Li <kevinli.ai.work@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-10-15 15:04:04 +00:00
MOVZXandHinata 0076adc78b [Qwen3.5] Project the refusal direction out of the residual stream at runtime 
orcarouter/Qwen3.8-27B-Uncensored removes refusal by editing weights. The
Swift release we serve is AutoRound W8A16, so re-editing weights means
re-quantizing. Project one unit direction r out of every residual-stream
writer output instead: x <- x - r (r . x), which equals the weight edit
W' = W - r(r^T W) on every writer feeding that stream (Arditi et al. 2024).

Sites: attention output for both the full-attention and GDN branches, the
MLP output, and token embeddings, in Qwen3NextDecoderLayer.forward and
Qwen3NextModel.embed_input_ids, which Qwen3_5DecoderLayer and Qwen3_5Model
inherit but whose base constructors they skip, so both register there too.
The MTP drafter reuses Qwen3_5DecoderLayer and calls its own
embed_input_ids, so the draft path is covered at the same width, and
_maybe_share_embeddings at PP=1 hands the draft the target embedding
module. Sequence parallelism shards dim 0 only, so the last dim is
hidden_size at every site.

VLLM_ABLITERATION names a JSON file with key refusal_direction, a
safetensors tensor, or a bare name resolved under
/opt/qwen38/abliteration then /data/abliteration. Empty or 0 is off.
register() always creates the buffer _abliteration_r, zeros when off, so
one compiled graph serves ON and OFF and toggling the switch
(declared in envs.py, hence in compile_factors) invalidates the compile
cache. project_() is an in-place rank-1 update with no full-size
temporary, and skips tensors whose width is not hidden_size.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-15 21:40:16 +07:00
Matthias Gehre 011fa68ca9 [ROCm][Perf] Pad fp16/bf16 GEMM weight row stride for L2 cache (#55090) 
Signed-off-by: Matthias Gehre <matthias.gehre@amd.com>
 2026-10-15 14:26:32 +00:00
MOVZX 76df77c04e Merge branch 'main' into patches 
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-15 21:16:39 +07:00
Ronaldandaoshen02 d12d5460a5 [Bugfix][MoE] Keep routing rows during DeepSeek V4.1 decoder replay (#61643) 
Signed-off-by: Ronald1995 <ronaldautomobile@163.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
 2026-10-15 14:02:06 +00:00
Zheng CaiClaude Opus 4.7zjy0516
 		24f632ca7e [Bugfix] Engage indexer TP row sharding on total rows, not per-rank rows (#61647) 
Signed-off-by: Zheng Cai <caizheng1993@gmail.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
 2026-10-15 13:54:14 +00:00
mosafariuk 69425846f8 [Bugfix][CPU] Report the real cause when the worker outgrows its memory reservation (#59472) 
Signed-off-by: Mo <mosafariuk@gmail.com>
 2026-10-15 21:52:05 +08:00
Kevin H. LuuClaude Opus 5.5Cyrus Leung
 		b7b20b1b8f [Bugfix][Frontend] Leave the final stop token out of chat derender text (#61757) 
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-10-15 21:51:34 +08:00
littlecircle0730andShanshan Shen 71536bd5f8 [MM][CG] Support ViT full CUDA graph for DeepSeek-VL2 (#46005) 
Signed-off-by: littlecircle0730 <littlecircle0730@gmail.com>
Signed-off-by: littlecircle0730 <43994952+littlecircle0730@users.noreply.github.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>
 2026-10-15 21:38:52 +08:00
Tuukka SarviShanshan ShenKevin H. Luu
 		0951fc183c [ROCm][DSv4.1] Disambiguate ROCm attention class names (#58689) 
Signed-off-by: Tuukka Sarvi <tuukka.sarvi@amd.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
 2026-10-15 21:38:39 +08:00
David CheungBenedikt FalkNick Hill
 		1463501553 [Bugfix] Terminate streaming sessions on ABORT/ERROR and prevent zombie sessions (#58942) 
Signed-off-by: David Cheung <d7cheung@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Benedikt Falk <bfalk@nvidia.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-10-15 13:32:19 +00:00
Xunzhuo 766c837bd5 [Model] Support every Decision 3.0 size (d3-flash, d3-mini, d3-nano, d3-lite, d3-edge) (#61778) 
Signed-off-by: Xunzhuo Liu <xunzhuo.liu@amd.com>
 2026-10-15 13:29:24 +00:00
Laith SakkaandNick Hill 9cede34a67 [Bugfix] Create cuBLAS handles before memory profiling to avoid OOM (#61044) 
Signed-off-by: Laith Sakka <lsakka@meta.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-10-15 08:59:22 -04:00
Martin Hickey 252be28c18 [Frontend] Replace return_mm_kwargs with an mm_payload enum on render requests (#61427) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-10-15 12:41:46 +00:00
Martin Hickey c6b772cf6b [Bugfix][Frontend] Emit kv/ec transfer params at the top level of render output (#61241) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-10-15 12:34:15 +00:00
Micah Williamson e8e1e96196 [ROCm][DSv4] AITER MegaMoE Fixed-Slot Dispatch For DeepSeek-V4 (#61262) 
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
 2026-10-15 19:10:59 +08:00
Divyansh LalotraCodebuffKevin H. Luu
 		8b27d27402 [Bugfix][KV Offload] Always attempt NIXL agent removal during peer teardown (#60279) 
Signed-off-by: nej1gotnochill <divyanshlalotra2@gmail.com>
Co-authored-by: Codebuff <noreply@codebuff.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
 2026-10-15 14:07:23 +03:00
Talor Abramovich 81e42c9987 [Feature][Frontend] Control the log level of request JSON body using env var (#60630) 
Signed-off-by: talora <talora@nvidia.com>
 2026-10-15 11:07:19 +00:00
jimmy-adams 365aaa77bd [Bugfix][DSv4] Remap Quark MTP layer_quant_config onto draft prefixes (#56054) 
Signed-off-by: jimmy-adams <41593649+jimmy-adams@users.noreply.github.com>
 2026-10-15 18:51:57 +08:00
Wen ZhouClaude Opus 4.6OpenAI CodexKevin H. LuuOr Ozeri
 		e10c10a644 [Core] Add PROMOTION_LATENCY histogram metric for tiering offload (#53910) 
Signed-off-by: Wen Zhou <wenzhou@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-10-15 13:47:41 +03:00
Pei LiandKevin H. Luu 6fe7401fac [Perf] Read VLLM_COMPUTE_NANS_IN_LOGITS once per IterationStats (#61710) 
Signed-off-by: Pei Li <peili@nvidia.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
 2026-10-15 18:35:54 +08:00
TimofeyandCodex 716c5225ae [Model] Add AliceAI support (#57963) 
Signed-off-by: timka <timka.byzov@yandex.ru>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-15 10:15:03 +00:00
Nicolò LucchesiClaude Opus 5.5Yifan QiaoCodex
 		7020600b1d [KV Connector][NIXL] Register packed MLA KV rows as a single region (#61074) 
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-15 09:18:15 +00:00
Flora FengandChauncey 80dd705c1d [Frontend] Make finish_reason and Responses item status truthful (#59324) 
Signed-off-by: Flora Feng <4florafeng@gmail.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
 2026-10-15 10:12:02 +01:00
Denis KhachyanandClaude Code 427915bed2 [Build][CPU] Install Run:ai Model Streamer in vllm-openai-cpu image (#61382) 
Signed-off-by: Denis Khachyan <khachyanda@gmail.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
 2026-10-15 09:11:51 +00:00
Kevin H. LuuandClaude Opus 5.5 7f730b4cea [CI] Run 4xH100 steps nightly only (#61782) 
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-15 01:32:43 -07:00
wang.yuqi 0f0bc4dbb1 [Test] Split the chat_utils tests. (#61765) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-10-15 08:26:22 +00:00
Harry MellorClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Kevin H. Luu
 		b965ee564d [Chore] Use upstream Transformers DeepseekV4Config (#61392) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
 2026-10-15 01:08:08 -07:00
Misha GoinandClaude Opus 5.5 1c4c319792 [Frontend] Advertise --enable-x/--disable-x for boolean CLI flags (#61479) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-15 01:08:02 -07:00
Akshat AnandcipheraxatClaude Opus 5.5
 		48f17b77bb [Bugfix] Reject mismatched float16/bfloat16 KV cache dtype (#60885) 
Signed-off-by: cipheraxat <cipheraxat@users.noreply.github.com>
Co-authored-by: cipheraxat <cipheraxat@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-15 08:06:59 +00:00
Sohom chakrabortyClaudeKevin H. Luu
 		fc8e08a2a6 [Bugfix][KV Offload] Do not let mark_miss() flip a pending or in-flight async lookup probe (#59096) 
Signed-off-by: Sohom Chakraborty <16609933+sohom-cs@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
 2026-10-15 10:57:46 +03:00
Bugen ZhaoandClaude Opus 5.5 8fa5ae844a [Frontend] Let an unset --tool-strict-level fall back to the tool parser's default (#61328) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-15 15:48:38 +08:00
Ganesh RKevin H. LuuCyrus Leung
 		91ed2859c7 [CPU][Zen] Update zentorch to 2.14.0.1 & enable float16 on Venice and later (#61411) 
Signed-off-by: Ganesh R <ganesh.r@amd.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-10-15 00:23:50 -07:00
Chaojun ZhangKevin H. LuuCyrus Leung
 		85c94c1706 [Tests][XPU] Skip modelopt_fp4 online MoE re-quantization test on XPU (#61723) 
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-10-15 00:23:45 -07:00
Kevin H. LuuandClaude Opus 5.5 8f111b22ee [CI] Run 4xB200 steps nightly only (#61541) 
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-15 00:08:48 -07:00
Kevin H. LuuandClaude Opus 5.5 32feecd91f [CI] Pin DeepSeek-V4.1-Flash-NVFP4 eval to its FP8-Engram revision (#61763) 
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-15 00:05:18 -07:00
Chaojun ZhangandKevin H. Luu ab593678b5 [CI][XPU] Ignore test_reload.py on Intel CI (#61740) 
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
 2026-10-15 15:03:34 +08:00
danielafrimiOpenAI CodexDaniel Afrimitomeras91
 		a9fe30e9f6 [Bugfix][Multimodal] Recover PyNvVideoCodec decoder slots after failures (#59597) 
Signed-off-by: Daniel Afrimi <dafrimi@nvidia.com>
Signed-off-by: danielafrimi <45691845+danielafrimi@users.noreply.github.com>
Signed-off-by: Daniel Afrimi <dafrimi@oci-hsg-cs-001-login-02.cm.cluster>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Daniel Afrimi <dafrimi@oci-hsg-cs-001-login-02.cm.cluster>
Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com>
 2026-10-15 09:17:19 +03:00
liranschourandKevin H. Luu c5f8adc0b1 [KV Offload] Expose OffloadingManager.lock()/unlock() and service the tiering control plane off the scheduler thread (#58168) 
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
 2026-10-15 08:50:56 +03:00
HarshilandCyrus Leung c54dca96bd [Bugfix][Benchmark] Fix probe latency and duration in bench serve (#60511) 
Signed-off-by: QHarshil <harshil_c@hotmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-10-15 05:50:30 +00:00
Itay EtelisRobert ShawCodexKevin H. LuuOr Ozeri
 		5341b5ba7d [KV Offload] Pin only each worker's own offload slots (#59544) 
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-10-15 08:49:42 +03:00
StaryCursorJiangyun Zhu
 		6605802904 [Perf][KV Connector][Mooncake] Send the KV layout once per peer (#59825) 
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
 2026-10-15 05:36:31 +00:00
Kevin H. LuuOpenAI CodexClaude Opus 5.5
 		a400804ce4 [CI] Relax MTP sharded sampling cross-boot noise bound (#57026) 
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-14 22:31:45 -07:00