Commit Graph
14744 Commits
Author SHA1 Message Date
hcl b5707bf994 [Bugfix][MoE] Enable cuBLAS out_dtype router GEMM on all CUDA archs (fixes family-120/GB10) (#54048)
Signed-off-by: Chenglun Hu <chenglunhu@gmail.com>
2026-08-29 19:18:42 -07:00
yzong-rh 680e2177e4 [CI][Ray] Fix flaky multi-node assignment test after placement-group teardown (#53621) 2026-08-29 23:48:22 +00:00
Giancarlo Delfin fe755c8899 [Bugfix][Model Runner V2][Spec Decode] Decouple the draft's gumbel noise stream from the target's (#54282)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
2026-08-29 09:58:29 -07:00
Kaif KohariandCyrus Leung 7fbfca2670 [Bugfix][Frontend] Only echo the assistant turn in batched chat completions (#52529)
Signed-off-by: Kaif <kaifkohari10@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-08-29 09:44:17 -07:00
jackandQwertyJack 5b0e5b69ac [Bugfix][Frontend] Validate stop_token_ids against vocab size (#54196)
Signed-off-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com>
Co-authored-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com>
2026-08-29 16:25:35 +00:00
Shenglei FuandClaude Fable 5 129087ddab [Perf] Reuse topk SparseMatrix routing metadata in GPT-OSS MoE forward (#45457)
Signed-off-by: Shenglei Fu <sfu@confluent.io>
Signed-off-by: Shenglei Fu <117230642+ShengleiFu@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-29 12:14:39 -04:00
Nicolò Lucchesi 7f4793eaa3 [Nixl][PD] DCP support for MLA models (#50611)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-08-29 16:03:53 +00:00
Isotr0py 4fc943b867 [Multimodal] Deprecate PyAV video decoder backend (#54231)
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
2026-08-29 15:13:28 +00:00
Taneem Ibrahim fd5d3aea94 [Mypy] Fix typing for J models (#54130)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
2026-08-29 08:50:15 -04:00
Rahul ChalamalaJanelle CaiDevin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin
d3d79ffc1e [Bugfix][Spec Decode] Capture the widest uniform decode batch by default (#50488)
Signed-off-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com>
Signed-off-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Signed-off-by: Rahul Chalamala <rahul@modal.com>
Co-authored-by: Janelle Cai <janelle.cai@modal.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai>
2026-08-28 22:38:13 -07:00
a758a9f671 [Perf][KV Connector] Pin token_indices before non_blocking H2D in hf3fs helper (#54293)
Signed-off-by: Keyi Li <likey6688@gmail.com>
Co-authored-by: Keyi Li <likey6688@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-08-28 22:20:17 -07:00
Oxana KorzhandClaude Opus 5 fb68025138 [Bugfix][MRV2] Release model and KV cache on in-process engine shutdown (#54162)
Signed-off-by: Oxana Korzh <okorzh@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 22:09:08 -07:00
Hotragn Pettugani 3958a420f0 [Bugfix][V1] Keep an encoder cache entry until its last occurrence is freed (#54284)
Signed-off-by: hotragn <hotragn.pettugani_2024@woxsen.edu.in>
2026-08-29 05:08:01 +00:00
Summer Yang 6d4562c59b [Attention][DCP] Enable FlashInfer MLA for DSpark drafting (#54277)
Signed-off-by: Summer Yang <girasoleyang@gmail.com>
2026-08-28 21:11:28 -07:00
b2f685834a [Hy4] support Hy4-preview model (#54160)
Signed-off-by: chengvjiang <chengvjiang@tencent.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: chengvjiang <chengvjiang@tencent.com>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: russellfeng <russellfeng@tencent.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
2026-08-29 12:07:28 +08:00
Nick Hill 6c18a54648 [Perf] Avoid h2d copies from non-pinned CPU tensors (#54299)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-08-29 12:06:30 +08:00
kaiyuanxieandCalvinXKY 738bc8811a [BugFix] Disable TP for Qwen3-Omni audio encoder when heads % TP != 0 (#50858)
Signed-off-by: CalvinXKY <xky@users.noreply.github.com>
Co-authored-by: CalvinXKY <xky@users.noreply.github.com>
2026-08-29 11:18:04 +08:00
46a83642f6 [Perf] Pin CPU tensors before non_blocking H2D in three MM paths (#54292)
Signed-off-by: Keyi Li <likey6688@gmail.com>
Co-authored-by: Keyi Li <likey6688@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-08-28 20:13:40 -07:00
93ab92be0c [Bugfix][AMD] Annotate draft KV cache groups on the hybrid grouping path (#52047)
Signed-off-by: okorzh-amd <okorzh-amd@users.noreply.github.com>
Co-authored-by: okorzh-amd <okorzh-amd@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
2026-08-28 21:53:46 -05:00
6b110badbb [Bugfix][Mooncake] Save exact Mamba boundary states (#51358)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Dao Le <daole@inferact.ai>
2026-08-29 02:40:53 +00:00
Ronaldandaoshen02 16d6c376bc [Bugfix][Models] Register sleep-managed runtime buffers (#53507)
Signed-off-by: Ronald1995 <ronaldautomobile@163.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-08-29 10:32:24 +08:00
Wei ZhaoandYifan Qiao 7fd9cc036e [KV Connector] Support MooncakeStore with hybrid DCP prefix caching (#53324)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Signed-off-by: wzhao18 <weizha@nvidia.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
2026-08-29 02:10:23 +00:00
43196f2458 [Perf][MLA Sparse] Pin req_id_per_token before non_blocking H2D on XPU and ROCm (#54295)
Signed-off-by: Keyi Li <likey6688@gmail.com>
Co-authored-by: Keyi Li <likey6688@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-08-29 02:00:46 +00:00
99013d77d3 [Bugfix][Distributed] Gate cross-node MNNVL custom all-reduce by group capability (#53253)
Signed-off-by: JulianZJN <148066842+JulianZJN@users.noreply.github.com>
Signed-off-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Summer Yang <girasoleyang@gmail.com>
2026-08-29 08:49:24 +08:00
Xuan-1998andVladimir Aerov 026d5af7f4 [Bugfix] Fix ncclCommQueryProperties heap overflow with NCCL >= 2.31 (#53008)
Signed-off-by: Xuan Jiang <xuanj@amazon.com>
Co-authored-by: Vladimir Aerov <vaerov@amazon.com>
2026-08-29 00:25:00 +00:00
zzaebok 68c52b5a93 fix: improve token_ids_cpu swap to copy only valid indices (#36255)
Signed-off-by: Jaebok Lee <jaebok9541@naver.com>
2026-08-28 17:22:39 -07:00
Ace EldeibandCodex 9662ab0835 [Bugfix] Set breakable graph env before Ray actor import (#53293)
Signed-off-by: Ace Eldeib <aeldeib@coreweave.com>
Co-authored-by: Codex <codex@openai.com>
2026-08-28 23:11:50 +00:00
ukannikaykamisetmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
ae5b8e4a8d [ROCm][PERF] Enable AITER PA gluon decode for MiniMax-M3 MTP and dense layers (#52849)
Signed-off-by: ukannika <uma.kannikanti@amd.com>
Signed-off-by: ykamiset <yaminipreethi.kamisetty@amd.com>
Co-authored-by: ykamiset <yaminipreethi.kamisetty@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-28 22:54:39 +00:00
Shinichi Hemmi df14152ac6 [Model] Support speculative decoding method for PLaMo3 (#54239)
Signed-off-by: Shinichi Hemmi <shemmi@preferred.jp>
2026-08-28 22:31:07 +00:00
Charlie Fu fd98d32f73 [ROCm][CI] Fix test_ray_v2_executor (#54249)
Signed-off-by: charlifu <charlifu@amd.com>
2026-08-28 15:11:42 -07:00
c274d36103 [Bugfix] Keep the Moondream3 MoE all-reduce out of the fused-path try (#54152)
Signed-off-by: Oxana Korzh <okorzh@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
2026-08-28 15:06:42 -07:00
Summer YangandNick Hill 2aac565cae [Core][KV Connector] Start async KV loads after the forward launch when no sync loads are scheduled (#53333)
Signed-off-by: Summer Yang <girasoleyang@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-08-28 21:58:00 +00:00
Nick HillandKimi Code 96242aa50d [Bugfix][MRV2] Release layer-bound KV cache memory in shutdown() (#54246)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Kimi Code <noreply@moonshot.ai>
2026-08-28 21:27:09 +00:00
Nick HillandKimi Code 70732942c6 [Bugfix][ROCm] Pre-allocate wvSplitKrc static workspaces before KV init (#54247)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Kimi Code <noreply@moonshot.ai>
2026-08-28 14:22:53 -07:00
Taneem IbrahimandWentao Ye 21fa2c5a2a [Mypy] Fix mypy typing for model interfaces and H/I models (#54079)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-08-28 16:49:19 -04:00
67e86d1e6f [BugFix] Bind RayExecutorV2 TCPStore before publishing its port (#50969)
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-08-28 12:27:22 -07:00
afriedriandAndreas Karatzas 2f1cba799e [ROCm] remove VLLM_ROCM_USE_AITER_FP4_ASM_GEMM environment variable; make w4a4 use the preshuffle triton+asm by default (#53141)
Signed-off-by: Andy Friedrich <afriedri@amd.com>
Signed-off-by: afriedri <afriedri@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-08-28 18:59:53 +00:00
Yongye ZhuandOpenAI Codex d9dabfa351 [Kimi-K3][Kernel] Optimize the low-M fused latent MoE tail (#54168)
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-08-28 10:45:36 -07:00
Keyi LiandKeyi Li 6f91e3d953 [Perf] Split xdrope_positions H2D copy into per-row transfers (#53412)
Signed-off-by: Keyi Li <likey6688@gmail.com>
Co-authored-by: Keyi Li <likey6688@gmail.com>
2026-08-28 10:28:43 -07:00
Chipmunk ffe690eca9 [MM][CG] Support ViT full CUDA graph for Idefics3 and SmolVLM (#47625)
Signed-off-by: Ryouichi Kurihara <ryouichi.kurihara.628@gmail.com>
2026-08-28 17:16:31 +00:00
xuebwang-amd 06569a8696 [ROCm][Quantization][MOE] Enable fused shared experts for block-quantized FP8 (#53097)
Signed-off-by: xuebwang-amd <xuebwang@amd.com>
2026-08-28 09:23:07 -07:00
Michał Ganczarenko d1922cb5a7 [Bugfix] Restore multimodal support on the plain "vllm" throughput backend (#52168)
Signed-off-by: Michal Ganczarenko <michal.ganczarenko@intel.com>
2026-08-28 10:28:25 +00:00
Thien Tran f956e1c343 [Kimi-K3][Bugfix] Fix low-latency GEMM fallback initialization (#54167)
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-08-28 08:32:20 +00:00
stefankoncarevic 9236159bb6 [ROCm][CI] Keep startup profiling from aborting when free memory grows (#53591)
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
2026-08-28 00:02:40 -07:00
maobaolongandCodex 6f7df92a8e [CPU][MLA] Fix prefill backend selection so MLA runs end-to-end on CPU (#51471)
Signed-off-by: baoloongmao <baoloongmao@tencent.com>
Co-authored-by: Codex <codex@openai.com>
2026-08-28 12:36:55 +08:00
e6bfe03ad7 Count store offers, not lookups, for CPU offload store_threshold (#52227)
Signed-off-by: okorzh-amd <okorzh-amd@users.noreply.github.com>
Co-authored-by: okorzh-amd <okorzh-amd@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-27 20:21:06 -07:00
haic0andCursor 53bf990502 [Bugfix][KV Cache] Prevent negative external block allocation (#52707)
Signed-off-by: haic0 <149741444+haic0@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-27 20:10:45 -07:00
Roy WangandClaude Fable 5 b74ac0bc61 [Bugfix] Allocate packed outputs in fused_q_kv_rmsnorm so q_b_proj keeps its low-latency GEMM path at decode (#53109)
Signed-off-by: yasong.wang <yasong.wang@inferact.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-28 02:08:27 +00:00
Nick Hill 8cfba696ee [Misc] Separate adaptive verification config validation (#54108)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-08-27 20:03:01 -06:00
Thien Tran 6ec92bcbc8 [Kimi-K3] Merge MLA gate into QKV-A projection (#54015)
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-08-27 17:47:18 -07:00