6c95a641e9
[2/N][Feat][Perf] Add new warmup infrastructure for JITs. Add predicate filtering for JIT warmup, and migrate Inkling FA4 ( #49315 )
...
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Signed-off-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-11 07:46:07 -04:00
liushujia122
d8f840071e
[Model] Enable tower and connector LoRA for Keye ( #51780 )
...
Signed-off-by: shujialiu <liushujia0122@163.com >
2026-08-11 10:35:59 +00:00
AlexHuang
0fec3d652b
[Bugfix][KV Offload] Centralize shared mmap cleanup in CPU worker ( #51622 )
...
Signed-off-by: Alex <jihui.huang@daocloud.io >
2026-08-11 13:22:15 +03:00
Dao007forever and OpenAI Codex
ce07118669
[Bugfix][Core] Preserve Mamba running CoW after external hits ( #51766 )
...
Signed-off-by: Dao Le <Dao007forever@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-11 18:15:24 +08:00
David Holtz
490259c1f6
profiler: add PrivateUse1 activity support for custom backends ( #50977 )
...
Signed-off-by: David Holtz <56723830+dmholtz@users.noreply.github.com >
2026-08-11 09:38:58 +00:00
Tianyu Guo and OpenAI Codex
3fb7bb46d6
[Perf] Avoid repeated multimodal prompt update scans ( #51774 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-11 09:23:24 +00:00
zofia and Kunshang Ji
52c70b210c
[XPU] [Linear] enable torch linear backend for blockwise gemm on xpu ( #50826 )
...
Signed-off-by: Zhu, Zufang <zufang.zhu@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-08-11 16:43:33 +08:00
Kyungmin Lee and Harry Mellor
529d010351
[Doc] Fix typos in speculative decoding docs ( #51500 )
...
Signed-off-by: lkm2835 <lkm2835@gmail.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-08-11 09:17:05 +01:00
Yizhou
a311916a29
[MRV2][Spec] Fuse AR speculator multi-step decodes back into one CUDA graph ( #46849 )
...
Signed-off-by: Yizhou Liu <liu_yizhou@outlook.com >
2026-08-11 15:06:22 +08:00
Fangzhou Ai
1044d19998
[ROCm][DSV4] Preserve native MXFP4 TP8 shard allocation ( #51473 )
...
Signed-off-by: Fangzhou Ai <fangzhou.ai@amd.com >
2026-08-10 23:57:27 -07:00
Harry Mellor
be2e274ef5
Fix docs on main ( #51773 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-08-10 23:52:04 -07:00
Priyansh Jain and Andreas Karatzas
2acb055ed7
[CPU][Zen] Route BF16 MoE inference through zentorch on AMD ( #44201 )
...
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-08-10 23:20:05 -07:00
Yan Ma and Kunshang Ji
b76cf758db
[XPU] install xpu-manager for device monitor ( #50831 )
...
Signed-off-by: Yan Ma <yan.ma@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-08-11 14:18:41 +08:00
stefankoncarevic and Andreas Karatzas
75903e5067
[CI][AMD] Persist the openai-harmony tiktoken vocab cache across jobs ( #51666 )
...
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com >
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-08-11 06:11:06 +00:00
Wei Zhao
608c12473f
[Attention] Fix MLA prefill workspace allocation size ( #51733 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
2026-08-11 05:34:30 +00:00
c76a425278
[Kernel] Optimize long-context MLA cache gathers ( #51739 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-10 22:23:43 -07:00
cherry77-cloud and OpenAI Codex
65b7662d3f
[Bugfix][Frontend] Report Cohere stop sequences correctly ( #51556 )
...
Signed-off-by: cherry77-cloud <1615405@qq.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-11 04:49:49 +00:00
Khushali Desai and wang.yuqi
dc5101fb1b
replace batch_norm to numerically identical without cudnn ( #51734 )
...
Signed-off-by: khushali9 <khushali.desai9@gmail.com >
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-08-11 12:43:20 +08:00
Clinton Thomas and Lucas Bourtoule
90fd4a333f
Preserve revision pins in secondary artifact loaders ( #51446 )
...
Signed-off-by: Clinton Thomas <1033162+KernelClint@users.noreply.github.com >
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com >
2026-08-11 04:33:10 +00:00
Frank
adc1200597
[Misc] Use VLLMValidationError in scoring input validation ( #51753 )
...
Signed-off-by: frank-suwen <suwenw2@outlook.com >
2026-08-11 04:12:22 +00:00
Ming Huang
07443bea29
[BugFix][Core] free_blocks: restore prepend (LIFO) reuse order when prefix caching is off ( #51482 )
...
Signed-off-by: theminghuang <theminghuang@gmail.com >
Signed-off-by: Ming Huang <mhhua@google.com >
2026-08-11 03:56:25 +00:00
Wentao Ye
0914ed2e81
[Perf] Adaptive budget for spec scheduled input tokens, ~60% better Kimi K3 DSpark TTFT ( #51725 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-08-10 20:52:21 -07:00
Itay Etelis and Itay Etelis
0f2ea973be
[KV Connector][Offloading] Keep per-layer KV registration when canonical_layout is requested ( #51688 )
...
Signed-off-by: Itay Etelis <itay.etelis@ibm.com >
Co-authored-by: Itay Etelis <itay.etelis@ibm.com >
2026-08-11 06:03:59 +03:00
Andreas Karatzas
419b51b385
[Bugfix][ROCm][CI] Stabilize build context and source caches ( #51721 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
2026-08-10 21:51:39 -05:00
aoshen02 and Roger Wang
99e62b802c
[Bugfix][Model Loader] Defer post-load attention weight processing ( #49519 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-08-10 19:16:57 -07:00
acmore
f37dff22fc
[Bugfix] Import each packed IPC export once on the consumer side ( #51259 )
...
Signed-off-by: acmore <hezj07+github@gmail.com >
2026-08-11 10:05:27 +08:00
33584901ba
[Frontend] Add content_parts to /inference/v1/generate for raw multim… ( #51478 )
...
Signed-off-by: Allen Shen <aoshen@inferact.ai >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-11 10:00:12 +08:00
Andreas Karatzas and OpenAI Codex
1a1727330a
[CI] Solidify speculative decoding E2E coverage ( #50713 )
...
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-10 19:56:39 -06:00
Tarun Kumar
48bada6ea4
Fix chat completion 500 on non-object JSON bodies ( #51654 )
...
Signed-off-by: Tarun Kumar <takumar@redhat.com >
2026-08-11 01:31:19 +00:00
Bugen Zhao and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f1e921b6d6
[Rust Frontend] Support dynamic tools from developer messages ( #51144 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-10 18:30:38 -07:00
Qiuyang Yue
b2506d62ae
[MM][CG][BugFix] Fix Ernie-4.5-VL encoder CG postprocess for multi-path outputs ( #51461 )
...
Signed-off-by: qyYue1389 <yueqiuyang1389@gmail.com >
2026-08-11 00:39:04 +00:00
Bugen Zhao and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
d8c70f2243
[Rust Frontend] Upgrade MiniJinja to 2.22 & remove method lookup workaround ( #51235 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-11 00:23:24 +00:00
Kevin H. Luu and OpenAI Codex
8a9f9f762a
[CI] Parallelize release image publishing ( #51735 )
...
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-10 16:38:22 -07:00
Myeongwon Kim
8977ea8895
[Perf] Skip detokenization in offline beam search ( #50333 )
...
Signed-off-by: samuelkim7 <samuelmwkim@gmail.com >
2026-08-10 16:17:38 -07:00
Flora Feng and Wentao Ye
8bcc916a98
[Bugfix] Fix DeepSeek V4/3.2 tokenizer vocab size overcount crashing guided decoding ( #51727 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-08-10 23:00:08 +00:00
Giancarlo Delfin and Woosuk Kwon
3e174bb73c
[Model Runner V2][MTP] Share topk index buffer between draft steps ( #47352 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
2026-08-10 15:46:13 -07:00
Almog Tavor
c3cac8c63d
[Bugfix][MiMo] Apply vision attention sinks in the window attention path ( #49815 )
...
Signed-off-by: almogtavor <almogtavor@gmail.com >
2026-08-10 22:36:42 +00:00
Tristan Rice
98a4144a41
[Bugfix] Preserve non-logitproc entry points in tests ( #51097 )
...
Signed-off-by: Tristan Rice <rice@fn.lc >
2026-08-10 18:14:53 -04:00
Connor Carpenter and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
fa722b9f01
[Rust Frontend][gRPC] Add explicit data-parallel rank routing ( #51178 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-10 15:05:24 -07:00
Kevin H. Luu and OpenAI Codex
3c6b2e9c08
[CI] Add /ci cancel command ( #51732 )
...
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-08-10 14:55:26 -07:00
Connor Carpenter and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
635dd6aae6
[Build][gRPC] Publish protobuf schemas to Buf ( #51276 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-10 13:34:39 -07:00
63ac04a61e
[Kimi-K3] DCP support ( #50484 )
...
Signed-off-by: girasoley <girasoley@inferact.ai >
Signed-off-by: Summer Yang <girasoleyang@gmail.com >
Signed-off-by: 云挚 <ningyunxiao.nyx@antgroup.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: 云挚 <ningyunxiao.nyx@antgroup.com >
Co-authored-by: foraxe <73625538+foraxe@users.noreply.github.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
2026-08-10 12:54:26 -07:00
Flora Feng
3f142bd85e
[Bugfix] Align deepseek v4 parser thinking default with tokenizer ( #51296 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-08-10 19:30:30 +00:00
d40c3e3c00
Fix DSpark warmup without sparse index buffer ( #50693 )
...
Signed-off-by: xijiade.aihemaiti <3146335281@qq.com >
Signed-off-by: xijiaat <52128022+xijiaat@users.noreply.github.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-10 15:12:19 -04:00
1482d2e015
[ROCm][DistInf] Enable vLLM DI CI with buildkite/slurm ( #47030 )
...
Signed-off-by: lcskrishna <lollachaitanya@gmail.com >
Co-authored-by: Sheral Kumar <shekumar@amd.com >
Co-authored-by: avininjamay8 <Avinash.Paul@amd.com >
Co-authored-by: tej <37236721+itej89@users.noreply.github.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-08-10 18:39:26 +00:00
Flora Feng
05f0a80016
[Bugfix][Structured Output] Mask request stop tokens in xgrammar until grammar terminates ( #49227 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-08-10 14:29:20 -04:00
405bc86768
[Perf] Launch the top-k/top-p Triton sampler kernel with 8 warps ( #51507 )
...
Signed-off-by: BabyDrangoner <148877251+BabyDrangoner@users.noreply.github.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-10 10:55:35 -07:00
Woosuk Kwon and OpenAI Codex
79c865b838
[Perf] Narrow DeepSeek V4 eager CUDA graph region ( #51430 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-10 10:52:10 -07:00
21c667aa64
[Docker] Cache test dependencies before vLLM install ( #51184 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-08-10 10:34:59 -07:00
wangxiyuan and Benjamin Chislett
355a338b8f
[BugFix][SpecDecode] Fix dspark parallel_drafting_token_id init bug ( #51602 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
2026-08-10 13:33:10 -04:00
