MOVZX and Hinata
5aa2ac64f8
[Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
...
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.
- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
render routes, and vllm/config/speech_to_text.py. GenerationTask is
generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
Multi, Offloading and Example connectors stay; production uses the
CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
arch gate, declarations and op registrations. The CUDA build never
defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
fail with "The model ... does not exist. Valid aliases: ...".
Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 21:06:19 +07:00
Stefano Castagnetta and Kevin H. Luu
784d1fff0c
[MRV2][Spec Decode] Quantized draft lm_head for drafters that share the target head ( #59973 )
...
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 22:06:40 +00:00
Laith Sakka and Nick Hill
9cede34a67
[Bugfix] Create cuBLAS handles before memory profiling to avoid OOM ( #61044 )
...
Signed-off-by: Laith Sakka <lsakka@meta.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-10-15 08:59:22 -04:00
Sherif Waly
be4ce2e505
[Bugfix] Share PCP-aware local GPU indexing for NUMA binding ( #61237 )
2026-10-14 10:54:03 +02:00
3ccc0a90b6
[KV Connector] Support sleep mode with MooncakeConnector over RDMA ( #59625 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Dakai An <dakaian108@gmail.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Dakai An <dakaian108@gmail.com >
Co-authored-by: Dakai An <77474977+andakai@users.noreply.github.com >
2026-10-14 04:18:47 +00:00
Almog Tavor and Itay Alroy
87e7dce3a7
[MRV2] Keep warmup out of the EPLB load stats ( #56716 )
...
Signed-off-by: almogtavor <almogtavor@gmail.com >
Co-authored-by: Itay Alroy <75032521+itayalroy@users.noreply.github.com >
2026-10-14 00:00:48 +00:00
aoshen02 and Codex
239a3fa3a8
[Sleep] Retain draft-private weights with existing parameter selector ( #61046 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
2026-10-11 23:30:56 +08:00
aoshen02
bd5bbce6e4
[RL] Release weight-transfer communicators without stopping inference ( #61060 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: fh3 <a@b>
2026-10-11 21:15:14 +08:00
Xun Sun
26a488ad93
[Fast Start][Bugfix] Snapshot GPU memory after external weights are ready ( #60966 )
2026-10-11 07:10:04 +00:00
Nick Hill
f8471164bf
[MRV2] Detatch weight-reload logic from MRV1 ( #60937 )
2026-10-10 21:15:05 -07:00
Misha Goin and Claude Opus 5.5
29e4d2d42e
[Frontend][Build] Add vllm download-kernels for FlashInfer precompiled kernels ( #58765 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Signed-off-by: Misha Goin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 21:22:35 -07:00
d8fa463e2b
[Bugfix][Prefix caching] Fix inconsistent hybrid Mamba prefix cache boundaries ( #60533 )
...
Signed-off-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-08 18:33:48 +00:00
Woosuk Kwon
194bd72be7
[Compile] Remove torch.compile-based sequence parallelism and AsyncTP ( #60517 )
2026-10-08 08:28:23 -07:00
czhu-cohere and aoshen02
6fddc9b21e
[Profiler] Add per-session profiling controls to Python and Rust frontends ( #57875 )
...
Signed-off-by: czhu <conway.zhu@cohere.com >
Signed-off-by: Conway Zhu <conway.zhu@cohere.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-06 05:21:19 +00:00
aoshen02 and Claude Opus 5.5
d09ff77637
[Bugfix] Give each data-parallel engine its own global RNG streams ( #59788 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-05 17:38:18 +00:00
Thiago Mota Martins and Fangzhou Ai
92044241a0
[Bugfix][ROCm] Preserve config during GPU memory profiling ( #58014 )
...
Co-authored-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com >
2026-10-04 22:28:33 -07:00
aoshen02 and Claude Opus 5.5
8e58ac22af
[Feature] Offload KV-init runtime state on sleep ( #59158 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-02 09:49:21 +00:00
aoshen02 and Claude Opus 5.5
866f02db8a
[Feature] Release WorkspaceManager scratch on sleep ( #59156 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-02 02:14:37 +00:00
e5e38ba9b7
[Model Runner V2] Support randomized dummy inputs ( #58411 )
...
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-10-01 00:52:26 -07:00
aoshen02 and Claude Opus 5.5
643bbef55c
[Refactor] Share workspace and model runner init between GPU and XPU workers ( #59200 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 14:29:12 +00:00
42d9f8ccdb
support rl feature : weight checker ( #51350 )
...
Signed-off-by: shiyuan680 <yangcheng104@huawei.com >
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: shiyuan680 <yangcheng104@huawei.com >
Co-authored-by: Ronald1995 <ronaldautomobile@163.com >
Co-authored-by: DeepSeek Harness <noreply@deepseek.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-30 03:13:43 +00:00
f5ed3d9c03
[Fast Start] Charge daemon-held weights against gpu_memory_utilization ( #57298 )
...
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Xun Sun <UNIDY2002@outlook.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Xun Sun <UNIDY2002@outlook.com >
2026-09-29 17:58:35 +00:00
eb0f2ca37f
[mooncake] support CUSTOM_MEM_POOL in vllm ( #49300 )
...
Signed-off-by: bruce.xu <bruce.xb@alibaba-inc.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: bruce.xu <bruce.xb@alibaba-inc.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-27 00:26:05 +00:00
6491f481a7
[Bugfix] Stop allocator fragmentation from shrinking the KV cache during memory profiling ( #58430 )
...
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com >
Signed-off-by: Robert Shaw <robertgshaw2-redhat@h100-01.nemg-001.lab.rdu2.dc.redhat.com >
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: Robert Shaw <robertgshaw2-redhat@h100-01.nemg-001.lab.rdu2.dc.redhat.com >
Co-authored-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-25 12:16:27 -04:00
Nick Hill and Kimi Code
e30559b58f
[Core] Skip JIT monitor when JIT warmup is disabled ( #58590 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Kimi Code <noreply@moonshot.cn >
2026-09-24 12:08:16 -07:00
Nick Hill
09fe178dba
Revert "[DSpark] Support pipeline-parallel targets in aggregated serving ( #56956 )" ( #58484 )
2026-09-24 08:48:36 -07:00
liuzhenwei
dcfc17e0b1
[XPU] enable XPU GRAPH by default ( #51600 )
...
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com >
2026-09-24 19:17:25 +08:00
e6dc16cebd
[PCP] Support prefill context parallelism with data parallelism ( #57075 )
...
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
Co-authored-by: QiuChunshuo <qiuchunshuo@huawei.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-23 22:32:36 -04:00
157bcb7c48
[DSpark] Support pipeline-parallel targets in aggregated serving ( #56956 )
...
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
Co-authored-by: Kimi Code <noreply@moonshot.cn >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
2026-09-23 04:52:28 -07:00
94f4170df3
[Bugfix] Set worker runtime threads before profiling and compilation ( #55891 )
...
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-22 21:12:20 -05:00
pengyihang and mgoin
415e11c6f1
[Bugfix][V1] Honor enable_jit_warmup for V2 kernel warmup ( #55146 )
...
Co-authored-by: mgoin <mgoin64@gmail.com >
2026-09-22 15:34:31 -04:00
639461eda4
[RL][Sleep] Retain frozen weights across level-2 sleep ( #57891 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-22 12:36:55 +08:00
0b7f11a1ee
[AuxOutput] Add block-keyed storage for routed-expert outputs ( #45635 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-21 10:25:05 +00:00
czhu-cohere
3918f3c5a3
[Profiler] Unify platform-aware torch profiling ( #57460 )
2026-09-20 19:17:23 -07:00
092bdd6d57
[Frontend][Core] Add release_kv_cache_memory() API ( #44890 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: AlanFokCo <alanfok2868@gmail.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com >
Co-authored-by: AlanFokCo <alanfok2868@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-09-18 07:12:12 +08:00
9e1cd262cb
[Bugfix] Use DP index for dense DP weight updates and EC CPU region ( #56950 )
...
Signed-off-by: lxy-alexander <lxy_alexander@outlook.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-17 22:44:36 +08:00
Matt
2d8a5c5741
[ROCm][Bugfix] Reduce CUDA graph divergences ( #57229 )
...
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com >
2026-09-17 05:12:13 +00:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
Taneem Ibrahim and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
ba2ae9f239
[MRV2] Run pooling post processing on non-final PP ranks ( #56666 )
...
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-14 16:11:11 -07:00
鐘天楽 and Nick Hill
0a5747d410
[Profiler] Add Proton CUDA graph attribution for MRV2 ( #51084 )
...
Signed-off-by: Luossu <zhluosuu@outlook.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-14 19:02:07 +00:00
Lucas Wilkinson and OpenAI Codex
7fe8fc803d
[NIXL][PCP][DCP] Expose PCP producer KV shards as transfer ranks ( #56645 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-09-13 20:23:15 +00:00
6fbb00b188
[EPD] Add ECMooncakeConnector for encoder cache over Mooncake TransferEngine ( #41567 )
...
Signed-off-by: Teng Ma <sima.mt@alibaba-inc.com >
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com >
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Tianyu Guo <guoty@inferact.ai >
Co-authored-by: Zhou ziheng <jiaranran2@gmail.com >
Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-07 06:21:43 +00:00
685074cd61
[Bugfix][V2] Warm up kernels before capturing CUDA graphs ( #55341 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-04 22:44:18 +00:00
Derek Weitzel and Claude Fable 5.1
859dd39512
[Bugfix][Core] Wait for the previous PP tensor sends before the next forward pass ( #54962 )
...
Signed-off-by: Derek Weitzel <djw8605@gmail.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-03 08:01:36 +00:00
4ac452ad98
[Core] Release NCCL communicator memory in sleep mode ( #51485 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-09-01 14:31:16 +08:00
AI-Infra Rookie
c92b29a1d4
[Bugfix] Make metadata send non-blocking in GroupCoordinator.isend_tensor_dict ( #49274 )
...
Signed-off-by: z00897177 <zhouxinyi6@huawei.com >
2026-08-30 23:51:25 +00:00
stefankoncarevic
9236159bb6
[ROCm][CI] Keep startup profiling from aborting when free memory grows ( #53591 )
...
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com >
2026-08-28 00:02:40 -07:00
Ronald and aoshen02
479eeb32d2
[Bugfix][MRV2] Isolate sleep-mode KV allocations ( #53508 )
...
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-08-27 15:08:28 -07:00
aoshen02
17da48596c
[Bugfix][DP] Synchronize the device on pause completion ( #52914 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
2026-08-26 09:50:33 -07:00