MOVZX and Hinata
5aa2ac64f8
[Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
...
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.
- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
render routes, and vllm/config/speech_to_text.py. GenerationTask is
generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
Multi, Offloading and Example connectors stay; production uses the
CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
arch gate, declarations and op registrations. The CUDA build never
defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
fail with "The model ... does not exist. Valid aliases: ...".
Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 21:06:19 +07:00
MOVZX
e0fe5df731
Merge branch 'main' into patches
2026-10-17 08:06:56 +00:00
b544587408
[Core][Feat] Pluggable KVCacheConfigBuilder for platform/model-specific KV cache planning ( #53558 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-10-16 23:16:47 +00:00
MOVZX
be95edb7f6
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-16 17:35:42 +07:00
d12ea8e309
[Bugfix][MRV2] Keep input_ids on later PP stages for raw-token models ( #61016 )
...
Signed-off-by: booth-algo <kevinlauofficial01@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 12:54:21 -07:00
MOVZX
76df77c04e
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:16:39 +07:00
bfdbfa1632
[Bugfix][PCP] Align MoE routing padding with gathered tokens ( #60946 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: wangyicong <wangyicong@bytedance.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-10-14 21:04:00 -04:00
Ayushman Singh
9b5c6707a3
[Bugfix][MRV2] Capture DCP FULL cudagraphs with KV context ( #61496 )
...
Signed-off-by: Ayushman Singh <40520701+ayush1399@users.noreply.github.com >
2026-10-14 19:20:48 +00:00
df6c7ad031
[Core][Refactor] Move KV cache planning logic to kv_cache_planning.py ( #61183 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-13 03:05:39 +00:00
MOVZX
ac305c44cb
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-09 17:41:13 +00:00
Wentao Ye
74c5cbcd7b
[Bugfix][MRV2][Spec Decode] Honor dynamic K in autoregressive speculators, 1.2~1.3x kernel perf improvement ( #57053 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-10-05 14:54:11 -04:00
MOVZX
f83ecfac16
Merge remote-tracking branch 'upstream/main' into patches
2026-10-05 08:52:31 +07:00
aoshen02 and Claude Opus 5.5
5e56e9af2a
[Feature] Release the CUDA graph pool on sleep ( #59160 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-04 01:13:10 +00:00
MOVZX and Hinata
ae0350c0de
[Spec Decode][MRV2] RecoverSSM for Qwen GDN and PLE short conv (PR #58863 )
...
Intake of open upstream PR vllm-project/vllm#58863 . Active only with
VLLM_USE_V2_MODEL_RUNNER=1; inert otherwise.
Speculative decoding on Gated DeltaNet snapshotted the full recurrent
state after every draft position and reserved one Mamba block per draft
position. On a 24 GB card that is reserved VRAM we never use, plus
per-step copy traffic.
RecoverSSM, the protocol Kimi-K3 KDA already uses, replaces this:
verify runs from a read-only checkpoint and stores a small per-token
replay record [correction (V), normalized key (K), gate (1)] in fp32;
commit replays only the accepted tokens and writes the recurrent state
once, plus the block-boundary state in align mode. The PLE dilated
short conv compacts its accepted window with the same plan.
Conflict in vllm/models/qwen4_exp/nvidia/mtp.py: our get_top_tokens()
work put LocalArgmaxMixin on Qwen4ExpMTP, the PR adds SupportsReplaySSM
to the same base list. Kept both.
Co-Authored-By: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-09-29 13:39:09 +07:00
Almog Tavor
171d1b9ef6
[Elastic EP] Support Model Runner V2 ( #53934 )
...
Signed-off-by: almogtavor <almogtavor@gmail.com >
Signed-off-by: Almog Tavor <70065337+almogtavor@users.noreply.github.com >
2026-09-29 00:20:40 +00:00
Jürgen Schmied and Claude
a80e41d690
[Mamba] Share RecoverSSM commit kernels and helpers across models
...
RecoverSSM's commit-plan and conv-window compaction kernels are model independent.
Move them from models/kimi_k3 into model_executor/layers/mamba/ops/recoverssm.py
(Kimi-K3 keeps its private names as import aliases; kernel source unchanged), add
mamba/recoverssm_utils.py (activation predicate, argument validation) and the
generic spec-row/request mapping helpers to v1/attention/backends/recoverssm_metadata.py.
CacheConfig.use_kda_recoverssm becomes use_recoverssm: the switch now selects RecoverSSM
for any model whose recurrent layers implement it, not only KDA.
No functional change for Kimi-K3.
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 00:07:04 +02:00
a4eb3f25d6
[Spec Decode] Enable Gemma4 DSpark adaptive verification with FlashInfer ( #57263 )
...
Signed-off-by: zixi-qi <zixi@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-26 17:01:17 +08:00
Andreas Karatzas and OpenAI Codex
23110a0c9e
[Bugfix] Capture prefill kernels for mixed FULL graphs ( #58275 )
...
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-24 02:05:07 -07:00
Nick Hill
afec265bca
[MRV2] Miscellaneous code cleanup ( #57980 )
2026-09-23 01:51:52 -07:00
Mi Jiazhi and jiazhimi
e378275a8f
[Model] Pass intermediate_tensors to the model when capturing CUDA gr… ( #57745 )
...
Co-authored-by: jiazhimi <jiazhimi@tencent.com >
2026-09-19 23:23:29 -07:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
676650397f
[Feature][PCP] Support decode-only FULL CUDA graphs ( #53867 )
...
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com >
Signed-off-by: QiuChunshuo <chunshuoq@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-15 16:27:46 +08:00
Anjie Hou and Codex
73d2a8cf86
[2/2][Model Runner V2] FULL CUDA graph capture for microbatched steps (DBO) ( #51700 )
...
Signed-off-by: specture724 <specture724@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-13 22:42:33 -07:00
d43bb2f37f
[3/N] HiSparse: host-resident sparse-MLA decode hot-buffering ( #53781 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Signed-off-by: Matthew Bonanni <matthew@neuralmagic.com >
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-09-12 00:36:47 -04:00
83990f5bcc
[Bugfix][MRV2] Initialize DCP metadata after batch partitioning ( #55212 )
...
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai >
Signed-off-by: Ziyang Zhang <hafuhafu@qq.com >
Signed-off-by: Sy03 <1370724210@qq.com >
Co-authored-by: Raphael Rialland <raphael.rialland@mistral.ai >
Co-authored-by: Ziyang Zhang <hafuhafu@qq.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-10 06:50:09 +00:00
Wentao Ye
07950d4734
[Kimi Bug] Fix kimi k3 startup cuda graph issue with recoverSSM ( #55774 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-08 10:29:53 -04:00
685074cd61
[Bugfix][V2] Warm up kernels before capturing CUDA graphs ( #55341 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-04 22:44:18 +00:00
3f41d102c5
[1/2][Model Runner V2] DBO support, eager mode ( #50945 )
...
Signed-off-by: specture724 <specture724@gmail.com >
Signed-off-by: Anjie Hou <149605198+specture724@users.noreply.github.com >
Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
2026-09-04 14:17:39 -07:00
Yizhou and OpenAI Codex
8f816a3f66
[MRV2][Metrics] Support CUDAGraphStat in MRV2 ( #52358 )
...
Signed-off-by: Yizhou Liu <liu_yizhou@outlook.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-09-04 07:19:50 +00:00
Isotr0py
1c26e57d3c
[Bugfix] Raise for unavailable piecewise CUDA graphs ( #54782 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
2026-09-02 08:52:24 +00:00
Canlin Guo and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
b383e16396
[Bugfix] Reset cached Mamba align metadata on profiling teardown ( #54044 )
...
Signed-off-by: Canlin Guo <canlinguosdu@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-30 00:43:20 -07:00
d3d79ffc1e
[Bugfix][Spec Decode] Capture the widest uniform decode batch by default ( #50488 )
...
Signed-off-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com >
Signed-off-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Signed-off-by: Rahul Chalamala <rahul@modal.com >
Co-authored-by: Janelle Cai <janelle.cai@modal.com >
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai >
2026-08-28 22:38:13 -07:00
Nick Hill and Kimi Code
96242aa50d
[Bugfix][MRV2] Release layer-bound KV cache memory in shutdown() ( #54246 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Kimi Code <noreply@moonshot.ai >
2026-08-28 21:27:09 +00:00
Qiu Chunshuo
b3af042abd
Bugfix: use PCP slot mappings for PIECEWISE capture ( #53869 )
...
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com >
2026-08-27 05:39:36 +00:00
Wei Zhao
94d96e2446
[Bugfix] Release CUDA graph profiling memory before KV cache allocation ( #53955 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
2026-08-27 01:35:59 +00:00
fanxingran
080a66a69c
[Bugfix][ROCm] Capture CUDA graphs on the current stream ( #53818 )
...
Signed-off-by: fanxingran <xingran.fan@amd.com >
2026-08-26 23:16:45 +08:00
Wentao Ye
a447955aca
[Kimi Bug] Fix k3 torch.AcceleratorError: CUDA error: an illegal memory access was encountered ( #53773 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-08-25 17:38:48 -07:00
Nick Hill
e6cc0899b6
[Bugfix][MRV2] Run cudagraph memory profiling in a throwaway graph pool ( #53682 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-08-25 18:52:15 +00:00
c74dd2d208
[Bugfix][MRV2] Dispatch uniform decode to a padded FULL cudagraph ( #53407 )
...
Signed-off-by: Xiaohu Guo <Xiaohu.Guo@amd.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: xiaohuguo2023 <149615094+xiaohuguo2023@users.noreply.github.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-08-25 06:59:38 +00:00
a4d70bef37
[Model Runner V2] Reserve CUDA graph memory ( #53306 )
...
Signed-off-by: Anh Tran <anh.tran.3889@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Anh Tran <anh.tran.3889@gmail.com >
Co-authored-by: anhtra3889 <anhtra@nvidia.com >
2026-08-23 23:11:19 -07:00
7f7a32cfec
[Spec Decode] DSpark confidence-scheduled verification ( #47808 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com >
Signed-off-by: Lucas Wilkinson <wilkinson.lucas@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-08-12 12:27:48 -07:00
Lucas Wilkinson and OpenAI Codex
02ac17851d
[Refactor][MRV2] Unify uniform decode token count helper ( #51917 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-12 05:58:43 +00:00
0a94d85a66
[Bugfix][MRV2] Require all requests to be decoding for uniform-decode dispatch ( #51865 )
...
Signed-off-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com >
Co-authored-by: Janelle Cai <janelle.cai@modal.com >
2026-08-11 17:15:06 -07:00
Nick Hill and Woosuk Kwon
8112b6c997
[MRV2] Always build attn metadata at capture time ( #49364 ) ( #49995 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
2026-07-27 17:25:17 +00:00
vllm-agent and vllm-agent CI bot
494845e79f
Revert "[MRV2] Always build attn metadata at capture time" ( #49364 ) ( #49451 )
...
Co-authored-by: vllm-agent CI bot <ci-bot@vllm-agent.local >
2026-07-23 09:51:33 -07:00
Woosuk Kwon
6049424b7e
[MRV2] Always build attn metadata at capture time ( #49364 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
2026-07-21 19:27:16 -07:00
Nick Hill and Claude Opus 4.8
de6ec294ef
[Bugfix] Fix DSA crash under breakable piecewise cudagraphs ( #49302 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-07-21 18:28:13 +01:00
Ilya Markov and Markov Ilya
f890e1dbe2
[BugFix] Set graph_pool_id before FULL CUDA graph capture in ModelRunner V2 ( #48843 )
...
Signed-off-by: Markov Ilya <markovilya19@gmail.com >
Co-authored-by: Markov Ilya <markovilya19@gmail.com >
2026-07-21 08:59:26 -04:00
251f7e478e
[Model] Add PW CUDA graph support for Inkling [2/N] ( #48822 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-07-16 10:28:56 -07:00