MOVZX and Hinata
5aa2ac64f8
[Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
...
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.
- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
render routes, and vllm/config/speech_to_text.py. GenerationTask is
generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
Multi, Offloading and Example connectors stay; production uses the
CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
arch gate, declarations and op registrations. The CUDA build never
defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
fail with "The model ... does not exist. Valid aliases: ...".
Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 21:06:19 +07:00
02aa10539f
[Bugfix][ModelLoader] Make enable_weights_track check quantized layers ( #57108 )
...
Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Shanshan Shen <467638484@qq.com >
2026-10-10 10:44:06 +08:00
0103a9b96f
[Quantization] Add per-token NVFP4 CuTe-DSL MoE backend ( #50030 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: S1ro1 <matej.sirovatka@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
2026-09-30 10:42:03 +00:00
e6dc16cebd
[PCP] Support prefill context parallelism with data parallelism ( #57075 )
...
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
Co-authored-by: QiuChunshuo <qiuchunshuo@huawei.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-23 22:32:36 -04:00
Mikko Tukiainen and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
17322375a2
[ROCm][Perf] Route the fused shared-expert gate GEMM through the platform dispatcher ( #54185 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-21 17:17:55 +00:00
Robert Shaw and Robert Shaw
c38f92aae7
[A2A] Allow EPLB + SharedExpert Overlap + DeepEPv2 ( #57236 )
...
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com >
Co-authored-by: Robert Shaw <robshaw@redhat.com >
2026-09-17 04:02:39 +00:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
4bac767695
[Perf][Nemotron] Skip redundant latent-MoE all-reduce at TP>1 (~13% decode win) ( #52301 )
...
Signed-off-by: Junhao Shen <junshen@nvidia.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-15 07:33:51 -07:00
Matt and Claude Opus 5
9163190dda
[ROCm][Bugfix][Perf] Tune multi-stream shared experts use; wvSplitKrc fixes ( #56098 )
...
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-10 15:14:22 -07:00
Li, Jiang
c3ccc0e957
[Model] Add DeepSeek-V4 CPU backend ( #55355 )
...
Signed-off-by: jiang1.li <jiang1.li@intel.com >
2026-09-10 14:14:48 +08:00
2c7d7dd64a
[Perf][ROCm] Dual-stream decode with hipgraphs ( #52033 )
...
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Shanshan Shen <467638484@qq.com >
2026-08-30 08:24:20 +00:00
Wentao Ye
29c9af5211
[Kimi K3 Refactor] Add UnfinalizedMoEOutput proto following up for #53152 ( #53310 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-08-24 11:21:11 -04:00
7a2fdbaac4
[K3 Perf] Fuse MXFP4 top-k finalization into latent-tail, ~5% E2E latency reduction ( #53152 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-21 13:29:59 -04:00
Simon Danielsson and Andreas Karatzas
3d204dfdaa
Revert "[Perf][ROCm] Dual-stream decode with hipgraphs" ( #52024 )
...
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-08-13 04:38:34 +00:00
Simon Danielsson
47ececb58e
[Perf][ROCm] Dual-stream decode with hipgraphs ( #48223 )
...
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com >
2026-08-12 09:32:24 +08:00
Wentao Ye
fd04edef3e
[Refactor] Delete dead code in models ( #51838 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-08-11 15:44:26 -04:00
Julien Debache
52be12cfac
feat: allow shared expert overlapping for FlashInfer one-sided all-to-all ( #50569 )
...
Signed-off-by: jdebache <jdebache@nvidia.com >
2026-08-11 10:06:22 -07:00
bnellnm
373fe8b83e
[MoE Refactor] Remove MoE legacy code ( #51078 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
2026-08-05 14:35:36 -07:00
Wentao Ye
d31de3c421
[Kimi K3 Perf] option to shard the shared expert for non mega case, 16.98 GiB memory/GPU saved ( #50912 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-08-04 14:00:44 -04:00
Jee Jee Li
b9d1e2437e
K3: Move LatentMoERunner ( #50678 )
...
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
2026-08-03 09:36:17 +00:00
Jee Jee Li
4635cc3e8f
Shard the K3 Latent-MoE up-projection on large batches ( #50383 )
2026-08-03 00:49:39 -07:00
bnellnm and Harry Mellor
6e311c6e20
[MoE Refactor] Rename FusedMoE to FusedMoEFactory ( #44941 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
Signed-off-by: bnellnm <49004751+bnellnm@users.noreply.github.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-31 11:28:18 +01:00
+13
aeeb36b1f1
[New model] Kimi K3 ( #50000 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Summer Yang <girasoleyang@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Bowen Wang <abmfy@icloud.com >
Co-authored-by: gnovack <novackgm@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com >
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-07-30 03:49:58 -07:00
+12
7c6729b769
[Model] Add Kimi K3 support: model files and kernels [1/N] ( #50089 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Summer Yang <girasoleyang@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Bowen Wang <abmfy@icloud.com >
Co-authored-by: gnovack <novackgm@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com >
Co-authored-by: Codex <codex@openai.com >
2026-07-29 14:10:58 +08:00
b6ff8a2f50
[Core] Add MRV2 virtual-batch PCP for MLA ( #46570 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-07-19 02:53:15 +00:00
Wei Zhao and Yongye Zhu
2afa3f7e95
[Perf] Minimax M3 - Support cross-layer allreduce-norm fusion ( #47631 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
2026-07-07 21:16:32 -07:00
Wentao Ye and Canlin Guo
b6cc46ec3b
[Feature] Support sequence parallel without the need for DP, 1.9%~5.0% E2E Throughput Improvement ( #47070 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
Co-authored-by: Canlin Guo <canlinguosdu@gmail.com >
2026-07-05 05:41:30 -07:00
Harry Mellor
1ab9522935
Remove more unnecessary load_weights methods ( #47058 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-06-30 15:22:16 +01:00
Woosuk Kwon
89876b0c54
[GLM5] Implement op fusion for GLM5/DSV3.2 ( #46876 )
2026-06-28 08:17:39 -07:00
Netanel Haber and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
c5e5c33fcd
[Bugfix][MoE] Restore routed output unpadding before shared expert add ( #45707 )
...
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-16 16:06:28 +03:00
bnellnm
dc68bd8c41
[MoE Refactor] FusedMoE/MoERunner inversion refactor ( #41184 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
2026-06-08 10:42:58 -04:00
Yongye Zhu
2d2c660104
[MoE] Remove inplace fused experts mechanism ( #43727 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
2026-05-27 20:00:19 -07:00
Jee Jee Li
ec5de7fa7d
[LoRA] Add one shot triton kernel For MoE LoRA ( #42290 )
...
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com >
2026-05-25 19:47:04 -07:00
aoshen02
8c79ad6580
Revert "[Core] Replace routing replay with device cache and async D2H pipeline" ( #39917 ) ( #42434 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
2026-05-13 23:49:01 -07:00
4d591db470
[MoE Refactor] Introduce RoutedExperts alias for FusedMoE and don't store SharedExperts in MK ( #40735 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: Robert Shaw <robertgshaw2@gmail.com >
2026-05-12 13:37:44 -04:00
2c6b59b807
[ROCm][Perf] Add Fused Shared Expert (FSE) support for Qwen3-Next ( #39280 )
...
Signed-off-by: nholmber <nholmber@users.noreply.github.com >
Signed-off-by: Tres Popp <tres.popp@amd.com >
Signed-off-by: Doug Lehr <douglehr@amd.com >
Co-authored-by: nholmber <nholmber@users.noreply.github.com >
Co-authored-by: Tres <tpopp@users.noreply.github.com >
Co-authored-by: Tres Popp <tres.popp@amd.com >
Co-authored-by: Doug Lehr <douglehr@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
2026-05-08 15:38:00 -04:00
Stefano Castagnetta and Yongye Zhu
54f548e9e5
[Bugfix] Restore moe_forward output shape invariant on TRTLLM MXFP4 path ( #41646 )
...
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
2026-05-07 15:26:06 -07:00
TomerBN-Nvidia
8189a15914
[Core] Replace routing replay with device cache and async D2H pipeline ( #39917 )
...
Signed-off-by: Tomer Barnatan <tbarnatan@nvidia.com >
2026-05-07 11:24:56 -07:00
b55b26520c
[MoE] Make MoERunnerInterface a PluggableLayer for OOT support ( #35178 )
...
Signed-off-by: wxsIcey <1790571317@qq.com >
Signed-off-by: Icey <1790571317@qq.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-30 03:31:08 -07:00
+7
4d51588e23
[Feat] DeepSeek V4 Rebased ( #40860 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Signed-off-by: qizixi <zixi@inferact.ai >
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <yongye@inferact.ai >
Co-authored-by: Simon Mo <simon@inferact.ai >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: youkaichao <youkaichao@gmail.com >
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com >
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com >
Co-authored-by: khluu <khluu000@gmail.com >
Co-authored-by: qizixi <zixi@inferact.ai >
Co-authored-by: Zhewen Li <zhewenli@inferact.ai >
2026-04-26 18:31:08 -07:00
Netanel Haber and Jee Jee Li
12a3f6454b
[Bugfix][MoE] Only unpad routed output before shared expert add or routed output transform ( #40865 )
...
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
2026-04-25 20:50:12 +00:00
Netanel Haber
e8eb0490ce
[Bugfix][MoE] Unpad routed output before shared expert add [ Fixes #35949 ] ( #40794 )
...
Signed-off-by: Netanel Haber <nhaber@nvidia.com >
2026-04-24 11:53:23 +00:00
bnellnm
4a6dd1c3cc
[Bugfix] Fix DeepSeek V2-Lite Accuracy drop ( #40673 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
2026-04-23 18:11:37 -04:00
Wentao Ye and gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
8f87eb4622
[Refactor] Clean up log once scope="local" ( #40540 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-04-22 16:42:43 -04:00
bnellnm
809d83c2dc
[MoE Refactor] Combine MoERunnerBase + DefaultMoERunner ( #40560 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
2026-04-22 14:43:17 +00:00
Martin Hickey
3951d3eacd
[MyPy] Enable mypy for vllm/model_executor/layers/ ( #40159 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-04-21 20:15:02 -07:00
bnellnm and Robert Shaw
6867bcd076
[Bugfix] Replace code that disabled shared expert overlap ( #39222 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-04-20 19:36:16 -04:00
bnellnm and Robert Shaw
726efe177b
[MoE Refactor] Move the shared/fused expert output sum into MoERunnerBase ( #35949 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-04-20 12:28:46 -04:00
bnellnm
19ec9a0a62
[MoE Refactor] Refactor ZeroExpertFusedMoE into new framework ( #35549 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
2026-04-14 16:11:20 -04:00
bnellnm and Robert Shaw
e1e318af01
[MoE Refactor] Remove MoE DP chunking ( #39107 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-04-14 09:48:05 -04:00