MOVZX and Hinata
5aa2ac64f8
[Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
...
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.
- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
render routes, and vllm/config/speech_to_text.py. GenerationTask is
generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
Multi, Offloading and Example connectors stay; production uses the
CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
arch gate, declarations and op registrations. The CUDA build never
defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
fail with "The model ... does not exist. Valid aliases: ...".
Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 21:06:19 +07:00
MOVZX and Hinata
e61dcb9143
[Mini-vLLM] SM86-only cut: models, backends, Rust, tests
...
- Keep only Qwen3.5, Qwen3.5-MTP and Qwen4Exp model paths; trim the model registry to 41 entries
- Delete idle attention backends (MLA, FlashInfer, ROCm-only, quant backends) and dead GDN CuTeDSL kernels
- Remove inc/quark quant loaders; keep compressed-tensors and Marlin
- Drop the Rust frontend, pooling entrypoints, and idle test suites
- Rebuild kernels for sm_80;sm_86 and strip the extensions
- Verified: full import sweep clean (optional deps only) and a dummy-load Qwen3.5 engine run on RTX 3090 via the Triton GDN path
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-18 19:19:56 +07:00
MOVZX
76df77c04e
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:16:39 +07:00
f9b65c30c7
[ROCm][Perf] Make AITER FlyDSL GDN prefill default on gfx942/gfx950 ( #60645 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-14 10:11:57 +00:00
Jiangyun Zhu and Claude Opus 5.5
badfcdbf91
[Perf][GDN] Internal prefill checkpoints for the FlashInfer GDN prefill backend ( #60050 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 14:49:46 +08:00
liuzhenwei and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
ff440c7805
[BreakableCG] Add eager break for Mamba-family ops ( #57881 )
...
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-10 14:06:57 +08:00
MOVZX
ac305c44cb
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-09 17:41:13 +00:00
Harry Mellor and Claude Opus 5.5
2c99ee9333
[Chore] Remove vendored code made redundant by Transformers 5.16.1 ( #60623 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 09:29:03 +00:00
Mikko Tukiainen and Cursor
548597367e
[ROCm][Perf] Add AITER FlyDSL GDN prefill backend ( #57560 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-10-07 09:44:22 -05:00
MOVZX
85c31d47ad
Merge remote-tracking branch 'upstream/main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-03 22:06:10 +07:00
e4294144a8
[Perf] Keep non-speculative GDN decode on the standard path ( #59735 )
...
Signed-off-by: DimensionSTP <65501090+DimensionSTP@users.noreply.github.com >
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Kimi Code <noreply@moonshot.ai >
2026-10-02 13:44:53 +00:00
Jürgen Schmied and Claude
03cae7837f
[Spec Decode][Qwen4Exp] RecoverSSM for Qwen GDN and the PLE short conv
...
With --use-replayssm and speculative decoding, Qwen4Exp's Gated DeltaNet layers verify the
draft window from a read-only checkpoint and keep a per-token replay record
[correction(V), normalized key(K), gate(1)] in fp32. After sampling, one commit replays the
accepted tokens and writes the recurrent state once (plus the block-boundary state in align
mode), instead of snapshotting the state after every draft position. The PLE dilated short
conv follows the same protocol (accepted-suffix compaction). num_speculative_blocks = 0,
so no per-draft Mamba blocks are reserved.
- GDN verify/commit kernels: model_executor/layers/mamba/gdn/recoverssm_gdn.py
- metadata/backends: v1/attention/backends/gdn_recoverssm.py, ple_recoverssm.py
- the verify reads strided q/k/v views of the conv output and writes in place
- builders declare AttentionCGSupport.NEVER: FULL_AND_PIECEWISE falls back to PIECEWISE;
the fallback now also counts breakable CUDA graphs (empty splitting_ops, with or
without torch.compile) as piecewise
attention instead of dropping to NONE
- the generic Mamba metadata builder skips the Mamba2 ReplaySSM ring-buffer workspaces
when --use-replayssm selects RecoverSSM (the PLE short conv uses that builder)
- tests: GDN verify/commit vs the native kernels (fp32 and bf16 state, align boundaries,
mixed and ragged batches, extreme gates), PLE vs token-by-token decode, config selection
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 01:17:24 +02:00
Harry Mellor and Claude Opus 5.5
d881812500
[Chore] Use Transformers v5 names and drop redundant processor use_fast ( #58550 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-09-24 14:14:10 +00:00
44bf011f44
[Dependency] Upgrade FlashInfer version to 0.7.0 ( #58069 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-09-23 22:45:23 -07:00
Wentao Ye
fae5cbfc8b
[Refactor] Remove dead code multiple places ( #58002 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-22 16:36:03 -04:00
Mikko Tukiainen and Cursor
68363d64ec
[ROCm][Perf] Avoid extra reshape kernel in Qwen GDN output norm ( #47842 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-22 14:38:16 +08:00
382970ee6c
[Kimi-K3][AMD] Return KDA and MLA projection outputs directly ( #50592 )
...
Signed-off-by: Liuyinfeng01 <yinfeliu@amd.com >
Co-authored-by: Liuyinfeng01 <199041580+LiuYinfeng01@users.noreply.github.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-21 15:44:41 -05:00
Yan Ma
d05da62e9c
[XPU] fix incorrect gdn kernel log for XPU path ( #54871 )
...
Signed-off-by: Yan Ma <yan.ma@intel.com >
2026-09-20 14:00:31 +08:00
fbe8a157fb
[ROCm][Perf] Enable the AITER GDN decode fast path for flat qkvz layouts ( #53623 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: TJian <tunjian.tan@embeddedllm.com >
2026-09-19 06:05:04 -07:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
975dca5bb5
[Perf][GDN] Scatter mixed speculative outputs into the caller buffer ( #57140 )
...
Signed-off-by: Arpan Tripathi <tripathiarpan20@gmail.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Xavier Lyu <9938360+xavierlyu@users.noreply.github.com >
Co-authored-by: Bohdan Podziubanchuk <114604629+Danbog32@users.noreply.github.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-16 13:03:57 +00:00
Stefano Castagnetta
f6326f53bd
[Perf][GDN] Enable the FlashInfer GDN prefill kernel on SM12x ( #55715 )
...
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
2026-09-08 12:26:03 +00:00
Thien Tran
d9105ea800
[Qwen3.8-Flash-Next] Remove torch.compile for NVIDIA implementation ( #55272 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-09-06 21:57:56 -07:00
Jee Jee Li
199cb9b964
[Kernel] Remove unused fake implementation ( #55535 )
...
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
2026-09-06 17:54:32 -07:00
Li, Jiang
fdbf2ddbd2
[Bugfix][CPU] Fix several bugs ( #54042 )
...
Signed-off-by: jiang1.li <jiang1.li@intel.com >
2026-08-31 15:33:06 +08:00
+5
e126687a9a
[Model] Support Qwen3.8-Flash-Next ( #53896 )
...
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com >
Signed-off-by: andyluo7 <andy.luo@amd.com >
Signed-off-by: lishunyang12 <lishunyang03@gmail.com >
Signed-off-by: Harry Huang <huanghaoyan.hhy@alibaba-inc.com >
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: KungYork <firelikest@163.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: andyluo7 <43718156+andyluo7@users.noreply.github.com >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: andyluo7 <andy.luo@amd.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: lishunyang12 <lishunyang03@gmail.com >
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-31 05:57:56 +00:00
7ab2923489
[Flashinfer] Upgrade Flashinfer version to 0.6.18 ( #54313 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com >
2026-08-30 16:01:05 -07:00
a4d70bef37
[Model Runner V2] Reserve CUDA graph memory ( #53306 )
...
Signed-off-by: Anh Tran <anh.tran.3889@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Anh Tran <anh.tran.3889@gmail.com >
Co-authored-by: anhtra3889 <anhtra@nvidia.com >
2026-08-23 23:11:19 -07:00
kiroxu
cdb8545a91
[Kernel][Perf] Support Qwen head ratios in fused GDN MTP ( #52539 )
...
Signed-off-by: kiroxu <148877251+BabyDrangoner@users.noreply.github.com >
2026-08-17 18:17:40 -07:00
1be3628367
[Kernel][Perf] Add fused CUDA post-conv MTP decode kernel for Qwen3.5 GDN ( #51674 )
...
Signed-off-by: Jie Fang <jief@nvidia.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-08-14 11:44:45 +08:00
kliuae
f96261637c
[ROCm][Perf] Kimi-K3 Remove prefill pipeline stall in chunk KDA ( #51862 )
...
Signed-off-by: kliuae <kuanfu.liu@embeddedllm.com >
2026-08-13 09:11:45 +00:00
Jiangyun Zhu and OpenAI Codex
5af7c8dad7
[Bugfix] Align Qwen GDN gates with speculative tokens ( #51812 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-11 23:35:30 +08:00
skysnow2001 and Douglas Lehr
f43e1d26e3
[ROCm] Enable AITER and FP8 inference on GFX120x ( #43615 )
...
Signed-off-by: skysnow2001 <skysnow9285@gmail.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
2026-08-03 16:45:37 -05:00
+13
aeeb36b1f1
[New model] Kimi K3 ( #50000 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Summer Yang <girasoleyang@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Bowen Wang <abmfy@icloud.com >
Co-authored-by: gnovack <novackgm@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com >
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-07-30 03:49:58 -07:00
Wentao Ye
626c90b2d5
[Refactor] Move fla to third party ( #48500 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-07-16 19:22:36 +01:00
Artem Perevedentsev and Matthew Bonanni
7cd1d57b74
[CI/Build][Docker] Bump nvidia-cutlass-dsl to 4.6.0 and drop packaging workarounds ( #47442 )
...
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-07-16 13:51:41 +00:00
Canlin Guo and yewentao256
a02984ed47
[Perf][Qwen] Replace MOE all-reduce with reduce-scatter ( #47006 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Co-authored-by: yewentao256 <zhyanwentao@126.com >
2026-07-12 06:14:49 +00:00
Jiangyun Zhu
300e33797f
[Perf] fuse more rmsnorm and all-reduce in qwen3.5 ( #46998 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
2026-07-10 15:37:51 +08:00
8d4f54966c
fix(quantization): Fix AWQ dequantize on Intel XPU and refactor AutoAWQ config ( #42727 )
...
Signed-off-by: Alex <alex.tech.lab@outlook.com >
Signed-off-by: AlexHuang <jihuihuang@tencent.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-18 20:12:28 +08:00
Vadim Gimpelson and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
fa27d4e9cf
[PERF] [Qwen3.5] Split mixed prefill+decode batches: route decodes to the recurrent kernel ( #44700 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-06 22:13:50 +08:00
Jiangyun Zhu
a55fccfc7c
[mamba] unify KDA conv states into one cache to match 2-state SSM layout ( #44539 )
2026-06-04 20:38:05 +02:00
IdoAtadTD
c91a87f01a
[BugFix] [GDN] Read linear_key_head_dim from hf_text_config for multimodal models ( #43978 )
...
Signed-off-by: IdoAtadTD <ido.atad@twodelta.com >
2026-06-02 17:17:55 +03:00
zexplorerhj and haojiangzheng
f3b2a819f7
[Perf][KDA] Fuse gate softplus, chunk-local cumsum, and RCP_LN2 scaling ( #43667 )
...
Signed-off-by: haojiangzheng <justineric096@gmail.com >
Co-authored-by: haojiangzheng <justineric096@gmail.com >
2026-05-28 13:47:08 +00:00
Thien Tran
d56612c621
[GDN] GDN Prefill kernel for SM100 ( #43273 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-05-26 14:02:11 +08:00
Artem Perevedentsev
23f7b11bf4
[Bugfix] Detect wrong libcute_dsl_runtime.so variant in FlashInfer GDN ( #43427 )
...
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com >
2026-05-22 19:33:33 +00:00
wangxiyuan
7e1b45a092
[Attention] Mamba attention module refactor ( #41126 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2026-05-22 17:13:12 +08:00