iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
23,342 Commits 3 Branches 0 Tags
sm86
Commit Graph
91 Commits
Author SHA1 Message Date
MOVZXandHinata 5aa2ac64f8 [Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces 
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.

- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
  scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
  CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
  CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
  render routes, and vllm/config/speech_to_text.py. GenerationTask is
  generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
  Multi, Offloading and Example connectors stay; production uses the
  CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
  arch gate, declarations and op registrations. The CUDA build never
  defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
  fail with "The model ... does not exist. Valid aliases: ...".

Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-19 21:06:19 +07:00
MOVZXandHinata e61dcb9143 [Mini-vLLM] SM86-only cut: models, backends, Rust, tests 
- Keep only Qwen3.5, Qwen3.5-MTP and Qwen4Exp model paths; trim the model registry to 41 entries
- Delete idle attention backends (MLA, FlashInfer, ROCm-only, quant backends) and dead GDN CuTeDSL kernels
- Remove inc/quark quant loaders; keep compressed-tensors and Marlin
- Drop the Rust frontend, pooling entrypoints, and idle test suites
- Rebuild kernels for sm_80;sm_86 and strip the extensions
- Verified: full import sweep clean (optional deps only) and a dummy-load Qwen3.5 engine run on RTX 3090 via the Triton GDN path

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-18 19:19:56 +07:00
Ilya BoytsovIlya BoytsovCodexMisha GoinClaude Opus 5.5
 		189bbfb9f4 Add MoNe experts pruning support (#50415) 
Signed-off-by: Ilya Boytsov <ieboytsov@nebius.com>
Signed-off-by: Misha Goin <mgoin64@gmail.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Ilya Boytsov <ieboytsov@nebius.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-12 23:18:43 +00:00
Harry MellorandClaude Opus 5.5 21d9758459 Remove Python 3.10 support now that it is EOL (#60402) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 02:30:52 -07:00
fxmarty-amdvllmellmTan Pin SiangMisha Goin
 		f8d92cae41 [Quantization] Layer re-quantization for linear layers through online quantization API (MXFP8 -> FP8 PTPC showcase) (#55684) 
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com>
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: Tan Pin Siang <tanpinsiang@gmail.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
 2026-10-06 11:38:18 -05:00
siyuandIsotr0py c723a831a8 [Fast Start] Cache the MTP draft model in a separate daemon group (#57312) 
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
 2026-09-22 03:13:38 +00:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 16:44:29 +00:00
aoshen02 eb61707851 [Bugfix] Fix stale HPC QK-norm weights after weight refit (#56706) 
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
 2026-09-14 21:34:41 -07:00
Harry Huangandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 3116c5d06b [Qwen4Exp] Support UVA PLE-offload and Engram tensor parallelism (#54371) 
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-09 07:32:34 -07:00
siyuliusy58Isotr0py
 		a69e75b9b6 Fast Start (#54921) 
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: liusy58 <liusy58@smail.nju.edu.cn>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
 2026-09-04 06:32:06 -07:00
Nick Hill 6c18a54648 [Perf] Avoid h2d copies from non-pinned CPU tensors (#54299) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-08-29 12:06:30 +08:00
Harry Mellor 1fe3a1571a Reduce AutoWeightsLoader kwargs (#53106) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-08-20 15:40:11 +00:00
Harry Mellor f0c14b4f77 Fix weight tying (#51665) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-08-20 10:31:55 +00:00
aoshen02andRoger Wang 99e62b802c [Bugfix][Model Loader] Defer post-load attention weight processing (#49519) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-08-10 19:16:57 -07:00
Isotr0py 073c510c91 [Migration] Migrate bitsandbytes support to OOT plugin (#43529) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
 2026-08-09 10:27:31 +08:00
Ning Xie 03c782eb91 [model registry] some simple typos (#50673) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2026-08-01 15:37:29 +00:00
Mehmet CagriClaude Opus 4.8Andreas Karatzas
 		12a34a6bc7 [ROCm][DSV4] B-preshuffle the attention fp8 projections (#46720) 
Signed-off-by: Mehmet Cagri Kaymak <mehmet.kaymak@amd.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-07-30 13:59:52 -05:00
+13
Jiangyun ZhuIsotr0pyThien TranBugen ZhaoYongye ZhuZiming HuangRoger WangIsotr0pyaoshen02Woosuk KwonJee Jee Liaoshen02Summer YangKevin H. LuuBowen WanggnovackNick HillYifan QiaoxiaozhoupyRoy WangJeff MaCodexWei Zhao
 		aeeb36b1f1 [New model] Kimi K3 (#50000) 
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Isotr0py <mozf@inferact.ai>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Bowen Wang <abmfy@icloud.com>
Co-authored-by: gnovack <novackgm@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com>
Co-authored-by: Roy Wang <yasong.wang@inferact.ai>
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
 2026-07-30 03:49:58 -07:00
alexxu-robloxYQ-WangalexhxuCursor
 		d96aee0951 [Bugfix] Re-sync parameter tp_rank after process_weights_after_loading (fix replicated / disable_tp weight reload) (#48025) 
Signed-off-by: Alex Xu <alexxu@roblox.com>
Co-authored-by: YQ-Wang <yiqingwang@roblox.com>
Co-authored-by: alexhxu <alex.xu1015@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-07-18 08:40:06 +00:00
Cheng JiangchengvjiangAndreas Karatzas
 		8cf7c4d8ad [Attention Backend] add HPC-Ops Attention backend (#46020) 
Signed-off-by: chengvjiang <chengvjiang@tencent.com>
Co-authored-by: chengvjiang <chengvjiang@tencent.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-06-30 18:17:43 +08:00
Harry Mellor 5051698e41 Remove unnecessary load_weights methods (#44589) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-06-29 01:52:23 -07:00
Michael GoinandClaude 5edf7ff489 [Core] Release cached device memory under pressure on UMA GPUs during weight loading (#45179) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-06-11 17:49:50 +03:00
Zhanda ZhuandYubo Gao 5d5c776444 [Perf] FP8 FlashInfer Attn for ViT (#38065) 
Signed-off-by: Zhanda Zhu <zhandazhu@gmail.com>
Co-authored-by: Yubo Gao <ybgao-nvidia@users.noreply.github.com>
 2026-04-27 13:44:15 +08:00
Tihomir Elek 8d825b87d6 [Bug] Fix TypeError when hf_config.architectures is None during model loading (#38849) 
Signed-off-by: Tihomir Elek <tiho.elek@gmail.com>
 2026-04-13 12:13:21 +01:00
Harry Mellor b3601da6e7 [Mypy] Fix mypy for vllm/model_executor (except vllm/model_executor/layers) (#37904) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-24 17:14:01 +00:00
Wei ZhaoandMichael Goin 59d53066d8 [Feature] Support CPU Offloading without Pytorch Pinned Memory that leads to doubled allocation (#32993) 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
 2026-02-13 08:11:26 -08:00
emricksini-h 325ab6b0a8 [Feature] OTEL tracing during loading (#31162) 2026-02-05 16:59:28 -08:00
Kyle Sayers f857a03f6b [QeRL] Layerwise Reloading (#32133) 
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
 2026-01-30 08:50:05 -07:00
Matthew Bonanni a608b4c6c2 [5/N][Attention] Finish eliminating vllm/attention folder (#32064) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-01-27 10:02:51 -05:00
Cyrus Leung 2b8a38b6d6 [Model] Extend collect_children and no_init_weights contexts (#32757) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-22 08:20:27 +00:00
Cyrus Leung 4753f3bf69 [Model] Use context managers for encoder- and LM-only mode (#32605) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-20 11:43:38 +08:00
dengyunyang 8f8f469b1b [BugFix] skip language model in Encoder (#30242) 
Signed-off-by: dengyunyang <584797741@qq.com>
 2025-12-22 05:25:59 -08:00
wang.yuqi 9e77ffca3f [Model][7/N] Improve all pooling task | Deprecation as_reward_model. Extract hidden states prefer using new multi-vector retrieval API (#26686) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2025-12-08 08:10:09 +00:00
Matthew Bonanni 430dd4d9eb [Attention] Remove imports from vllm/attention/__init__.py (#29342) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2025-11-26 10:53:15 -07:00
Cyrus Leung ae66818379 [Misc] Fix pre-commit (#29238) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-22 06:48:01 -08:00
Nandan Vallamdasugemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Cyrus Leung
 		6965a392a4 Fix: Resolve circular import in model_loader/utils.py (#29189) 
Signed-off-by: nandan2003 <nandan.vallamdasu@outlook.com>
Signed-off-by: Nandan Vallamdasu  <nandan.vallamdasu@outlook.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2025-11-22 04:58:22 -08:00
Jerry Zhang da94c7c0eb Move online quantization to model.load_weights (#26327) 
Signed-off-by: Jerry Zhang <jerryzh168@gmail.com>
 2025-11-18 16:52:41 -08:00
Jonathan Chen ca76486a16 [Chore] Separate out vllm.utils.platform_utils.py (#27374) 
Signed-off-by: Jonathan <chenleejonathan@gmail.com>
 2025-10-23 19:08:06 +00:00
Isotr0py 6ac5e06f7c [Chore] Clean up pytorch helper functions in vllm.utils (#26908) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: isotr0py <2037008807@qq.com>
 2025-10-18 09:48:22 -07:00
Harry Mellor 8fcaaf6a16 Update Optional[x] -> x | None and Union[x, y] to x | y (#26633) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-12 09:51:31 -07:00
Isotr0py d1ddf340c8 [V0 deprecation] Remove QKVCrossParallelLinear implementation (#26475) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2025-10-09 10:52:27 +00:00
Naveenraj KamalakannanandLuka Govedič e614ab7806 Separate MLAAttention class from Attention (#25103) 
Signed-off-by: Naveenraj Kamalakannan <therealnaveenkamal@gmail.com>
Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
 2025-10-08 17:11:11 -07:00
Harry Mellor d6953beb91 Convert formatting to use ruff instead of yapf + isort (#26247) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-05 07:06:22 -07:00
Jerry Zhang c31246800c Support RL online quantization with torchao (#23014) 
Signed-off-by: Jerry Zhang <jerryzh168@gmail.com>
 2025-10-01 16:39:29 -07:00
Cyrus Leung 89fa54e6f7 [Optimization] Use a cheaper cache key in get_model_architecture (#25682) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-09-25 17:54:20 -04:00
Cyrus Leung c60e6137f0 [Optimization] Avoid repeated model architecture conversion for pooling models (#25261) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-09-20 13:30:22 +08:00
Harry Mellor aed16879a9 Move ModelConfig from config/__init__.py to config/model.py (#25252) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-09-19 16:22:33 +00:00
Maximilien de Bayser e090b7b45b Enable conversion of multimodal models to pooling tasks (#24451) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
 2025-09-12 03:30:41 +00:00
Jee Jee Li 8d7f39b48c [Model] Remove quantized mixtral (#24437) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2025-09-08 11:02:14 -07:00
Cyrus LeungandHarry Mellor 86ae693f20 [Deprecation][2/N] Replace --task with --runner and --convert (#21470) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-07-27 19:42:40 -07:00