MOVZX and Hinata
5aa2ac64f8
[Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
...
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.
- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
render routes, and vllm/config/speech_to_text.py. GenerationTask is
generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
Multi, Offloading and Example connectors stay; production uses the
CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
arch gate, declarations and op registrations. The CUDA build never
defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
fail with "The model ... does not exist. Valid aliases: ...".
Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 21:06:19 +07:00
MOVZX and Hinata
2066d3143f
[Mini-VLLM] Cut non-OpenAI surfaces, dead kernels, Mistral stack
...
- Remove Anthropic, Cohere, SageMaker, decisions, structured-decisions,
rl, render-launcher and batch entrypoints; keep scale-out token routes
- Drop the Swagger static bundle and the /docs, /redoc routes
- Delete device-keyed tuning configs for fused_moe, quant utils, mamba
select-state and helion; no sm86 entry ever existed for any of them
- Remove ten unused rope variants plus the fbgemm_fp8 and fp_quant
quantizers and their config, metrics and loader plumbing
- Cut the CPU device build: csrc/cpu, cpu_extension.cmake, cpu
requirements and the setup.py cpu target path
- Delete 29 environment variables nothing reads and 14 chat-template
fallbacks for models this fork no longer registers
- Remove nine dead CUDA ops (deepseek_v4, kimi_k3, minimax, muse_spark,
dsv3 gemm) with their kernels, bindings, wrappers and CMake entries
- Delete the connector test suites that targeted removed connectors
- Cut the Mistral tokenizer and parser stack, its tests, and the
mistral_common and ijson dependency pins
- Restore the Qwen4Exp registry rows and nvidia/mtp.py that an earlier
cut deleted while keeping the model tree
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 17:10:56 +07:00
MOVZX and Hinata
e61dcb9143
[Mini-vLLM] SM86-only cut: models, backends, Rust, tests
...
- Keep only Qwen3.5, Qwen3.5-MTP and Qwen4Exp model paths; trim the model registry to 41 entries
- Delete idle attention backends (MLA, FlashInfer, ROCm-only, quant backends) and dead GDN CuTeDSL kernels
- Remove inc/quark quant loaders; keep compressed-tensors and Marlin
- Drop the Rust frontend, pooling entrypoints, and idle test suites
- Rebuild kernels for sm_80;sm_86 and strip the extensions
- Verified: full import sweep clean (optional deps only) and a dummy-load Qwen3.5 engine run on RTX 3090 via the Triton GDN path
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-18 19:19:56 +07:00
Subhash Polisetti
f425a21208
[Bugfix][Frontend] Support scalar priority and wire priority into LLM.chat() ( #57243 )
...
Signed-off-by: subhash polisetti <subhashr161347@gmail.com >
2026-10-11 10:03:33 +00:00
wang.yuqi
c1692c85f1
[Misc] mv chat_utils.py to renderers. ( #60522 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-10-10 06:50:34 +00:00
wang.yuqi
459764104d
[Frontend] Move offline_utils.py to entrypoints/common. ( #58052 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-10-08 09:56:05 +00:00
Harry Mellor and Claude Opus 5.5
21d9758459
Remove Python 3.10 support now that it is EOL ( #60402 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 02:30:52 -07:00
4187ade7b8
[Frontend] Consolidate rl entrypoints. ( #57849 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-07 21:37:11 -07:00
czhu-cohere and aoshen02
6fddc9b21e
[Profiler] Add per-session profiling controls to Python and Rust frontends ( #57875 )
...
Signed-off-by: czhu <conway.zhu@cohere.com >
Signed-off-by: Conway Zhu <conway.zhu@cohere.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-06 05:21:19 +00:00
e5f3d08d72
[Frontend][RL] Align sleep-mode API responses and operation metrics ( #52864 )
...
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-29 23:16:38 -07:00
Mark McLoughlin and AI Assistant
45d56c0baf
[Core] Model console logging as CLI configuration ( #57205 )
...
Add `--logging-config` CLI argument which can be supplied as
JSON or using dotted arguments. The `--log-level` argument
is provided for convenience, and `--log-config-file` is deprecated
in favor of `--logging-config.pylogging_config_file`.
Signed-off-by: Mark McLoughlin <markmc@redhat.com >
Co-authored-by: AI Assistant <noreply@openai.com >
2026-09-25 11:36:59 +01:00
Harry Mellor and Claude Opus 5.5
d636456b1f
[Frontend] Remove the slow tokenizer mode ( #58545 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-09-24 08:48:28 -07:00
092bdd6d57
[Frontend][Core] Add release_kv_cache_memory() API ( #44890 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: AlanFokCo <alanfok2868@gmail.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com >
Co-authored-by: AlanFokCo <alanfok2868@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-09-18 07:12:12 +08:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
50ba4bc6b2
[Feature] Mask Replay ( #49577 )
...
Signed-off-by: vx120 <893600387@qq.com >
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-13 08:44:31 +00:00
wang.yuqi
f5ffc59b6a
[Renderer] Warm up the renderer properly. ( #50408 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-07-31 11:46:53 +00:00
Shuolei Wang
9069a57139
[Core][Frontend] Add weight version tagging for RL rollouts ( #49040 )
...
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com >
Signed-off-by: Shuolei Wang <948904026@qq.com >
2026-07-28 14:35:29 +08:00
wang.yuqi
3e90d015ba
[Frontend] Overlap preprocessing and computation for pooling models offline inference ( #47699 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-07-16 14:00:20 +00:00
wangxiyuan
94c0ef3001
[Misc] Clean up "swap_space" ( #48549 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2026-07-14 04:43:45 +00:00
fc1c548093
Runtime Draft Weight Update for Speculative Decoding ( #46725 )
...
Signed-off-by: vx120 <893600387@qq.com >
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: crp0128 <191679376@qq.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
2026-07-11 22:51:53 -07:00
Aaron Hao and Claude Sonnet 4.6
77a9c5ae28
Weight sync refactor + move sparse nccl engine ( #44353 )
...
Signed-off-by: hao-aaron <ahao@anyscale.com >
Signed-off-by: haoaaron <ahao@anyscale.com >
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-07-01 01:25:19 -07:00
Sumanth R Hegde
7ff7f5c8eb
Revert "Fix Stale Encoder Cache After Weight Update" ( #46125 )
2026-06-20 07:09:09 -07:00
littlecircle0730
08985351f3
Fix Stale Encoder Cache After Weight Update ( #45093 )
...
Signed-off-by: littlecircle0730 <littlecircle0730@gmail.com >
2026-06-18 09:32:10 +00:00
Vic Wen
ef3af56a97
Fix LLM.wait_for_completion output type docstring ( #44617 )
...
Signed-off-by: viiccwen <viiccwen@gmail.com >
2026-06-05 00:16:38 -07:00
wang.yuqi
d01d0b4646
[Frontend] Consolidate online serving utils. ( #44479 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-04 06:49:31 +00:00
Siddharth Bedekar and OpenAI Codex
266b9d9c64
[Frontend][Core] Add sparse NCCL weight transfer support for in-place updates ( #40096 )
...
Signed-off-by: Siddharth Bedekar <bedeksid@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-06-01 15:37:30 -04:00
wang.yuqi and gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
d5cf7b4a2c
[Frontend] Split the offline inference APIs and utils. ( #43553 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-05-26 05:20:24 +00:00
Daoyuan Li
a6682d1d25
[Bugfix] Warn when renderer_num_workers has no effect on offline LLM ( #42905 )
...
Signed-off-by: Daoyuan Li <94409450+DaoyuanLi2816@users.noreply.github.com >
2026-05-20 21:35:08 -07:00
fadf5d332c
add enqueue all option to throughput benchmark ( #42975 )
...
Signed-off-by: Philip Maybank <pmaybank@amd.com >
Signed-off-by: pmaybank <113125070+pmaybank@users.noreply.github.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-19 20:16:02 -07:00
wang.yuqi
301d986473
[Frontend] Consolidate beam search by BeamSearchMixin. ( #42946 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-05-19 07:37:40 +00:00
Michael Goin and Claude
239b5ff30c
[Frontend] Add --spec-method/--spec-model/--spec-tokens CLI aliases ( #42476 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-05-18 17:22:27 -07:00
wang.yuqi
75fd68c7a5
[Entrypoints] Split the pooling offline API into PoolingOfflineMixin. ( #42267 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-05-15 08:05:57 +00:00
Michael Goin
8efd508204
[Quantization] Rework quantization_config to use QuantKey and allow for activation override ( #41566 )
2026-05-13 16:58:32 -04:00
Sumanth R Hegde
e3b65a5ba0
[feat] Add explicit /start_weight_update and /finish_weight_update APIs for weight transfer ( #39212 )
2026-05-08 18:03:33 -07:00
Luis 🚀 and Andrew Sansom
14043dfecd
feat: Enable prompt_embeds Content Part Support in vLLM Chat Completions API ( #40720 )
...
Signed-off-by: Luis Robaina <luis@protopia.ai >
Signed-off-by: Luis Robaina 🚀 <luisfabian1545@gmail.com >
Signed-off-by: LuisRobaina <luis@protopia.ai >
Co-authored-by: Andrew Sansom <qthequartermasterman@gmail.com >
2026-05-01 10:05:55 +08:00
wang.yuqi
a8208e6a81
[Examples] Resettle features examples. ( #40995 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-04-28 00:33:41 -07:00
9744b699ba
[Deprecate] Deprecate LLM.reward offline api, use LLM.encode instead. ( #40688 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com >
2026-04-24 05:37:50 +00:00
Matthew Bonanni and Tyler Michael Smith
96a85c5750
[Startup][UX] Enable CUDAGraph memory profiling by default ( #38284 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
2026-04-21 18:16:59 -04:00
d2e2e856ad
[Frontend] Remove frontend pooling multi task support. ( #37861 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-21 12:27:44 +00:00
wuyingjun
77fd2c8631
[Bugfix] Forward mm_processor_kwargs in offline generate APIs ( #40251 )
...
Signed-off-by: wuyingjun <wuyingjun_yewu@cmss.chinamobile.com >
2026-04-20 00:56:56 -07:00
38907e4391
[Frontend] Preserve structured output special tokens in offline LLM.chat ( #39352 )
...
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com >
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-04-18 19:46:07 -04:00
daiyu1111
a302a8fd1b
[Bugfix] Fix LLM priority normalization for single-string prompts ( #40011 )
...
Signed-off-by: daiyu1111 <2356690121@qq.com >
2026-04-16 07:56:06 -07:00
wang.yuqi and gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
4e8c3f1c19
[Frontend][last/5] Improve pooling entrypoints | clean up. ( #39675 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-04-16 07:53:23 -07:00
wang.yuqi
66c079ae83
[Frontend][4/n] Improve pooling entrypoints | pooling. ( #39153 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-04-09 10:09:45 +00:00
Vasiliy Kuznetsov
7b1a7423be
[Frontend] new online quantization frontend ( #38138 )
...
Signed-off-by: Vasiliy Kuznetsov <vasiliy@meta.com >
2026-04-03 11:58:39 -04:00
wang.yuqi
a9b4f07ba2
[Frontend] Re-enable running MaxSim on GPU ( #38620 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-04-03 00:03:13 +08:00
wang.yuqi
d9d21eb8e3
[Frontend][3/n] Improve pooling entrypoints | scoring. ( #28631 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-03-31 07:52:00 +00:00
Cyrus Leung
ba2f0acc2d
[Misc] Reorganize inputs ( #35182 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-03-25 10:22:54 -07:00
Harry Mellor
d215d1efca
[Mypy] Better fixes for the mypy issues in vllm/config ( #37902 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-03-25 06:14:43 -07:00
1b6cb920e6
[Deprecate] Deprecate pooling multi task support. ( #37956 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com >
2026-03-24 14:07:47 +00:00