MOVZX and Hinata
5aa2ac64f8
[Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
...
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.
- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
render routes, and vllm/config/speech_to_text.py. GenerationTask is
generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
Multi, Offloading and Example connectors stay; production uses the
CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
arch gate, declarations and op registrations. The CUDA build never
defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
fail with "The model ... does not exist. Valid aliases: ...".
Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 21:06:19 +07:00
MOVZX and Hinata
5f3c7bd8ab
[Mini-VLLM] Remove the ROCm device path, debug routes and arch-gated tests
...
- Delete csrc/rocm, hipify, the _rocm_C extension and the setup.py rocm
target path; drop the rocm requirement files
- Delete the aiter, rdna3, skinny-gemm and ROCm fused-MoE kernel modules
and retarget the linear kernel registries away from them
- Reduce platforms/rocm.py to the GCN architecture predicates that live
code still imports, and delete the RocmPlatform class and plugin
- Drop the VLLM_SERVER_DEV_MODE debug surface: reset, collective_rpc,
server_info and sleep routes plus the log_error_stack flag
- Delete lm-format-enforcer, lark and otel-semantic-conventions-ai pins;
the live server process maps none of them
- Delete tests that only build architectures this fork no longer
registers, and test directories left without any test file
- Restore the tests/v1/kv_connector mock helpers that scheduler tests
import at module level
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 19:31:07 +07:00
MOVZX and Hinata
2066d3143f
[Mini-VLLM] Cut non-OpenAI surfaces, dead kernels, Mistral stack
...
- Remove Anthropic, Cohere, SageMaker, decisions, structured-decisions,
rl, render-launcher and batch entrypoints; keep scale-out token routes
- Drop the Swagger static bundle and the /docs, /redoc routes
- Delete device-keyed tuning configs for fused_moe, quant utils, mamba
select-state and helion; no sm86 entry ever existed for any of them
- Remove ten unused rope variants plus the fbgemm_fp8 and fp_quant
quantizers and their config, metrics and loader plumbing
- Cut the CPU device build: csrc/cpu, cpu_extension.cmake, cpu
requirements and the setup.py cpu target path
- Delete 29 environment variables nothing reads and 14 chat-template
fallbacks for models this fork no longer registers
- Remove nine dead CUDA ops (deepseek_v4, kimi_k3, minimax, muse_spark,
dsv3 gemm) with their kernels, bindings, wrappers and CMake entries
- Delete the connector test suites that targeted removed connectors
- Cut the Mistral tokenizer and parser stack, its tests, and the
mistral_common and ijson dependency pins
- Restore the Qwen4Exp registry rows and nvidia/mtp.py that an earlier
cut deleted while keeping the model tree
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 17:10:56 +07:00
MOVZX and Hinata
81074e94a3
[Mini-vLLM] Cut dead subsystems: models, entrypoints, backends, utils, tests
...
- Drop dead model modules, keeping files live imports require
- Remove anthropic, cohere, codegen entrypoints and batch STT modes
- Delete unused attention backends, MoE experts, and NPU/XPU util leaves
- Strip dead third_party trees and ROCm leaves with no CUDA path
- Remove mooncake EC connectors: dependency missing since branch base
- Re-audit verified by full import sweep and Qwen3.5 smoke test
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 00:20:48 +07:00
Jovan and OpenAI Codex
5be4266f5c
[Bugfix][Frontend] Preserve Cohere and Anthropic streaming errors ( #61947 )
...
Signed-off-by: Kevin <jia__kuan@163.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-10-17 06:10:16 +00:00
Shuolei Wang and aoshen02
b3a7e2ff6e
[RL][Frontend] Return weight-version spans on OpenAI completions and chat ( #61347 )
...
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-17 02:08:07 +00:00
Flora Feng and Chauncey
80dd705c1d
[Frontend] Make finish_reason and Responses item status truthful ( #59324 )
...
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-10-15 10:12:02 +01:00
1bc6e072b0
[Rust Frontend] Add per-request timing metrics to chat and completions ( #59314 )
...
Signed-off-by: wxin <wxin@nvidia.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 15:39:10 -07:00
1a3a7a8f01
[Bugfix] Treat max_tokens as output upper bound, not input-space reservation ( #57035 )
...
Signed-off-by: YuvrajSinghBhadoria2 <245782664+YuvrajSinghBhadoria2@users.noreply.github.com >
Signed-off-by: YuvrajSinghBhadoria2 <yuvrajsinghbhado2030@gmail.com >
Signed-off-by: Yuvraj Singh Bhadoria <yuvrajsinghbhado2030@gmail.com >
Co-authored-by: YuvrajSinghBhadoria2 <245782664+YuvrajSinghBhadoria2@users.noreply.github.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-10-10 01:15:22 +08:00
dingdangmao and Flora Feng
bab0ac61f8
[Bugfix][Frontend] Forward the reasoning wiring to the engine for batch chat completions ( #59263 )
...
Signed-off-by: apex-mochen <2756823972@qq.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-08 19:32:46 +00:00
Mai Hoàng Đức and Flora Feng
5cd8a2626b
[Bugfix][Structured Output] Cap JSON schema nesting to prevent 500s and API-server stalls ( #60036 )
...
Signed-off-by: Mazukiri <maihoangduc04@gmail.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-06 00:06:59 +00:00
Ting SUN and Flora Feng
4a705c7b54
[Bugfix][Frontend] Preserve logprobs when top_logprobs is null ( #47838 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-05 22:43:31 +00:00
Raphaël Rialland and Simon Veitner
c32495d58f
[watermark] compatibility validation ( #56801 )
...
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai >
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Simon Veitner <sveitner@redhat.com >
2026-10-05 14:37:13 -04:00
Stefan Wang and Robert Shaw
b0eb87fe49
[Bugfix][Frontend] Return streaming errors before the first token ( #40986 )
...
Signed-off-by: 1fanwang <1fannnw@gmail.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-10-04 16:42:49 +00:00
9bd993564f
[Frontend] Parse tool calls and reasoning from checkpoint response templates ( #58604 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Signed-off-by: Yoni Gozlan <yonigozlan@users.noreply.github.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-10-02 02:33:00 +00:00
Shijie Lyu
9aaef4296f
[Bugfix][Frontend] Check reused prompt token ids against the vocab before streaming ( #59555 )
...
Signed-off-by: Shijie Lyu <lshjhf@gmail.com >
2026-10-01 09:26:55 -07:00
c6843d397c
[Bugfix][Frontend] Validate reused prompt token ids; render messages for media and echo ( #55771 )
...
Signed-off-by: Shijie Lyu <sjlyu@amazon.com >
Co-authored-by: Shijie Lyu <sjlyu@amazon.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-29 10:36:22 +08:00
Flora Feng
6f65fde419
[Bugfix][Frontend] Apply Harmony adjust_request in batched chat completions ( #58958 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-09-28 04:12:04 +00:00
Robert Shaw and Robert Shaw
dfab504333
[Bugfix][Frontend] Detect Anthropic inline-system merge against the resolved chat template ( #58754 )
...
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Robert Shaw <robshaw@redhat.com >
2026-09-26 14:00:56 +00:00
Juan Pérez de Algaba and Cursor
31f2e70cd3
[Security] Gate per-request multimodal processor kwargs ( #58830 )
...
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-26 11:10:35 +00:00
8b47e8b22c
[Perf][Frontend] Defer reasoning usage recounts for non-continuous chat streams ( #56067 )
...
Signed-off-by: Cheng Rui <286040359@qq.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-09-25 14:08:23 -07:00
f339e0e757
[Bugfix][Frontend] Keep logprobs of parser-suppressed streaming chunks ( #58583 )
...
Signed-off-by: errmakov <ide404@gmail.com >
Co-authored-by: Yanxiao Zhao <39199723+sdpkjc@users.noreply.github.com >
Co-authored-by: Prakhar Agarwal <270064960+agarwalprakhar2511@users.noreply.github.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-09-25 16:33:19 +00:00
Ting SUN
b33c8a657d
[Bugfix][Frontend] Keep length finish_reason for max_tokens-truncated streaming tool calls ( #46303 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
2026-09-23 14:31:10 +08:00
Cyrus Leung
c8824a1e42
[MM] Move supports_multimodal_inputs and cache out of registry ( #57913 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-09-21 07:43:07 -07:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
18553c5c9f
[Kernel] Use Murmur3 RNG for Gumbel sampling ( #51367 )
...
Signed-off-by: freyfwt <110148731+freyfwt@users.noreply.github.com >
Signed-off-by: freyfwt <freytian1996@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com >
2026-09-15 23:52:26 -07:00
e962733e08
[Security] Validate cache salts before they reach LMCache ( #51444 )
...
Signed-off-by: Clinton Thomas <1033162+KernelClint@users.noreply.github.com >
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-09-05 03:56:06 +00:00
Bugen Zhao
d6bce42983
[Rust Frontend] Report reasoning tokens in chat completion usage ( #54883 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-09-03 16:47:24 -07:00
Pratham Shirbhate and Cyrus Leung
0ad5652a52
[Bugfix][Frontend] Restore the chat template content format mismatch warning ( #54622 )
...
Signed-off-by: Pratham Shirbhate <prathamshirbhate1909@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-09-01 18:17:54 +00:00
Nicolò Lucchesi
dafbef15a1
[Core] Add max_num_queued_reqs and max_num_queued_tokens for queue size management ( #49445 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-08-31 14:26:49 +00:00
wang.yuqi and Cyrus Leung
eeb549a74d
[Frontend] Move engine/protocol.py out openai folder ( #54492 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com >
2026-08-31 10:13:50 +00:00
Kaif Kohari and Cyrus Leung
7fbfca2670
[Bugfix][Frontend] Only echo the assistant turn in batched chat completions ( #52529 )
...
Signed-off-by: Kaif <kaifkohari10@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-08-29 09:44:17 -07:00
Chauncey and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
ac2ae8798c
[Frontend] Support count_reasoning_tokens in the Streaming Parser Engine ( #45802 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-15 06:51:14 -07:00
Andreas Karatzas and OpenAI Codex
1ab2801dde
[ROCm] Remove stale SDPA and skinny GEMM workarounds ( #50907 )
...
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-11 14:19:16 -04:00
Tarun Kumar
48bada6ea4
Fix chat completion 500 on non-object JSON bodies ( #51654 )
...
Signed-off-by: Tarun Kumar <takumar@redhat.com >
2026-08-11 01:31:19 +00:00
Chauncey and Nick Hill
72c0d67657
[Feat] Support thinking_token_budget in Model Runner V2 ( #46727 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-07 03:36:18 +00:00
Seiji Eicher
6370e53f24
[Frontend] Reuse prefill token ids on the decode chat path for disaggregated serving ( #48145 )
...
Signed-off-by: Seiji Eicher <seiji@anyscale.com >
2026-07-29 10:02:57 +02:00
Zach Zhu
58f9659397
[Frontend][Core] Standardize request error handling with VLLMError hierarchy ( #49665 )
...
Signed-off-by: Zach Zhu <zzqshu@126.com >
2026-07-29 04:20:40 +00:00
Harry Mellor and Andreas Karatzas
ca0defa343
Make bare hugging_face imports forbidden ( #49726 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-07-25 04:18:39 -07:00
hcl
530ee36a0d
fix(openai): reject non-numeric logprobs with 400 instead of 500 ( #49144 )
...
Signed-off-by: Chenglun Hu <chenglunhu@gmail.com >
2026-07-20 10:04:50 +00:00
yzong-rh
a287eb163f
[Front-end] [Messages] Populate num_cache_creation_tokens ( #48535 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
2026-07-18 13:04:35 -04:00
26c909ed74
[Model] Support TranslateGemma-12b-it ( #41599 )
...
Signed-off-by: Zhang Jian <jianmusings@gmail.com >
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-07-17 07:17:59 +00:00
langzhao-netizen and Claude
8b8af2caf7
[Frontend] Expose logprob_token_ids on Python OpenAI endpoints ( #43463 )
...
Signed-off-by: Lang Zhao <lang.zhao@galileo.ai >
Co-authored-by: Claude <noreply@anthropic.com >
2026-07-13 14:40:21 -07:00
alberto and Chauncey
5715fde12c
[Feature][Parser] Support include_reasoning param for non-Harmony models ( #44301 )
...
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-07-10 15:34:02 +08:00
Muhammad Fawaz and Mahad Durrani
d35eba302f
[Bugfix] Avoid leaking Pydantic repr in tool_choice error message ( #47028 )
...
Signed-off-by: muhammadfawaz1 <135441198+muhammadfawaz1@users.noreply.github.com >
Co-authored-by: Mahad Durrani <114791389+mahadrehmann@users.noreply.github.com >
2026-07-08 15:00:59 +08:00
86db6c3070
[Frontend] add per-request timing metrics field to response body of Chat/Completions APIs ( #46768 )
...
Signed-off-by: Nicholas Edelman <nedelman@nvidia.com >
Signed-off-by: Simon Mo <simon@inferact.ai >
Co-authored-by: Simon Mo <simon@inferact.ai >
Co-authored-by: GPT-5.5 <noreply@cursor.com >
2026-07-06 17:48:29 -07:00
akii96 and Andreas Karatzas
5769a7382c
[ROCm][CI][Bugfix] Fix flaky parallel tool-call streaming (test assertion + Mistral/Granite parsers) ( #47550 )
...
Signed-off-by: Aakif Nawaz <aakif.nawaz@amd.com >
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-07-06 19:06:19 -04:00
David Feng and Claude
18f658bb31
[Bugfix][Frontend] Fix batch chat endpoint corrupting logprobs when return_token_ids is set ( #47384 )
...
Signed-off-by: David Feng <fenghourun@meta.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-07-03 03:01:34 -07:00
Muhammad Fawaz and Mahad Durrani
97b5ce5c39
[Bugfix] Raise VLLMValidationError for non-integer logit_bias keys ( #46612 )
...
Signed-off-by: muhammadfawaz1 <135441198+muhammadfawaz1@users.noreply.github.com >
Co-authored-by: Mahad Durrani <114791389+mahadrehmann@users.noreply.github.com >
2026-06-30 06:18:59 +00:00