MOVZX and Hinata
5aa2ac64f8
[Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
...
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.
- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
render routes, and vllm/config/speech_to_text.py. GenerationTask is
generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
Multi, Offloading and Example connectors stay; production uses the
CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
arch gate, declarations and op registrations. The CUDA build never
defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
fail with "The model ... does not exist. Valid aliases: ...".
Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 21:06:19 +07:00
Jovan and OpenAI Codex
5be4266f5c
[Bugfix][Frontend] Preserve Cohere and Anthropic streaming errors ( #61947 )
...
Signed-off-by: Kevin <jia__kuan@163.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-10-17 06:10:16 +00:00
Shuolei Wang and aoshen02
b3a7e2ff6e
[RL][Frontend] Return weight-version spans on OpenAI completions and chat ( #61347 )
...
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-17 02:08:07 +00:00
3dcfaeb4e8
[Frontend] Apply each endpoint's top_logprobs cut in derender ( #59518 )
...
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com >
Co-authored-by: Yin Li <kevinli.ai.work@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-15 15:04:04 +00:00
Flora Feng and Chauncey
80dd705c1d
[Frontend] Make finish_reason and Responses item status truthful ( #59324 )
...
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-10-15 10:12:02 +01:00
Bugen Zhao and Claude Opus 5.5
8fa5ae844a
[Frontend] Let an unset --tool-strict-level fall back to the tool parser's default ( #61328 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 15:48:38 +08:00
wang.yuqi
c1692c85f1
[Misc] mv chat_utils.py to renderers. ( #60522 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-10-10 06:50:34 +00:00
dingdangmao and Flora Feng
bab0ac61f8
[Bugfix][Frontend] Forward the reasoning wiring to the engine for batch chat completions ( #59263 )
...
Signed-off-by: apex-mochen <2756823972@qq.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-08 19:32:46 +00:00
Ting SUN and Flora Feng
4a705c7b54
[Bugfix][Frontend] Preserve logprobs when top_logprobs is null ( #47838 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-05 22:43:31 +00:00
Yida Weng and Flora Feng
18c8a65edd
[BugFix][Frontend] Pass reasoning_ended through render → generate ( #60059 ) ( #60062 )
...
Signed-off-by: Yida Weng <84164537+YidaWeng@users.noreply.github.com >
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-05 20:39:25 +00:00
Stefan Wang and Robert Shaw
b0eb87fe49
[Bugfix][Frontend] Return streaming errors before the first token ( #40986 )
...
Signed-off-by: 1fanwang <1fannnw@gmail.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-10-04 16:42:49 +00:00
9bd993564f
[Frontend] Parse tool calls and reasoning from checkpoint response templates ( #58604 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Signed-off-by: Yoni Gozlan <yonigozlan@users.noreply.github.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-10-02 02:33:00 +00:00
Flora Feng
027b6f3a22
[Bugfix][Frontend] Use a fresh parser per choice in non-streaming chat completions ( #58939 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-09-28 02:01:35 +00:00
8b47e8b22c
[Perf][Frontend] Defer reasoning usage recounts for non-continuous chat streams ( #56067 )
...
Signed-off-by: Cheng Rui <286040359@qq.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-09-25 14:08:23 -07:00
f339e0e757
[Bugfix][Frontend] Keep logprobs of parser-suppressed streaming chunks ( #58583 )
...
Signed-off-by: errmakov <ide404@gmail.com >
Co-authored-by: Yanxiao Zhao <39199723+sdpkjc@users.noreply.github.com >
Co-authored-by: Prakhar Agarwal <270064960+agarwalprakhar2511@users.noreply.github.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-09-25 16:33:19 +00:00
Ting SUN
b33c8a657d
[Bugfix][Frontend] Keep length finish_reason for max_tokens-truncated streaming tool calls ( #46303 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
2026-09-23 14:31:10 +08:00
dee37d8911
[Frontend] Add --tool-strict-level for server-side control for structural tag activation ( #56268 )
...
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: lazymio <mio@lazym.io >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-09-18 04:32:39 +00:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
Nicolò Lucchesi
dafbef15a1
[Core] Add max_num_queued_reqs and max_num_queued_tokens for queue size management ( #49445 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-08-31 14:26:49 +00:00
wang.yuqi and Cyrus Leung
eeb549a74d
[Frontend] Move engine/protocol.py out openai folder ( #54492 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com >
2026-08-31 10:13:50 +00:00
andrewbcohere
f94dcde5c5
Fix Cohere ChatV2 citation and tool handling issues ( #52175 )
...
Signed-off-by: Andrew Berneshawi <andrewberneshawi@cohere.com >
2026-08-21 15:23:17 -07:00
7cfb97e337
[Frontend][Core][Spec Decode] Per-request acceptance stats in OpenAI API responses ( #48915 )
...
Signed-off-by: Matthew Kotila <7692737+matthewkotila@users.noreply.github.com >
Co-authored-by: Reed Meyerson <31574681+reed-meyerson@users.noreply.github.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-08-20 15:42:49 -07:00
Chauncey and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
ac2ae8798c
[Frontend] Support count_reasoning_tokens in the Streaming Parser Engine ( #45802 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-15 06:51:14 -07:00
7553aac77b
[Frontend] Add routed-experts prompt offset ( #51906 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-13 08:09:00 -07:00
Chauncey
22170354d8
[Feature] Parse request priority from HTTP header ( #51089 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
2026-08-05 21:10:29 -07:00
Karen Chung
f57123aa2d
feat(frontend): session id plumbing into requests ( #48048 )
...
Signed-off-by: Karen Chung <karenc@nvidia.com >
2026-08-03 22:18:58 +00:00
andrewbcohere and Cyrus Leung
9c110fa522
[Frontend] Cohere chat v2 api support ( #47189 )
...
Signed-off-by: Andrew Berneshawi <andrewberneshawi@cohere.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-07-31 22:53:54 -07:00
wang.yuqi
f5ffc59b6a
[Renderer] Warm up the renderer properly. ( #50408 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-07-31 11:46:53 +00:00
Julien Denize
1a20d23dab
[PARSER][Mistral] unified engine-based parser for reasoning and tool calls ( #48947 )
2026-07-30 15:38:00 +02:00
yzong-rh
a287eb163f
[Front-end] [Messages] Populate num_cache_creation_tokens ( #48535 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
2026-07-18 13:04:35 -04:00
langzhao-netizen and Claude
8b8af2caf7
[Frontend] Expose logprob_token_ids on Python OpenAI endpoints ( #43463 )
...
Signed-off-by: Lang Zhao <lang.zhao@galileo.ai >
Co-authored-by: Claude <noreply@anthropic.com >
2026-07-13 14:40:21 -07:00
omerpaz95 and Or Ozeri
8df14cfc8c
[EC Connector] Add EC Transfer Params ( #42433 )
...
Signed-off-by: omerpaz95 <omerpaz95@gmail.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-07-12 14:35:33 +03:00
alberto and Chauncey
5715fde12c
[Feature][Parser] Support include_reasoning param for non-Harmony models ( #44301 )
...
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-07-10 15:34:02 +08:00
86db6c3070
[Frontend] add per-request timing metrics field to response body of Chat/Completions APIs ( #46768 )
...
Signed-off-by: Nicholas Edelman <nedelman@nvidia.com >
Signed-off-by: Simon Mo <simon@inferact.ai >
Co-authored-by: Simon Mo <simon@inferact.ai >
Co-authored-by: GPT-5.5 <noreply@cursor.com >
2026-07-06 17:48:29 -07:00
wang.yuqi
a14f57a3ac
[Frontend] Refine the entrypoint class's inheritance hierarchy. ( #47498 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-07-03 10:50:06 +00:00
Chauncey
84c2f9f0fb
[Frontend] Fix Kimi K2 tool call IDs for required tool choice ( #46344 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
2026-06-24 19:59:40 +00:00
wang.yuqi
31ca9504b1
[Frontend] Split ServingRender into renderer and entrypoint. ( #44285 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-23 11:19:09 +00:00
Flora Feng
0d80979644
[Chore] Consolidate reasoning/tool parser attributes into unified Parser in chat serving ( #45548 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-06-15 11:16:45 -04:00
Ting SUN
48df95c43e
[Feature][Frontend] Report multimodal token counts in usage.prompt_tokens_details ( #45458 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
2026-06-15 05:20:58 +00:00
Martin Hickey and Isotr0py
43f0e024bc
[Render] Add /derender endpoints for disaggregated postprocessing ( #43606 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn >
2026-06-13 13:55:33 +08:00
42ae5e7ac6
[Bugfix] Fix --enable-prompt-tokens-details omitting zero cached tokens ( #44383 )
...
Signed-off-by: Sasindharan Sankar <sasindharansankar@email.com >
Co-authored-by: Sasindharan Sankar <sasindharansankar@email.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-06-11 20:37:42 -07:00
yzong-rh
e0871ad225
[Refactor] Chat Completions Streaming Harmony Refactor and Bugfixes ( #45104 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
2026-06-12 01:09:47 +00:00
yzong-rh
f712fd0d7d
[Refactor] Chat Completions Harmony Refactor, non-streaming path. ( #45171 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
2026-06-11 21:18:30 +00:00
Flora Feng
6aec99f030
[Refactor] Remove dead states from chat completion serving ( #45081 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-06-09 22:20:15 -07:00
Flora Feng
d82ac00923
[Refactor][Mistral] Extract parsing logic into MistralParser ( #44596 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-06-10 00:12:23 +00:00
wang.yuqi
d01d0b4646
[Frontend] Consolidate online serving utils. ( #44479 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-04 06:49:31 +00:00
Flora Feng
68dafcca75
[Refactor] Unify reasoning + tool-call parsing behind Parser.parse() ( #44267 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-06-02 15:11:42 +08:00
Flora Feng
9affc17a05
[Refactor] Move unstreamed tool-arg flush from serving layer to parser ( #44017 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-06-02 10:37:43 +08:00
yzong-rh
46409fd2a1
[Fronten] Clean up stop_token_ids override for Harmony ( #44009 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
2026-05-29 13:28:06 -07:00
Aaron Hao and Isotr0py
2d6b3489b9
[R3] Add routed experts to openai entrypoint ( #38939 )
...
Signed-off-by: ahao-anyscale <ahao@anyscale.com >
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn >
2026-05-20 09:07:59 -07:00