MOVZX and Hinata
5aa2ac64f8
[Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
...
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.
- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
render routes, and vllm/config/speech_to_text.py. GenerationTask is
generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
Multi, Offloading and Example connectors stay; production uses the
CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
arch gate, declarations and op registrations. The CUDA build never
defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
fail with "The model ... does not exist. Valid aliases: ...".
Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 21:06:19 +07:00
153a17166e
[Frontend] Add the Beam reasoning and tool parsers ( #62052 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Yoni Gozlan <74535834+yonigozlan@users.noreply.github.com >
Co-authored-by: neilkg <33635204+neilkg@users.noreply.github.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-16 14:08:29 -07:00
Flora Feng and Chauncey
80dd705c1d
[Frontend] Make finish_reason and Responses item status truthful ( #59324 )
...
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-10-15 10:12:02 +01:00
Bugen Zhao and Claude Opus 5.5
8fa5ae844a
[Frontend] Let an unset --tool-strict-level fall back to the tool parser's default ( #61328 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 15:48:38 +08:00
Kay Yan and Codex
c41b2639e2
[Bugfix][Responses API] Complete cache-write token accounting ( #60973 )
...
Signed-off-by: Kay Yan <kay.yan@daocloud.io >
Co-authored-by: Codex <noreply@openai.com >
2026-10-10 09:08:36 +00:00
wang.yuqi
c1692c85f1
[Misc] mv chat_utils.py to renderers. ( #60522 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-10-10 06:50:34 +00:00
Yufeng He and Flora Feng
6f75cc7d5d
[Bugfix][Responses API] Reuse streamed item ids in final harmony response ( #59859 )
...
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-04 03:01:04 +00:00
9bd993564f
[Frontend] Parse tool calls and reasoning from checkpoint response templates ( #58604 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Signed-off-by: Yoni Gozlan <yonigozlan@users.noreply.github.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-10-02 02:33:00 +00:00
Juan Pérez de Algaba and Cursor
cff08b461e
[Security] Fix chat template resource-exhaustion DoS (GHSA-4hhp-h66f-… ( #50300 )
...
Signed-off-by: jperezde <jperezde@redhat.com >
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-30 08:32:00 -07:00
Flora Feng
7ca31ef599
[Bugfix][Responses API] Build streamed final response from streamed items ( #59307 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-09-30 00:16:05 +00:00
Flora Feng
797505747f
[Bugfix][Frontend] Honor parallel_tool_calls=false in the Responses API ( #59298 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-09-29 22:36:11 +00:00
Flora Feng
3184226984
[Bugfix][Frontend] Count Responses reasoning tokens per tool round ( #58927 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-09-28 00:34:38 +00:00
7871963fcc
[Bugfix][Frontend] Respect max_output_tokens in the Harmony tool-call loop ( #58551 )
...
Signed-off-by: errmakov <ide404@gmail.com >
Co-authored-by: Du Bin <8174807+dubin555@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-25 14:00:39 -07:00
dee37d8911
[Frontend] Add --tool-strict-level for server-side control for structural tag activation ( #56268 )
...
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: lazymio <mio@lazym.io >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-09-18 04:32:39 +00:00
xinnywinne and Benjamin Chislett
fbf2c5e8be
[Frontend] Add per-request metrics to Responses API ( #55084 )
...
Signed-off-by: wxin <wxin@nvidia.com >
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
2026-09-16 22:52:05 +00:00
yzong-rh and Flora Feng
5a6ccc5892
[Bugfix] Add Responses cache_write_tokens for API compat and CC parity ( #57222 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-09-16 21:52:34 +00:00
Francisco Javier Arceo and OpenAI Codex
6a2a2bb02b
[Frontend] Add stateless /v1/responses/render endpoint ( #50195 )
...
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-07 12:50:42 +00:00
Nicolò Lucchesi
dafbef15a1
[Core] Add max_num_queued_reqs and max_num_queued_tokens for queue size management ( #49445 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-08-31 14:26:49 +00:00
wang.yuqi and Cyrus Leung
eeb549a74d
[Frontend] Move engine/protocol.py out openai folder ( #54492 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com >
2026-08-31 10:13:50 +00:00
Chauncey and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
ac2ae8798c
[Frontend] Support count_reasoning_tokens in the Streaming Parser Engine ( #45802 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-15 06:51:14 -07:00
Chauncey
22170354d8
[Feature] Parse request priority from HTTP header ( #51089 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
2026-08-05 21:10:29 -07:00
Karen Chung
f57123aa2d
feat(frontend): session id plumbing into requests ( #48048 )
...
Signed-off-by: Karen Chung <karenc@nvidia.com >
2026-08-03 22:18:58 +00:00
yzong-rh
dc818c198d
[GPT-OSS] Strict tool call and constrained decoding for Harmony ( #45560 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
2026-08-01 12:10:26 -04:00
yzong-rh and Ben Browning
e8b358b4c0
[Bugfix][Responses] Add tests for Chat Completions Responses API Render Parity ( #50334 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
Co-authored-by: Ben Browning <56071+bbrowning@users.noreply.github.com >
2026-07-31 15:45:40 -04:00
omerpaz95 and Or Ozeri
8df14cfc8c
[EC Connector] Add EC Transfer Params ( #42433 )
...
Signed-off-by: omerpaz95 <omerpaz95@gmail.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-07-12 14:35:33 +03:00
alberto and Chauncey
5715fde12c
[Feature][Parser] Support include_reasoning param for non-Harmony models ( #44301 )
...
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-07-10 15:34:02 +08:00
Zhong and zhongjing123
98ba9b9583
[Frontend] Support OpenAI Responses API namespace tools ( #47024 )
...
Signed-off-by: zhongjing123 <jimzhong5193@gmail.com >
Co-authored-by: zhongjing123 <jimzhong5193@gmail.com >
2026-07-06 06:21:27 +00:00
wang.yuqi
a14f57a3ac
[Frontend] Refine the entrypoint class's inheritance hierarchy. ( #47498 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-07-03 10:50:06 +00:00
yzong-rh
b1190d03cc
[Refactor][GPT-OSS] Harmony Responses API Refactor to use HarmonyParser ( #47185 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
2026-06-30 19:23:20 -04:00
Ben Browning
dccb412e2c
[Bugfix][Parser] Pass token IDs to parser.parse() in Responses API and batch serving ( #46843 )
...
Signed-off-by: Ben Browning <bbrownin@redhat.com >
2026-06-26 19:29:52 +00:00
Chauncey
84c2f9f0fb
[Frontend] Fix Kimi K2 tool call IDs for required tool choice ( #46344 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
2026-06-24 19:59:40 +00:00
yzong-rh and Ben Browning
f59db63732
[Bugfix] GPT-OSS Autodrop reasoning in Response API and cleanup ( #45048 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
Co-authored-by: Ben Browning <bbrownin@redhat.com >
2026-06-23 09:36:33 -04:00
wang.yuqi
31ca9504b1
[Frontend] Split ServingRender into renderer and entrypoint. ( #44285 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-23 11:19:09 +00:00
Chauncey and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
3ce5823762
[Refactor] Responses API parser state into conversation context ( #46030 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-23 13:42:58 +08:00
Flora Feng
6e4a547176
[Refactor] Deprecate ResponsesParser wrapper, inline parsing into ParsableContext ( #45431 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-06-12 16:15:41 -04:00
2043258dec
[Frontend] Support strict mode for tool calling ( #45003 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-12 07:51:48 +00:00
yzong-rh
f712fd0d7d
[Refactor] Chat Completions Harmony Refactor, non-streaming path. ( #45171 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
2026-06-11 21:18:30 +00:00
Flora Feng
3a04061701
[Refactor][Parser] Unify Response API to use parser.parse() like Chat Completion API ( #45190 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-06-11 04:37:51 +00:00
yzong-rh
703fb17b13
[Bugfix] GPT-OSS instruction rendering ( #44330 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
2026-06-05 13:52:32 -04:00
wang.yuqi
d01d0b4646
[Frontend] Consolidate online serving utils. ( #44479 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-04 06:49:31 +00:00
Flora Feng
209709a8c1
[Bugfix] Fix unstreamed tool call args dropped in Responses API streaming ( #44348 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-06-03 03:19:08 -07:00
alberto
e30313220c
[Parser] Migrate ResponsesParser to unified Parser interface ( #42977 )
...
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com >
2026-06-02 08:50:05 +00:00
yzong-rh
46409fd2a1
[Fronten] Clean up stop_token_ids override for Harmony ( #44009 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
2026-05-29 13:28:06 -07:00
Chauncey
87f12e5c7c
[Frontend]Responses API supports chat_template_kwargs ( #43761 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
2026-05-29 07:58:19 +00:00
Ignacio Sica and sfeng33
7909f82a45
[Bugfix][Frontend] streaming tool-call serializer drops first args chunk when name and args share a DeltaMessage ( #42683 )
...
Signed-off-by: ignaciosica <mignacio.sica@gmail.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: sfeng33 <4florafeng@gmail.com >
2026-05-28 05:20:55 +00:00
Ben Browning
0f69128a37
[Bugfix] Handle real-world gpt-oss tool call output in Harmony parsing ( #42454 )
...
Signed-off-by: Ben Browning <bbrownin@redhat.com >
2026-05-13 17:54:46 +00:00
Chauncey
617239b70c
[Frontend]Responses API supports chat_template_kwargs ( #42272 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
2026-05-11 11:59:39 +00:00
Dao007forever
3f5bd482f5
[Bugfix][KV Transfer][NIXL] Notify P node on pre-admission rejection to free stranded KV blocks ( #41269 )
2026-05-09 22:52:09 -07:00
Viktor Pus
d5b31c954d
[Bugfix] Account for truncate_prompt_tokens when computing max_tokens ( #41800 )
...
Signed-off-by: Viktor Pus <viktorpus@tenstorrent.com >
2026-05-06 16:10:17 +00:00
Bugen Zhao and wang.yuqi
a07642667d
[Bugfix] Pass reasoning parser kwargs to structured output ( #41199 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-04-30 23:38:02 -07:00