97 Commits
Author SHA1 Message Date
153a17166e [Frontend] Add the Beam reasoning and tool parsers (#62052)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Yoni Gozlan <74535834+yonigozlan@users.noreply.github.com>
Co-authored-by: neilkg <33635204+neilkg@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 14:08:29 -07:00
Flora FengandChauncey 80dd705c1d [Frontend] Make finish_reason and Responses item status truthful (#59324)
Signed-off-by: Flora Feng <4florafeng@gmail.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-10-15 10:12:02 +01:00
Bugen ZhaoandClaude Opus 5.5 8fa5ae844a [Frontend] Let an unset --tool-strict-level fall back to the tool parser's default (#61328)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:48:38 +08:00
Kay YanandCodex c41b2639e2 [Bugfix][Responses API] Complete cache-write token accounting (#60973)
Signed-off-by: Kay Yan <kay.yan@daocloud.io>
Co-authored-by: Codex <noreply@openai.com>
2026-10-10 09:08:36 +00:00
wang.yuqi c1692c85f1 [Misc] mv chat_utils.py to renderers. (#60522)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-10-10 06:50:34 +00:00
Yufeng HeandFlora Feng 6f75cc7d5d [Bugfix][Responses API] Reuse streamed item ids in final harmony response (#59859)
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-04 03:01:04 +00:00
9bd993564f [Frontend] Parse tool calls and reasoning from checkpoint response templates (#58604)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yoni Gozlan <yonigozlan@users.noreply.github.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
2026-10-02 02:33:00 +00:00
Juan Pérez de AlgabaandCursor cff08b461e [Security] Fix chat template resource-exhaustion DoS (GHSA-4hhp-h66f-… (#50300)
Signed-off-by: jperezde <jperezde@redhat.com>
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-30 08:32:00 -07:00
Flora Feng 7ca31ef599 [Bugfix][Responses API] Build streamed final response from streamed items (#59307)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-09-30 00:16:05 +00:00
Flora Feng 797505747f [Bugfix][Frontend] Honor parallel_tool_calls=false in the Responses API (#59298)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-09-29 22:36:11 +00:00
Flora Feng 3184226984 [Bugfix][Frontend] Count Responses reasoning tokens per tool round (#58927)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-09-28 00:34:38 +00:00
Denis ErmakovDu BinClaude Opus 5.5Flora Fengmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
7871963fcc [Bugfix][Frontend] Respect max_output_tokens in the Harmony tool-call loop (#58551)
Signed-off-by: errmakov <ide404@gmail.com>
Co-authored-by: Du Bin <8174807+dubin555@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-25 14:00:39 -07:00
dee37d8911 [Frontend] Add --tool-strict-level for server-side control for structural tag activation (#56268)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: lazymio <mio@lazym.io>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
2026-09-18 04:32:39 +00:00
xinnywinneandBenjamin Chislett fbf2c5e8be [Frontend] Add per-request metrics to Responses API (#55084)
Signed-off-by: wxin <wxin@nvidia.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com>
2026-09-16 22:52:05 +00:00
yzong-rhandFlora Feng 5a6ccc5892 [Bugfix] Add Responses cache_write_tokens for API compat and CC parity (#57222)
Signed-off-by: Yifan Zong <yzong@redhat.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-09-16 21:52:34 +00:00
Francisco Javier ArceoandOpenAI Codex 6a2a2bb02b [Frontend] Add stateless /v1/responses/render endpoint (#50195)
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-07 12:50:42 +00:00
Nicolò Lucchesi dafbef15a1 [Core] Add max_num_queued_reqs and max_num_queued_tokens for queue size management (#49445)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-08-31 14:26:49 +00:00
wang.yuqiandCyrus Leung eeb549a74d [Frontend] Move engine/protocol.py out openai folder (#54492)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
2026-08-31 10:13:50 +00:00
Chaunceyandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> ac2ae8798c [Frontend] Support count_reasoning_tokens in the Streaming Parser Engine (#45802)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-15 06:51:14 -07:00
Chauncey 22170354d8 [Feature] Parse request priority from HTTP header (#51089)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-08-05 21:10:29 -07:00
Karen Chung f57123aa2d feat(frontend): session id plumbing into requests (#48048)
Signed-off-by: Karen Chung <karenc@nvidia.com>
2026-08-03 22:18:58 +00:00
yzong-rh dc818c198d [GPT-OSS] Strict tool call and constrained decoding for Harmony (#45560)
Signed-off-by: Yifan Zong <yzong@redhat.com>
2026-08-01 12:10:26 -04:00
yzong-rhandBen Browning e8b358b4c0 [Bugfix][Responses] Add tests for Chat Completions Responses API Render Parity (#50334)
Signed-off-by: Yifan Zong <yzong@redhat.com>
Co-authored-by: Ben Browning <56071+bbrowning@users.noreply.github.com>
2026-07-31 15:45:40 -04:00
omerpaz95andOr Ozeri 8df14cfc8c [EC Connector] Add EC Transfer Params (#42433)
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-12 14:35:33 +03:00
albertoandChauncey 5715fde12c [Feature][Parser] Support include_reasoning param for non-Harmony models (#44301)
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-07-10 15:34:02 +08:00
Zhongandzhongjing123 98ba9b9583 [Frontend] Support OpenAI Responses API namespace tools (#47024)
Signed-off-by: zhongjing123 <jimzhong5193@gmail.com>
Co-authored-by: zhongjing123 <jimzhong5193@gmail.com>
2026-07-06 06:21:27 +00:00
wang.yuqi a14f57a3ac [Frontend] Refine the entrypoint class's inheritance hierarchy. (#47498)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-07-03 10:50:06 +00:00
yzong-rh b1190d03cc [Refactor][GPT-OSS] Harmony Responses API Refactor to use HarmonyParser (#47185)
Signed-off-by: Yifan Zong <yzong@redhat.com>
2026-06-30 19:23:20 -04:00
Ben Browning dccb412e2c [Bugfix][Parser] Pass token IDs to parser.parse() in Responses API and batch serving (#46843)
Signed-off-by: Ben Browning <bbrownin@redhat.com>
2026-06-26 19:29:52 +00:00
Chauncey 84c2f9f0fb [Frontend] Fix Kimi K2 tool call IDs for required tool choice (#46344)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-06-24 19:59:40 +00:00
yzong-rhandBen Browning f59db63732 [Bugfix] GPT-OSS Autodrop reasoning in Response API and cleanup (#45048)
Signed-off-by: Yifan Zong <yzong@redhat.com>
Co-authored-by: Ben Browning <bbrownin@redhat.com>
2026-06-23 09:36:33 -04:00
wang.yuqi 31ca9504b1 [Frontend] Split ServingRender into renderer and entrypoint. (#44285)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-06-23 11:19:09 +00:00
Chaunceyandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 3ce5823762 [Refactor] Responses API parser state into conversation context (#46030)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-23 13:42:58 +08:00
Flora Feng 6e4a547176 [Refactor] Deprecate ResponsesParser wrapper, inline parsing into ParsableContext (#45431)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-06-12 16:15:41 -04:00
Chaunceycjackalmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2043258dec [Frontend] Support strict mode for tool calling (#45003)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-12 07:51:48 +00:00
yzong-rh f712fd0d7d [Refactor] Chat Completions Harmony Refactor, non-streaming path. (#45171)
Signed-off-by: Yifan Zong <yzong@redhat.com>
2026-06-11 21:18:30 +00:00
Flora Feng 3a04061701 [Refactor][Parser] Unify Response API to use parser.parse() like Chat Completion API (#45190)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-06-11 04:37:51 +00:00
yzong-rh 703fb17b13 [Bugfix] GPT-OSS instruction rendering (#44330)
Signed-off-by: Yifan Zong <yzong@redhat.com>
2026-06-05 13:52:32 -04:00
wang.yuqi d01d0b4646 [Frontend] Consolidate online serving utils. (#44479)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-06-04 06:49:31 +00:00
Flora Feng 209709a8c1 [Bugfix] Fix unstreamed tool call args dropped in Responses API streaming (#44348)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-06-03 03:19:08 -07:00
alberto e30313220c [Parser] Migrate ResponsesParser to unified Parser interface (#42977)
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
2026-06-02 08:50:05 +00:00
yzong-rh 46409fd2a1 [Fronten] Clean up stop_token_ids override for Harmony (#44009)
Signed-off-by: Yifan Zong <yzong@redhat.com>
2026-05-29 13:28:06 -07:00
Chauncey 87f12e5c7c [Frontend]Responses API supports chat_template_kwargs (#43761)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-05-29 07:58:19 +00:00
Ignacio Sicaandsfeng33 7909f82a45 [Bugfix][Frontend] streaming tool-call serializer drops first args chunk when name and args share a DeltaMessage (#42683)
Signed-off-by: ignaciosica <mignacio.sica@gmail.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
2026-05-28 05:20:55 +00:00
Ben Browning 0f69128a37 [Bugfix] Handle real-world gpt-oss tool call output in Harmony parsing (#42454)
Signed-off-by: Ben Browning <bbrownin@redhat.com>
2026-05-13 17:54:46 +00:00
Chauncey 617239b70c [Frontend]Responses API supports chat_template_kwargs (#42272)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-05-11 11:59:39 +00:00
Dao007forever 3f5bd482f5 [Bugfix][KV Transfer][NIXL] Notify P node on pre-admission rejection to free stranded KV blocks (#41269) 2026-05-09 22:52:09 -07:00
Viktor Pus d5b31c954d [Bugfix] Account for truncate_prompt_tokens when computing max_tokens (#41800)
Signed-off-by: Viktor Pus <viktorpus@tenstorrent.com>
2026-05-06 16:10:17 +00:00
Bugen Zhaoandwang.yuqi a07642667d [Bugfix] Pass reasoning parser kwargs to structured output (#41199)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-04-30 23:38:02 -07:00
Chauncey 146f44b77d [Frontend]Responses API supports Tool/Function calling with streaming with required (#40700)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-04-27 19:36:58 -07:00