Commit Graph
107 Commits
Author SHA1 Message Date
86db6c3070 [Frontend] add per-request timing metrics field to response body of Chat/Completions APIs (#46768)
Signed-off-by: Nicholas Edelman <nedelman@nvidia.com>
Signed-off-by: Simon Mo <simon@inferact.ai>
Co-authored-by: Simon Mo <simon@inferact.ai>
Co-authored-by: GPT-5.5 <noreply@cursor.com>
2026-07-06 17:48:29 -07:00
Ting SUN d2afe39647 [Bugfix][Frontend] Preserve default sampling params in batch chat (#47597)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
2026-07-04 19:06:39 +00:00
Taneem Ibrahim 2f21224527 [Misc] Update request-extras parity for batch chat completion (#47333)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
2026-07-04 10:19:04 -04:00
wang.yuqi a14f57a3ac [Frontend] Refine the entrypoint class's inheritance hierarchy. (#47498)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-07-03 10:50:06 +00:00
David FengandClaude 18f658bb31 [Bugfix][Frontend] Fix batch chat endpoint corrupting logprobs when return_token_ids is set (#47384)
Signed-off-by: David Feng <fenghourun@meta.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-07-03 03:01:34 -07:00
ganesh fca432e60a [Bugfix] Propagate default stop_token_ids to per-request SamplingParams (#35076)
Signed-off-by: sriganesh123 <arjulasriganesh@gmail.com>
2026-06-30 12:10:09 +08:00
RanranandBenjamin Chislett 09841ae705 [Render][Speculator] Add return_loss_mask to render endpoint for training data generation (#46846)
Signed-off-by: Ranran Haoran Zhang <ranzhang@redhat.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
2026-06-28 00:07:33 -07:00
Ben Browning dccb412e2c [Bugfix][Parser] Pass token IDs to parser.parse() in Responses API and batch serving (#46843)
Signed-off-by: Ben Browning <bbrownin@redhat.com>
2026-06-26 19:29:52 +00:00
Hyunkyun Moon 950ee4c2e4 [API] Add token offsets to render endpoints (/v1/.../render) (#44226)
Signed-off-by: HyunKyun Moon <mhg5303@gmail.com>
2026-06-26 05:02:52 -07:00
Chauncey 84c2f9f0fb [Frontend] Fix Kimi K2 tool call IDs for required tool choice (#46344)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-06-24 19:59:40 +00:00
wang.yuqi 31ca9504b1 [Frontend] Split ServingRender into renderer and entrypoint. (#44285)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-06-23 11:19:09 +00:00
6c427dd401 [BugFix] Omit empty tool_calls from OpenAI chat responses (#44105)
Signed-off-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com>
Signed-off-by: Chauncey <chaunceyjiang@gmail.com>
Co-authored-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-06-23 13:43:53 +08:00
Flora Feng 0d80979644 [Chore] Consolidate reasoning/tool parser attributes into unified Parser in chat serving (#45548)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-06-15 11:16:45 -04:00
Ting SUN 48df95c43e [Feature][Frontend] Report multimodal token counts in usage.prompt_tokens_details (#45458)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
2026-06-15 05:20:58 +00:00
Martin HickeyandIsotr0py 43f0e024bc [Render] Add /derender endpoints for disaggregated postprocessing (#43606)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
2026-06-13 13:55:33 +08:00
Chaunceycjackalmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2043258dec [Frontend] Support strict mode for tool calling (#45003)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-12 07:51:48 +00:00
42ae5e7ac6 [Bugfix] Fix --enable-prompt-tokens-details omitting zero cached tokens (#44383)
Signed-off-by: Sasindharan Sankar <sasindharansankar@email.com>
Co-authored-by: Sasindharan Sankar <sasindharansankar@email.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-06-11 20:37:42 -07:00
yzong-rh e0871ad225 [Refactor] Chat Completions Streaming Harmony Refactor and Bugfixes (#45104)
Signed-off-by: Yifan Zong <yzong@redhat.com>
2026-06-12 01:09:47 +00:00
yzong-rh f712fd0d7d [Refactor] Chat Completions Harmony Refactor, non-streaming path. (#45171)
Signed-off-by: Yifan Zong <yzong@redhat.com>
2026-06-11 21:18:30 +00:00
Flora Feng 6aec99f030 [Refactor] Remove dead states from chat completion serving (#45081)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-06-09 22:20:15 -07:00
Flora Feng d82ac00923 [Refactor][Mistral] Extract parsing logic into MistralParser (#44596)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-06-10 00:12:23 +00:00
wang.yuqi d01d0b4646 [Frontend] Consolidate online serving utils. (#44479)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-06-04 06:49:31 +00:00
Flora Feng 68dafcca75 [Refactor] Unify reasoning + tool-call parsing behind Parser.parse() (#44267)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-06-02 15:11:42 +08:00
Andreas Karatzas 54d0c36fff [CI] Stabilize OpenAI schema fuzzing for malformed structural tags (#44131)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
2026-06-01 19:56:15 -07:00
Flora Feng 9affc17a05 [Refactor] Move unstreamed tool-arg flush from serving layer to parser (#44017)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-06-02 10:37:43 +08:00
Umut Polat f46e6be169 [Misc] Use VLLMValidationError consistently in chat completion and completion protocol validators (#36254)
Signed-off-by: umut-polat <52835619+umut-polat@users.noreply.github.com>
2026-06-01 04:04:11 +00:00
yzong-rh 46409fd2a1 [Fronten] Clean up stop_token_ids override for Harmony (#44009)
Signed-off-by: Yifan Zong <yzong@redhat.com>
2026-05-29 13:28:06 -07:00
Ashwin GiridharanandChauncey 52a31ccecc [Bugfix] Map reasoning_effort to enable_thinking in chat template kwargs (#43401)
Signed-off-by: Ashwin Giridharan <girida@amazon.com>
Signed-off-by: Chauncey <chaunceyjiang@gmail.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-05-27 05:39:49 -07:00
linzm1007andmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 739af5c7e1 [Reasoning] [Bugfix] Reject invalid thinking_token_budget values (#43402)
Signed-off-by: linzm1007 <linzm1007@126.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-26 03:37:30 -07:00
Andreas Karatzas 5940590855 [ROCm][CI] Stabilize 400 error return code for invalid schema inputs (#43016)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
2026-05-24 10:06:49 +00:00
ruizhangandRui Zhang 60af5c16ee [Frontend] Add truncation side to OpenAI endpoints (#43260)
Signed-off-by: Rui Zhang <rza21.bc@gmail.com>
Signed-off-by: Rui Zhang <rui.zhang@globalrelay.net>
Co-authored-by: Rui Zhang <rui.zhang@globalrelay.net>
2026-05-21 20:32:31 -07:00
Ben Browningandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 346cf163a1 [Frontend] Normalize reasoning_content to reasoning for client compatibility (#42664)
Signed-off-by: Ben Browning <bbrownin@redhat.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-20 21:23:47 -07:00
Aaron HaoandIsotr0py 2d6b3489b9 [R3] Add routed experts to openai entrypoint (#38939)
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
2026-05-20 09:07:59 -07:00
aoshen02 8c79ad6580 Revert "[Core] Replace routing replay with device cache and async D2H pipeline" (#39917) (#42434)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
2026-05-13 23:49:01 -07:00
Ben Browning 0f69128a37 [Bugfix] Handle real-world gpt-oss tool call output in Harmony parsing (#42454)
Signed-off-by: Ben Browning <bbrownin@redhat.com>
2026-05-13 17:54:46 +00:00
汪志鹏Wang, Zhipeng | RASIA <zhipeng.wang@rakuten.com>CursorChauncey
581b5e9afc [Frontend] Return rendered prompt text in chat completion response (#42052)
Signed-off-by: Wang, Zhipeng | RASIA <zhipeng.wang@rakuten.com>
Co-authored-by: Wang, Zhipeng | RASIA <zhipeng.wang@rakuten.com>
Co-authored-by: Cursor <cursor@cursor.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-05-11 13:53:39 +08:00
Dao007forever 3f5bd482f5 [Bugfix][KV Transfer][NIXL] Notify P node on pre-admission rejection to free stranded KV blocks (#41269) 2026-05-09 22:52:09 -07:00
TomerBN-Nvidia 8189a15914 [Core] Replace routing replay with device cache and async D2H pipeline (#39917)
Signed-off-by: Tomer Barnatan <tbarnatan@nvidia.com>
2026-05-07 11:24:56 -07:00
Flora Feng 8eb401134e [Refactor] Consolidate required/named tool_choice streaming into DelegatingParser (#41876)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-05-07 09:50:59 -07:00
Viktor Pus d5b31c954d [Bugfix] Account for truncate_prompt_tokens when computing max_tokens (#41800)
Signed-off-by: Viktor Pus <viktorpus@tenstorrent.com>
2026-05-06 16:10:17 +00:00
6467213a9f fix(openai): tolerate empty content in forced tool choice (#40148)
Signed-off-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com>
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com>
Co-authored-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-05-06 07:16:03 -07:00
Bugen Zhaoandwang.yuqi a07642667d [Bugfix] Pass reasoning parser kwargs to structured output (#41199)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-04-30 23:38:02 -07:00
11b69129e2 [Frontend] Add defer_loading and tool_reference support for Anthropic and OpenAI APIs (#40190)
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-04-29 04:35:50 -07:00
Bugen Zhao 33f36d4260 [DSV4] Support max reasoning effort (#40982)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
2026-04-29 11:03:47 +00:00
Chauncey 3885d340a4 [Frontend]Responses API supports Tool/Function calling with streaming with named tool/function (#41110)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-04-29 09:11:27 +00:00
Chauncey 146f44b77d [Frontend]Responses API supports Tool/Function calling with streaming with required (#40700)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-04-27 19:36:58 -07:00
Simon MoandClaude ebf862c351 Add system_fingerprint field to OpenAI-compatible API responses (#40537)
Co-authored-by: Claude <noreply@anthropic.com>
2026-04-27 16:17:52 +08:00
Neil Schemenauer 56bdf85e10 [Feature] Avoid eager import of the "mistral_common" package. (#40043)
Signed-off-by: Neil Schemenauer <nas@arctrix.com>
2026-04-24 02:49:16 +00:00
Bugen Zhao 0210024ae7 [Bugfix] Pass effective chat template kwargs to reasoning parsers (#40460)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
2026-04-21 22:17:51 -07:00
Jared Wenandsfeng33 ceade1952c [BugFix] Support custom tool parsers when tool_choice is required and named function (#39870)
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
2026-04-17 16:38:10 +00:00