Commit Graph
66 Commits
Author SHA1 Message Date
Flora Feng 0d80979644 [Chore] Consolidate reasoning/tool parser attributes into unified Parser in chat serving (#45548)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-06-15 11:16:45 -04:00
Ting SUN 48df95c43e [Feature][Frontend] Report multimodal token counts in usage.prompt_tokens_details (#45458)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
2026-06-15 05:20:58 +00:00
Martin HickeyandIsotr0py 43f0e024bc [Render] Add /derender endpoints for disaggregated postprocessing (#43606)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
2026-06-13 13:55:33 +08:00
42ae5e7ac6 [Bugfix] Fix --enable-prompt-tokens-details omitting zero cached tokens (#44383)
Signed-off-by: Sasindharan Sankar <sasindharansankar@email.com>
Co-authored-by: Sasindharan Sankar <sasindharansankar@email.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-06-11 20:37:42 -07:00
yzong-rh e0871ad225 [Refactor] Chat Completions Streaming Harmony Refactor and Bugfixes (#45104)
Signed-off-by: Yifan Zong <yzong@redhat.com>
2026-06-12 01:09:47 +00:00
yzong-rh f712fd0d7d [Refactor] Chat Completions Harmony Refactor, non-streaming path. (#45171)
Signed-off-by: Yifan Zong <yzong@redhat.com>
2026-06-11 21:18:30 +00:00
Flora Feng 6aec99f030 [Refactor] Remove dead states from chat completion serving (#45081)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-06-09 22:20:15 -07:00
Flora Feng d82ac00923 [Refactor][Mistral] Extract parsing logic into MistralParser (#44596)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-06-10 00:12:23 +00:00
wang.yuqi d01d0b4646 [Frontend] Consolidate online serving utils. (#44479)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-06-04 06:49:31 +00:00
Flora Feng 68dafcca75 [Refactor] Unify reasoning + tool-call parsing behind Parser.parse() (#44267)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-06-02 15:11:42 +08:00
Flora Feng 9affc17a05 [Refactor] Move unstreamed tool-arg flush from serving layer to parser (#44017)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-06-02 10:37:43 +08:00
yzong-rh 46409fd2a1 [Fronten] Clean up stop_token_ids override for Harmony (#44009)
Signed-off-by: Yifan Zong <yzong@redhat.com>
2026-05-29 13:28:06 -07:00
Aaron HaoandIsotr0py 2d6b3489b9 [R3] Add routed experts to openai entrypoint (#38939)
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
2026-05-20 09:07:59 -07:00
aoshen02 8c79ad6580 Revert "[Core] Replace routing replay with device cache and async D2H pipeline" (#39917) (#42434)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
2026-05-13 23:49:01 -07:00
汪志鹏Wang, Zhipeng | RASIA <zhipeng.wang@rakuten.com>CursorChauncey
581b5e9afc [Frontend] Return rendered prompt text in chat completion response (#42052)
Signed-off-by: Wang, Zhipeng | RASIA <zhipeng.wang@rakuten.com>
Co-authored-by: Wang, Zhipeng | RASIA <zhipeng.wang@rakuten.com>
Co-authored-by: Cursor <cursor@cursor.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-05-11 13:53:39 +08:00
Dao007forever 3f5bd482f5 [Bugfix][KV Transfer][NIXL] Notify P node on pre-admission rejection to free stranded KV blocks (#41269) 2026-05-09 22:52:09 -07:00
TomerBN-Nvidia 8189a15914 [Core] Replace routing replay with device cache and async D2H pipeline (#39917)
Signed-off-by: Tomer Barnatan <tbarnatan@nvidia.com>
2026-05-07 11:24:56 -07:00
Flora Feng 8eb401134e [Refactor] Consolidate required/named tool_choice streaming into DelegatingParser (#41876)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-05-07 09:50:59 -07:00
Viktor Pus d5b31c954d [Bugfix] Account for truncate_prompt_tokens when computing max_tokens (#41800)
Signed-off-by: Viktor Pus <viktorpus@tenstorrent.com>
2026-05-06 16:10:17 +00:00
6467213a9f fix(openai): tolerate empty content in forced tool choice (#40148)
Signed-off-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com>
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com>
Co-authored-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-05-06 07:16:03 -07:00
Bugen Zhaoandwang.yuqi a07642667d [Bugfix] Pass reasoning parser kwargs to structured output (#41199)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-04-30 23:38:02 -07:00
Chauncey 3885d340a4 [Frontend]Responses API supports Tool/Function calling with streaming with named tool/function (#41110)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-04-29 09:11:27 +00:00
Chauncey 146f44b77d [Frontend]Responses API supports Tool/Function calling with streaming with required (#40700)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-04-27 19:36:58 -07:00
Simon MoandClaude ebf862c351 Add system_fingerprint field to OpenAI-compatible API responses (#40537)
Co-authored-by: Claude <noreply@anthropic.com>
2026-04-27 16:17:52 +08:00
Neil Schemenauer 56bdf85e10 [Feature] Avoid eager import of the "mistral_common" package. (#40043)
Signed-off-by: Neil Schemenauer <nas@arctrix.com>
2026-04-24 02:49:16 +00:00
Bugen Zhao 0210024ae7 [Bugfix] Pass effective chat template kwargs to reasoning parsers (#40460)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
2026-04-21 22:17:51 -07:00
Jared Wenandsfeng33 ceade1952c [BugFix] Support custom tool parsers when tool_choice is required and named function (#39870)
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
2026-04-17 16:38:10 +00:00
Julien Denize c0722f22de [Mistral Grammar] Fix tool and reasoning parsing (#39217)
Signed-off-by: juliendenize <julien.denize@mistral.ai>
2026-04-15 21:05:04 -07:00
Flora Feng db8a6d66bf [Refactor][Parser] Migrate chat completion auto-tool/reasoning/plain streaming to parse_delta (#39446)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-04-14 04:39:45 +00:00
yzong-rhandChauncey 3683fe6c06 [Bugfix] Fix shared-object aliasing in n>1 streaming with tool calls (#38158)
Signed-off-by: Yifan Zong <yzong@redhat.com>
Signed-off-by: Yifan <yzong@redhat.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-03-30 10:12:13 +00:00
Flora Feng e2db2b4234 [Tool Parser][1/3] Pass tools to ToolParser constructor (#38029)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-03-26 10:29:06 +08:00
Cyrus Leung ba2f0acc2d [Misc] Reorganize inputs (#35182)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-03-25 10:22:54 -07:00
Chauncey a32783bb35 [Bugfix] Fix IndexError when accessing prev_tool_call_arr in OpenAIToolParser (#37958)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-03-25 12:06:21 +08:00
DorBernsohn c63ca2b2e6 [Bugfix] Add Kimi-K2.5 reasoning/tool parser aliases and tool_call_id support (#37438)
Signed-off-by: DorBernsohn <dor.bernsohn@gmail.com>
2026-03-19 21:08:00 +08:00
Julien Denize 5db91f0aaf Fix some Mistral parser issues (#37209)
Signed-off-by: juliendenize <julien.denize@mistral.ai>
2026-03-17 00:08:56 +00:00
Andreas Karatzas 57a314d155 [CI][Bugfix] Fix 500 errors from priority overflow and TemplateError subclasses in schema fuzz tests (#37127)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
2026-03-16 05:27:21 +00:00
Sage a2268617cf [Frontend] Delegate preprocessing to OpenAIServingRender (#36483)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-03-13 00:39:43 -07:00
Chauncey 5a71cdd76e [Bugfix] Fix crash when tool_choice=required exceeds max_tokens (#36841)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-03-12 03:28:45 -07:00
Julien Denize a3ea760ea5 Add 'none' reasoning effort to ChatCompletionRequest (#36238)
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
2026-03-11 15:45:34 +00:00
Wentao Ye 941e52c298 [Refactor] Simplify chat_completion_full_generator for tool parsers (#35634)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-03-09 23:33:46 +08:00
Cyrus Leung 3ec2115015 [Frontend] Move warmup into Renderer (#36482)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-03-09 06:03:21 -07:00
Ning Xie 176c799f4c [openai api] log exception in exception handler (1/N) (#31164)
Signed-off-by: Andy Xie <andy.xning@gmail.com>
2026-03-05 16:00:12 +00:00
Martin VitandClaude Opus 4.6 8ebd872f50 [Tool Parser] Fix Qwen3Coder streaming parameter loss with speculative decode (#35615)
Signed-off-by: Martin Vit <martin@voipmonitor.org>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 09:40:37 +08:00
Neil Schemenauer 54e2f83d0a [Feature] Lazy import for the "mistral" tokenizer module. (#34651)
Signed-off-by: Neil Schemenauer <nas@arctrix.com>
2026-02-23 00:43:01 -08:00
Roger Wang 40f88d8318 [Bugfix] Fix Qwen3/Qwen3.5 Reasoning Parser (#34779)
Signed-off-by: Roger Wang <hey@rogerw.io>
2026-02-21 23:15:35 -08:00
Cyrus Leung 574fe75245 [Renderer] Move InputPreprocessor into Renderer (2/2) (#34560)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-02-17 05:29:01 -08:00
Almog Tavor 72d5951d02 [Bugfix] Treat generation_config max_tokens as default not ceiling (#34063)
Signed-off-by: almogtavor <almogtavor@gmail.com>
2026-02-16 07:58:24 -08:00
Cyrus Leung fb455ed547 [V0 Deprecation] Remove code related to per-request logits processors (#34400)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-02-12 20:44:28 +08:00
Cyrus Leung b96f7314b4 [Refactor] Pass Renderer to Input Processor (#34329)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-02-11 19:38:11 -08:00
Cyrus Leung 11a4c9d30d [Misc] Simplify get_max_tokens (#34036)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-02-07 00:59:49 -08:00