GuanLuo and sfeng33
6d1daa9fe5
[Bugfix][Frontend] Expose completion response schemas in OpenAPI ( #60926 )
...
Signed-off-by: Guan Luo <gluo@nvidia.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: sfeng33 <4florafeng@gmail.com >
2026-10-13 18:54:06 +00:00
wang.yuqi
c1692c85f1
[Misc] mv chat_utils.py to renderers. ( #60522 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-10-10 06:50:34 +00:00
Chauncey
fa5cfd48d5
[Refactor] Deduplicate trace header handling in BaseServing ( #60828 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
2026-10-10 05:27:58 +00:00
Harry Mellor and Claude Opus 5.5
21d9758459
Remove Python 3.10 support now that it is EOL ( #60402 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 02:30:52 -07:00
4187ade7b8
[Frontend] Consolidate rl entrypoints. ( #57849 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-07 21:37:11 -07:00
Sriraj and Flora Feng
db6e3cd8c4
[Bugfix][Frontend] Log the full request body when max_log_len is unset ( #60310 )
...
Signed-off-by: sricursion <sricursion@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-07 00:42:17 +00:00
czhu-cohere and aoshen02
6fddc9b21e
[Profiler] Add per-session profiling controls to Python and Rust frontends ( #57875 )
...
Signed-off-by: czhu <conway.zhu@cohere.com >
Signed-off-by: Conway Zhu <conway.zhu@cohere.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-06 05:21:19 +00:00
Tyrone and Harry Mellor
7867d6c52d
[Frontend] Name the served models in the model-not-found 404 ( #59889 )
...
Signed-off-by: TyroneNel <71038642+TyroneNel@users.noreply.github.com >
Signed-off-by: TyroneNel <Tyrone.Nel@gmail.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-10-04 20:45:51 +00:00
592c6f3fbc
[Bugfix][Pooling] Handle multimodal cache misses without crashing ( #58975 )
...
Signed-off-by: linchaoying6 <linchaoying6@jd.com >
Co-authored-by: linchaoying6 <linchaoying6@jd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-10-02 10:14:33 -04:00
713ec07655
[Frontend][RL] Track HTTP weight operation outcomes and concurrency ( #55781 )
...
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-30 09:19:43 +00:00
e5f3d08d72
[Frontend][RL] Align sleep-mode API responses and operation metrics ( #52864 )
...
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-29 23:16:38 -07:00
42d9f8ccdb
support rl feature : weight checker ( #51350 )
...
Signed-off-by: shiyuan680 <yangcheng104@huawei.com >
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: shiyuan680 <yangcheng104@huawei.com >
Co-authored-by: Ronald1995 <ronaldautomobile@163.com >
Co-authored-by: DeepSeek Harness <noreply@deepseek.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-30 03:13:43 +00:00
Flora Feng
797505747f
[Bugfix][Frontend] Honor parallel_tool_calls=false in the Responses API ( #59298 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-09-29 22:36:11 +00:00
cd94b21264
[Bugfix][Frontend] Return 400 for malformed RL dev route bodies ( #59236 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-29 16:46:56 +00:00
Cyrus Leung
379e9a1ea8
[Security] Harden message sanitization ( #58832 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-09-26 16:28:32 +00:00
Mark McLoughlin and AI Assistant
45d56c0baf
[Core] Model console logging as CLI configuration ( #57205 )
...
Add `--logging-config` CLI argument which can be supplied as
JSON or using dotted arguments. The `--log-level` argument
is provided for convenience, and `--log-config-file` is deprecated
in favor of `--logging-config.pylogging_config_file`.
Signed-off-by: Mark McLoughlin <markmc@redhat.com >
Co-authored-by: AI Assistant <noreply@openai.com >
2026-09-25 11:36:59 +01:00
Talor Abramovich
28282ffe34
[Feature][Frontend] Request JSON body debug logging on --enable-log-requests flag ( #58163 )
...
Signed-off-by: talora <talora@nvidia.com >
2026-09-24 08:08:40 +00:00
90531daf6d
[Core][Frontend] Bind KV-event publishers at port 0 and expose the bound endpoints ( #55844 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: mukeshbhandarkar <shell00035@gmail.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-18 18:11:05 -07:00
092bdd6d57
[Frontend][Core] Add release_kv_cache_memory() API ( #44890 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: AlanFokCo <alanfok2868@gmail.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com >
Co-authored-by: AlanFokCo <alanfok2868@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-09-18 07:12:12 +08:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
wenjinhust
0d8173d153
[Bugfix][Frontend] Lazy-import model_hosting_container_standards to prevent log suppression ( #51366 )
...
Signed-off-by: wenjinhust <wenjin.sh@huawei.com >
2026-09-16 08:25:49 +00:00
wang.yuqi
fe4d4109c2
[Frontend][last/N] Move all non-OpenAI content out of the OpenAI folder. ( #56369 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-09-11 05:33:26 +00:00
ea40bb9e90
[feature] Watermarked generation and detection (Gumbel-max algorithm) ( #54053 )
...
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-09 21:00:15 -07:00
erdholion and Ilir
131e0285f6
[Frontend] Warn when removed guided-decoding fields are present in a request ( #54285 )
...
Signed-off-by: Ilir <ljeci.ilir@gmail.com >
Co-authored-by: Ilir <ljeci.ilir@gmail.com >
2026-09-04 16:15:40 +00:00
lzhan011
fa99a6fea6
[Bugfix][Security] Bound the validation-error response body ( #54684 )
...
Signed-off-by: lzhan011 <35493221+lzhan011@users.noreply.github.com >
2026-09-01 06:51:16 +00:00
Nicolò Lucchesi
dafbef15a1
[Core] Add max_num_queued_reqs and max_num_queued_tokens for queue size management ( #49445 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-08-31 14:26:49 +00:00
wang.yuqi and Cyrus Leung
eeb549a74d
[Frontend] Move engine/protocol.py out openai folder ( #54492 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com >
2026-08-31 10:13:50 +00:00
Moe Huzaifa
3e833933dc
[Bugfix][Frontend] Keep credentials out of the Rust frontend launch log ( #53738 )
...
Signed-off-by: mhuzaifa3 <mhuzaifa3@outlook.com >
2026-08-25 16:32:12 +00:00
Ning Xie
23310bc3de
[sleep functionality] code refactor about sleep/wake_up ( #50431 )
...
Signed-off-by: Andy Xie <andy.xning@gmail.com >
2026-08-25 07:45:20 +00:00
wang.yuqi
735daf8ef6
[Frontend] Move cli_args.py and dp_supervisor.py out openai folder ( #53659 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-08-25 07:02:32 +00:00
Moe Huzaifa
d3e2888c75
[Bugfix][Frontend] Redact hf_token in the non-default args log ( #53625 )
...
Signed-off-by: mhuzaifa3 <mhuzaifa3@outlook.com >
2026-08-25 03:06:46 +00:00
wang.yuqi
eac636a7fa
[Frontend] Move api_server.py out openai folder ( #52131 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-08-19 09:43:46 +00:00
LuckyStep
9a9aa2b017
[Bugfix] Redact api_key in non-default args log ( #52523 )
...
Signed-off-by: LuckyStep <67696304+Andy365-365@users.noreply.github.com >
2026-08-19 15:52:53 +08:00
Itay Etelis and Itay Etelis
d36cc4254e
[Bugfix][Elastic EP] Reject scale below the minimum data parallel size ( #52702 )
...
Signed-off-by: Itay Etelis <itay.etelis@ibm.com >
Co-authored-by: Itay Etelis <itay.etelis@ibm.com >
2026-08-19 01:37:29 +00:00
Rajath Pai
b01728b088
[Bugfix] Return 4xx for client-caused errors in /detokenize ( #52622 )
...
Signed-off-by: rajathpi <rajathpai2000@gmail.com >
2026-08-18 07:07:15 +00:00
wang.yuqi
93550cc4cd
[Frontend] Consolidate entrypoint middleware ( #52309 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-08-17 06:17:15 +00:00
Kaif Kohari
a6a2a93f9b
[Bugfix][Frontend] Guard remaining before-validators against non-object JSON bodies ( #52528 )
...
Signed-off-by: Kaif <kaifkohari10@gmail.com >
2026-08-17 02:27:59 +00:00
wang.yuqi
b8165e5e58
[Frontend] Consolidate entrypoint exception handler ( #52261 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-08-14 09:27:24 +00:00
yang rui
152c913a63
[Frontend] Log output token IDs at DEBUG level ( #52098 )
...
Signed-off-by: ruirui6946 <142162413+ruirui6946@users.noreply.github.com >
2026-08-13 13:11:51 +00:00
andrewbcohere and Cyrus Leung
9c110fa522
[Frontend] Cohere chat v2 api support ( #47189 )
...
Signed-off-by: Andrew Berneshawi <andrewberneshawi@cohere.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-07-31 22:53:54 -07:00
Martin Hickey
0a31372e5f
[Frontend] Add detokenization streaming derender for disaggregated serving ( #47301 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-07-29 18:01:20 -07:00
Zach Zhu
58f9659397
[Frontend][Core] Standardize request error handling with VLLMError hierarchy ( #49665 )
...
Signed-off-by: Zach Zhu <zzqshu@126.com >
2026-07-29 04:20:40 +00:00
Itay Alroy
35efdf6b34
[Elastic EP] Async preparation ( #47288 )
...
Signed-off-by: Itay Alroy <ialroy@nvidia.com >
2026-07-28 05:19:13 -07:00
Shuolei Wang
9069a57139
[Core][Frontend] Add weight version tagging for RL rollouts ( #49040 )
...
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com >
Signed-off-by: Shuolei Wang <948904026@qq.com >
2026-07-28 14:35:29 +08:00
0b0bd2b5f6
[Feature] Add fault tolerance framework (simplified) for DP+EP external LB deployments ( #44428 )
...
Signed-off-by: fangyuchu <fangyuchu@qq.com >
Signed-off-by: a798347923 <2645302020@qq.com >
Signed-off-by: TianZhuo <2770730562@qq.com >
Signed-off-by: a798347923 <39047817+a798347923@users.noreply.github.com >
Signed-off-by: 205150940 <112750056+205150940@users.noreply.github.com >
Signed-off-by: w00689259 <wangzhuo66@huawei.com >
Signed-off-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com >
Signed-off-by: zWaNg3 <389750525@qq.com >
Signed-off-by: yzchang-plus <1078477584@qq.com >
Signed-off-by: Jade Zheng <zheng.shoujian@outlook.com >
Co-authored-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com >
Co-authored-by: a798347923 <2645302020@qq.com >
Co-authored-by: TianZhuo <2770730562@qq.com >
Co-authored-by: 205150940 <112750056+205150940@users.noreply.github.com >
Co-authored-by: a798347923 <39047817+a798347923@users.noreply.github.com >
Co-authored-by: w00689259 <wangzhuo66@huawei.com >
Co-authored-by: zWaNg3 <389750525@qq.com >
Co-authored-by: yzchang-plus <1078477584@qq.com >
Co-authored-by: Jade Zheng <zheng.shoujian@outlook.com >
2026-07-25 11:49:09 -04:00
Nick Hill
4809de7317
[Misc] Fix terminal output logo coloring ( #49344 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-07-22 00:21:29 +00:00
Ning Xie
b7c20d0cfa
[chore] adjust logo be more friendly to white background terminal ( #48938 )
...
Signed-off-by: Andy Xie <andy.xning@gmail.com >
2026-07-20 15:15:28 -07:00
83762b77b0
[Frontend] Add /abort_requests to the RLHF dev API router ( #47173 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-12 14:21:02 +08:00
fc1c548093
Runtime Draft Weight Update for Speculative Decoding ( #46725 )
...
Signed-off-by: vx120 <893600387@qq.com >
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: crp0128 <191679376@qq.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
2026-07-11 22:51:53 -07:00
