wang.yuqi
eac636a7fa
[Frontend] Move api_server.py out openai folder ( #52131 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-08-19 09:43:46 +00:00
wang.yuqi
93550cc4cd
[Frontend] Consolidate entrypoint middleware ( #52309 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-08-17 06:17:15 +00:00
wang.yuqi
b8165e5e58
[Frontend] Consolidate entrypoint exception handler ( #52261 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-08-14 09:27:24 +00:00
wang.yuqi
f5ffc59b6a
[Renderer] Warm up the renderer properly. ( #50408 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-07-31 11:46:53 +00:00
Zach Zhu
58f9659397
[Frontend][Core] Standardize request error handling with VLLMError hierarchy ( #49665 )
...
Signed-off-by: Zach Zhu <zzqshu@126.com >
2026-07-29 04:20:40 +00:00
0b0bd2b5f6
[Feature] Add fault tolerance framework (simplified) for DP+EP external LB deployments ( #44428 )
...
Signed-off-by: fangyuchu <fangyuchu@qq.com >
Signed-off-by: a798347923 <2645302020@qq.com >
Signed-off-by: TianZhuo <2770730562@qq.com >
Signed-off-by: a798347923 <39047817+a798347923@users.noreply.github.com >
Signed-off-by: 205150940 <112750056+205150940@users.noreply.github.com >
Signed-off-by: w00689259 <wangzhuo66@huawei.com >
Signed-off-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com >
Signed-off-by: zWaNg3 <389750525@qq.com >
Signed-off-by: yzchang-plus <1078477584@qq.com >
Signed-off-by: Jade Zheng <zheng.shoujian@outlook.com >
Co-authored-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com >
Co-authored-by: a798347923 <2645302020@qq.com >
Co-authored-by: TianZhuo <2770730562@qq.com >
Co-authored-by: 205150940 <112750056+205150940@users.noreply.github.com >
Co-authored-by: a798347923 <39047817+a798347923@users.noreply.github.com >
Co-authored-by: w00689259 <wangzhuo66@huawei.com >
Co-authored-by: zWaNg3 <389750525@qq.com >
Co-authored-by: yzchang-plus <1078477584@qq.com >
Co-authored-by: Jade Zheng <zheng.shoujian@outlook.com >
2026-07-25 11:49:09 -04:00
Zach Zhu
5d5fab0061
[Bugfix][Frontend] Fix http_requests_total metric recording some 4xx errors as 5xx ( #44303 )
...
Signed-off-by: Zach Zhu <zzqshu@126.com >
2026-07-08 05:33:21 +00:00
Martin Hickey
f7fc0ca993
[Frontend] Add endpoint plugins framework ( #47454 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-07-08 10:00:41 +08:00
Alexis K.
40cc2e8327
[Bugfix] Return HTTP 422 for unprocessable image URLs instead of 500 ( #47165 )
...
Signed-off-by: Alexis Kinsella <alexis.kinsella@gmail.com >
2026-07-06 11:56:23 +00:00
Bugen Zhao
ab3b6d97aa
[Frontend] Limit SO_REUSEPORT to multi-worker serving ( #47529 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-04 01:26:24 +00:00
wang.yuqi
3483240b7e
[Frontend] Consolidate scale out entrypoints ( #44512 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-29 03:18:53 -07:00
wang.yuqi
31ca9504b1
[Frontend] Split ServingRender into renderer and entrypoint. ( #44285 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-23 11:19:09 +00:00
wang.yuqi
1eb2cc961e
[Frontend] Refactor ServingTokenization entrypoint. ( #46022 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-22 06:27:58 +00:00
aoshen02 and Martin Hickey
f57ac274b2
[Render] Add reasoning/tool parsing to /derender + fix byte-fallback FFFD ( #45919 )
...
Signed-off-by: aoshen524 <aoshen524@gmail.com >
Co-authored-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-06-20 19:43:32 -04:00
2043258dec
[Frontend] Support strict mode for tool calling ( #45003 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-12 07:51:48 +00:00
yzong-rh and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
6ec7dcd641
[Frontend][Metrics] Add vllm:tool_call_parser_invocations_total Prometheus metric ( #44448 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-10 10:29:11 -04:00
Andreas Karatzas
f0f6805d8a
[CI] Stabilize the multi-audio OpenAI server path ( #44051 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-06-07 15:54:32 +08:00
wang.yuqi
d01d0b4646
[Frontend] Consolidate online serving utils. ( #44479 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-04 06:49:31 +00:00
wang.yuqi
b623f7ea95
[Frontend] Consolidate dev entrypoints. ( #44170 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-02 06:30:21 -07:00
wang.yuqi
0910f7e0e1
[Frontend] Resettle generative scoring entrypoint. ( #44153 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-01 07:54:59 +00:00
Andreas Karatzas
5940590855
[ROCm][CI] Stabilize 400 error return code for invalid schema inputs ( #43016 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-05-24 10:06:49 +00:00
0ddd7dd656
[Frontend] DP Supervisor ( #40841 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com >
Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: robertgshaw2-redhat <robertgshaw2@gmail.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-05-21 20:33:16 -07:00
Nick Hill and Bugen Zhao
f2ace1d57d
[Frontend][RFC] Rust front-end integration ( #40848 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
2026-05-21 12:24:48 +08:00
wang.yuqi and gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
d37e25ffbe
[Frontend] Consolidate Speech to Text entrypoints. ( #42370 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-05-12 07:06:57 +00:00
844df54269
feat: update xgrammar==0.2.0 to use structural tags for strict tool calling + reasoning for more models ( #40894 )
...
Signed-off-by: Yuchuan <yuchuan.7streams@gmail.com >
Signed-off-by: Michael Goin <mgoin64@gmail.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Signed-off-by: Ubospica <ubospica@gmail.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: Ubospica <ubospica@gmail.com >
Co-authored-by: sfeng33 <4florafeng@gmail.com >
2026-05-04 12:45:24 -07:00
wang.yuqi
3975eb6de6
Revert "[Startup] Parallelize torch/transformers import + weight prefetch + forkserver prewarm" ( #40438 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-04-21 08:47:18 +00:00
Simon Mo
8256833fe6
[Startup] Parallelize torch/transformers import + weight prefetch + forkserver prewarm ( #40331 )
...
Signed-off-by: simon-mo <simon@inferact.ai >
2026-04-21 10:49:32 +08:00
wang.yuqi and gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
4e8c3f1c19
[Frontend][last/5] Improve pooling entrypoints | clean up. ( #39675 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-04-16 07:53:23 -07:00
wang.yuqi
66c079ae83
[Frontend][4/n] Improve pooling entrypoints | pooling. ( #39153 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-04-09 10:09:45 +00:00
8477fe427d
[Tool] adjust_request to reasoning parser, and Gemma4 fixes ( #39027 )
...
Signed-off-by: Ben Browning <bbrownin@redhat.com >
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-04-08 19:04:04 +00:00
2e56975657
Generative Scoring ( #34539 )
...
Signed-off-by: Vedant Jhaveri <vjhaveri@linkedin.com >
Co-authored-by: Vedant Jhaveri <vjhaveri@linkedin.com >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-03-31 16:02:11 -07:00
wang.yuqi
ed359c497a
[Model] Deprecate the score task (this will not affect users). ( #37537 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-03-20 08:07:56 +00:00
Flora Feng
9040151fe1
[V0 Deprecation] Deprecate --disable-frontend-multiprocessing ( #37612 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-03-20 11:31:43 +08:00
Sage
00f8e0d211
[Frontend] Delegate tokenization serving preprocessing to OpenAIServingRender ( #37266 )
...
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com >
2026-03-17 11:22:54 +00:00
Chauncey
6682c231fa
[Bugfix] Add error handling for FINISHED_ERROR in OpenAIServing ( #37148 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
2026-03-16 16:27:47 +00:00
Sergey Zinchenko
4a718e770d
[Bug] Fix Failure in /v1/chat/completions/render for Multimodal Requests ( https://github.com/vllm-project/vllm/issues/35665 ) ( #35684 )
2026-03-14 14:10:11 +00:00
Sage
06e0bc21d2
[Frontend] Split OpenAIServingModels into OpenAIModelRegistry + OpenAIServingModels ( #36536 )
...
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com >
2026-03-12 03:29:37 -07:00
Sage
4497431df6
[Frontend] Add GPU-less render serving path (vllm launch render) ( #36166 )
2026-03-08 16:35:09 +01:00
Ning Xie
176c799f4c
[openai api] log exception in exception handler (1/N) ( #31164 )
...
Signed-off-by: Andy Xie <andy.xning@gmail.com >
2026-03-05 16:00:12 +00:00
Hyunkyun Moon
bc6be89d16
[Frontend] Add vllm launch command for GPU-less preprocessing serving ( #34551 )
...
Signed-off-by: HyunKyun Moon <mhg5303@gmail.com >
2026-03-04 18:41:52 +00:00
1659b2e058
[Feature] Add basic metrics for /realtime endpoint ( #35500 )
...
Signed-off-by: Thomas Pouget-Abadie <thomaspou@microsoft.com >
Signed-off-by: pougetat <thomas.pougetabadie@gmail.com >
Co-authored-by: Thomas Pouget-Abadie <thomaspou@microsoft.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-03-04 19:56:32 +08:00
wang.yuqi
dab1de9f38
[Frontend][CI] Consolidate instrumentator entrypoints ( #34123 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-02-10 07:30:19 +00:00
kourosh hakhamaneshi and Cyrus Leung
a75a5b54c7
[bug-fix] supported_tasks is breaking backward compatibility at init_app_state ( #34027 )
...
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com >
Signed-off-by: kourosh hakhamaneshi <31483498+kouroshHakha@users.noreply.github.com >
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com >
2026-02-09 09:46:46 +08:00
emricksini-h
325ab6b0a8
[Feature] OTEL tracing during loading ( #31162 )
2026-02-05 16:59:28 -08:00
Nicolò Lucchesi
20f5d185a6
[Misc] Rename translations to speech_to_text for OAI serving component ( #33904 )
...
Signed-off-by: NickLucche <nlucches@redhat.com >
2026-02-05 19:16:52 +00:00
Patrick von Platen and Nick Hill
10152d2194
[Realtime API] Adds minimal realtime API based on websockets ( #33187 )
...
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-01-30 18:41:29 +08:00
wang.yuqi
7cbbca9aaa
[Frontend] Cleanup api server ( #33158 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
2026-01-27 15:18:10 +00:00
wang.yuqi and Cyrus Leung
76139d0801
[Frontend] Frontend will only attach supported tasks corresponding entrypoints. ( #33139 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com >
2026-01-27 12:15:43 +00:00
Jared Wen
6ee7f18f33
[Logging] add --disable-access-log-for-endpoints CLI option ( #30011 )
...
Add a new CLI option --disable-access-log-for-endpoints to suppress
uvicorn access logs for specified endpoints (e.g., /health, /metrics, /ping).
This addresses the need to reduce log noise in production environments
where health check endpoints are frequently polled by load balancers or
monitoring systems, generating excessive log entries that obscure
meaningful request logs.
Fixes #29982
Signed-off-by: JaredforReal <w13431838023@gmail.com >
2026-01-26 21:49:03 +00:00
7. Sun
0f19427db5
[Perf] Cache exc.errors() result in validation exception handler ( #32984 )
...
Signed-off-by: 7. Sun <jhao.sun@gmail.com >
2026-01-24 02:01:35 -08:00
wang.yuqi