Martin Hickey
0a31372e5f
[Frontend] Add detokenization streaming derender for disaggregated serving ( #47301 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-07-29 18:01:20 -07:00
Zach Zhu
58f9659397
[Frontend][Core] Standardize request error handling with VLLMError hierarchy ( #49665 )
...
Signed-off-by: Zach Zhu <zzqshu@126.com >
2026-07-29 04:20:40 +00:00
Itay Alroy
35efdf6b34
[Elastic EP] Async preparation ( #47288 )
...
Signed-off-by: Itay Alroy <ialroy@nvidia.com >
2026-07-28 05:19:13 -07:00
Shuolei Wang
9069a57139
[Core][Frontend] Add weight version tagging for RL rollouts ( #49040 )
...
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com >
Signed-off-by: Shuolei Wang <948904026@qq.com >
2026-07-28 14:35:29 +08:00
0b0bd2b5f6
[Feature] Add fault tolerance framework (simplified) for DP+EP external LB deployments ( #44428 )
...
Signed-off-by: fangyuchu <fangyuchu@qq.com >
Signed-off-by: a798347923 <2645302020@qq.com >
Signed-off-by: TianZhuo <2770730562@qq.com >
Signed-off-by: a798347923 <39047817+a798347923@users.noreply.github.com >
Signed-off-by: 205150940 <112750056+205150940@users.noreply.github.com >
Signed-off-by: w00689259 <wangzhuo66@huawei.com >
Signed-off-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com >
Signed-off-by: zWaNg3 <389750525@qq.com >
Signed-off-by: yzchang-plus <1078477584@qq.com >
Signed-off-by: Jade Zheng <zheng.shoujian@outlook.com >
Co-authored-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com >
Co-authored-by: a798347923 <2645302020@qq.com >
Co-authored-by: TianZhuo <2770730562@qq.com >
Co-authored-by: 205150940 <112750056+205150940@users.noreply.github.com >
Co-authored-by: a798347923 <39047817+a798347923@users.noreply.github.com >
Co-authored-by: w00689259 <wangzhuo66@huawei.com >
Co-authored-by: zWaNg3 <389750525@qq.com >
Co-authored-by: yzchang-plus <1078477584@qq.com >
Co-authored-by: Jade Zheng <zheng.shoujian@outlook.com >
2026-07-25 11:49:09 -04:00
Nick Hill
4809de7317
[Misc] Fix terminal output logo coloring ( #49344 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-07-22 00:21:29 +00:00
Ning Xie
b7c20d0cfa
[chore] adjust logo be more friendly to white background terminal ( #48938 )
...
Signed-off-by: Andy Xie <andy.xning@gmail.com >
2026-07-20 15:15:28 -07:00
83762b77b0
[Frontend] Add /abort_requests to the RLHF dev API router ( #47173 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-12 14:21:02 +08:00
fc1c548093
Runtime Draft Weight Update for Speculative Decoding ( #46725 )
...
Signed-off-by: vx120 <893600387@qq.com >
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: crp0128 <191679376@qq.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
2026-07-11 22:51:53 -07:00
Muhammad Fawaz and Mahad Durrani
e87521626f
Sanitize server file paths from validation error responses ( #46415 )
...
Signed-off-by: muhammadfawaz1 <135441198+muhammadfawaz1@users.noreply.github.com >
Co-authored-by: Mahad Durrani <114791389+mahadrehmann@users.noreply.github.com >
2026-07-09 17:46:29 +08:00
dcdd756d75
[CI] GSM8K eval integration test for KV offloading ( #46893 )
...
Signed-off-by: Tyler Michael Smith <tyler@tylermsmith.com >
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Signed-off-by: Itay Etelis <itay.etelis@ibm.com >
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Signed-off-by: Or Ozeri <oro@il.ibm.com >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Co-authored-by: Itay Etelis <itay.etelis@ibm.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-08 17:59:48 -04:00
Alexis K.
40cc2e8327
[Bugfix] Return HTTP 422 for unprocessable image URLs instead of 500 ( #47165 )
...
Signed-off-by: Alexis Kinsella <alexis.kinsella@gmail.com >
2026-07-06 11:56:23 +00:00
wang.yuqi
a14f57a3ac
[Frontend] Refine the entrypoint class's inheritance hierarchy. ( #47498 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-07-03 10:50:06 +00:00
Aaron Hao and Claude Sonnet 4.6
77a9c5ae28
Weight sync refactor + move sparse nccl engine ( #44353 )
...
Signed-off-by: hao-aaron <ahao@anyscale.com >
Signed-off-by: haoaaron <ahao@anyscale.com >
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-07-01 01:25:19 -07:00
wang.yuqi
3483240b7e
[Frontend] Consolidate scale out entrypoints ( #44512 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-29 03:18:53 -07:00
Ranran and Benjamin Chislett
09841ae705
[Render][Speculator] Add return_loss_mask to render endpoint for training data generation ( #46846 )
...
Signed-off-by: Ranran Haoran Zhang <ranzhang@redhat.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
2026-06-28 00:07:33 -07:00
Hyunkyun Moon
950ee4c2e4
[API] Add token offsets to render endpoints (/v1/.../render) ( #44226 )
...
Signed-off-by: HyunKyun Moon <mhg5303@gmail.com >
2026-06-26 05:02:52 -07:00
Muhammad Fawaz and Mahad Rehmann
1bf149f334
Filter Pydantic-internal markers from validation error param ( #46457 )
...
Signed-off-by: muhammadfawaz1 <135441198+muhammadfawaz1@users.noreply.github.com >
Co-authored-by: Mahad Rehmann <114791389+mahadrehmann@users.noreply.github.com >
2026-06-23 13:20:50 +00:00
wang.yuqi
31ca9504b1
[Frontend] Split ServingRender into renderer and entrypoint. ( #44285 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-23 11:19:09 +00:00
wang.yuqi
1eb2cc961e
[Frontend] Refactor ServingTokenization entrypoint. ( #46022 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-22 06:27:58 +00:00
3e6e33526d
[Disagg] return routed_experts on streaming generate responses ( #44638 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-06-21 07:37:10 -07:00
aoshen02 and Martin Hickey
f57ac274b2
[Render] Add reasoning/tool parsing to /derender + fix byte-fallback FFFD ( #45919 )
...
Signed-off-by: aoshen524 <aoshen524@gmail.com >
Co-authored-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-06-20 19:43:32 -04:00
Muhammad Fawaz and Mahad Durrani
69bdd34542
[Bugfix] Fall back to Pydantic loc for param in validation errors ( #46038 )
...
Signed-off-by: professorsab <135441198+professorsab@users.noreply.github.com >
Co-authored-by: Mahad Durrani <114791389+mahadrehmann@users.noreply.github.com >
2026-06-19 19:11:11 +08:00
Zang Peiyu
25ee659db0
Fix parallel_tool_calls: null treated as false instead of default true ( #44955 )
...
Signed-off-by: factnn <166481866+factnn@users.noreply.github.com >
2026-06-15 21:14:10 +00:00
yzong-rh and Ben Browning
51ec5cf08f
[Bugfix] Chat Completions Harmony Refactor Clean up ( #45464 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
Co-authored-by: Ben Browning <bbrownin@redhat.com >
2026-06-15 14:45:19 -04:00
Roger Wang and Claude Opus 4.8
e8d3e22c88
Fix included router missing path for FastAPI >=0.137 ( #45629 )
...
Signed-off-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
2026-06-15 04:28:52 +00:00
Martin Hickey and Isotr0py
43f0e024bc
[Render] Add /derender endpoints for disaggregated postprocessing ( #43606 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn >
2026-06-13 13:55:33 +08:00
2043258dec
[Frontend] Support strict mode for tool calling ( #45003 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-12 07:51:48 +00:00
Ekagra Ranjan and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
e0b9fb1290
[ASR] Optimize CPU preproc to get 2.5x RTFx via multi-threading ( #44612 )
...
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-11 21:05:11 -07:00
42ae5e7ac6
[Bugfix] Fix --enable-prompt-tokens-details omitting zero cached tokens ( #44383 )
...
Signed-off-by: Sasindharan Sankar <sasindharansankar@email.com >
Co-authored-by: Sasindharan Sankar <sasindharansankar@email.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-06-11 20:37:42 -07:00
yzong-rh
703fb17b13
[Bugfix] GPT-OSS instruction rendering ( #44330 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
2026-06-05 13:52:32 -04:00
wang.yuqi
d01d0b4646
[Frontend] Consolidate online serving utils. ( #44479 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-04 06:49:31 +00:00
wang.yuqi
b623f7ea95
[Frontend] Consolidate dev entrypoints. ( #44170 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-02 06:30:21 -07:00
Andreas Karatzas
5940590855
[ROCm][CI] Stabilize 400 error return code for invalid schema inputs ( #43016 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-05-24 10:06:49 +00:00
hallerite and Claude Opus 4.7
cb600d1cdb
[Frontend] Forward X-data-parallel-rank header on /inference/v1/generate ( #42330 )
...
Signed-off-by: hallerite <git@hallerite.com >
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-05-20 08:58:46 +00:00
Sage
a78b842d0e
[Bugfix] Fix top logprobs token placeholders in /inference/v1/generate ( #42887 )
...
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com >
2026-05-19 10:21:49 +00:00
7d5b033782
[LoRA] Support 2D and 3D MoE LoRA adapter at the same time ( #42242 )
...
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com >
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-05-18 15:22:26 +08:00
c7560af424
[RFC] Replace shared-memory routed experts with ModelRunnerOutput transfer and HTTP support ( #39568 )
...
Signed-off-by: xhx1022 <1737006628@qq.com >
Signed-off-by: arlenxu <arlenxu@tencent.com >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: arlenxu <arlenxu@tencent.com >
Co-authored-by: Junjie Zhang <junj.jay.zhang@gmail.com >
2026-05-14 14:12:30 +00:00
hallerite and Claude Opus 4.7
97c4317bf5
[Bugfix][Frontend] Default max_tokens server-side on /inference/v1/generate ( #42329 )
...
Signed-off-by: hallerite <git@hallerite.com >
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-05-13 11:16:46 +02:00
Sumanth R Hegde
e3b65a5ba0
[feat] Add explicit /start_weight_update and /finish_weight_update APIs for weight transfer ( #39212 )
2026-05-08 18:03:33 -07:00
Viktor Pus
d5b31c954d
[Bugfix] Account for truncate_prompt_tokens when computing max_tokens ( #41800 )
...
Signed-off-by: Viktor Pus <viktorpus@tenstorrent.com >
2026-05-06 16:10:17 +00:00
Luis 🚀 and Andrew Sansom
14043dfecd
feat: Enable prompt_embeds Content Part Support in vLLM Chat Completions API ( #40720 )
...
Signed-off-by: Luis Robaina <luis@protopia.ai >
Signed-off-by: Luis Robaina 🚀 <luisfabian1545@gmail.com >
Signed-off-by: LuisRobaina <luis@protopia.ai >
Co-authored-by: Andrew Sansom <qthequartermasterman@gmail.com >
2026-05-01 10:05:55 +08:00
Chenguang Zheng and OpenAI Codex
07351e0883
[Feature] Warm up readonly multimodal processor during renderer startup ( #40797 )
...
Signed-off-by: Chenguang ZHENG <645327136@qq.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-04-25 03:57:41 +00:00
Neil Schemenauer
56bdf85e10
[Feature] Avoid eager import of the "mistral_common" package. ( #40043 )
...
Signed-off-by: Neil Schemenauer <nas@arctrix.com >
2026-04-24 02:49:16 +00:00
Bugen Zhao
0210024ae7
[Bugfix] Pass effective chat template kwargs to reasoning parsers ( #40460 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-04-21 22:17:51 -07:00
Nithin Chalapathi and Cyrus Leung
80b18230e0
[Frontend] Add multimodal support to /inference/v1/generate endpoint ( #38405 )
...
Signed-off-by: Nithin Chalapathi <nithin.ch10@gmail.com >
Signed-off-by: Nithin Chalapathi <nithinc@berkeley.edu >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-04-17 20:31:56 -07:00
Julien Denize
c0722f22de
[Mistral Grammar] Fix tool and reasoning parsing ( #39217 )
...
Signed-off-by: juliendenize <julien.denize@mistral.ai >
2026-04-15 21:05:04 -07:00
70ed01550c
[Reasoning][Frontend] Add model config to adjust_request in reasoning parser ( #37848 )
...
Signed-off-by: rishitdholakia13 <rishit+github@cohere.com >
Signed-off-by: rishitdholakia13 <123388671+rishitdholakia13@users.noreply.github.com >
Signed-off-by: Aaron Pham <contact@aarnphm.xyz >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Aaron Pham <contact@aarnphm.xyz >
2026-04-14 20:29:06 +00:00
wang.yuqi
66c079ae83
[Frontend][4/n] Improve pooling entrypoints | pooling. ( #39153 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-04-09 10:09:45 +00:00
8477fe427d
[Tool] adjust_request to reasoning parser, and Gemma4 fixes ( #39027 )
...
Signed-off-by: Ben Browning <bbrownin@redhat.com >
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-04-08 19:04:04 +00:00
Martin Hickey