iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
16,919 Commits 3 Branches 0 Tags
1806d1adfc9b598bc6eb94de38a330aaad04c291
Commit Graph
1563 Commits
Author SHA1 Message Date
Andreas Karatzas 5940590855 [ROCm][CI] Stabilize 400 error return code for invalid schema inputs (#43016) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-05-24 10:06:49 +00:00
ffggsandCyrus Leung d8b385b7ea [Bugfix][Frontend] Fix input_audio parsing when uuid is present (#43414) 
Signed-off-by: ffggs <314137448@qq.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-05-23 09:03:19 -07:00
Russell BryantandClaude Opus 4.6 2b94d1c0ca [Frontend] Simplify AuthenticationMiddleware path extraction (#43426) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-22 11:59:14 -07:00
Wentao Yerobertgshaw2-redhatRobert ShawNick Hill
 		0ddd7dd656 [Frontend] DP Supervisor (#40841) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com>
Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: robertgshaw2-redhat <robertgshaw2@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-05-21 20:33:16 -07:00
ruizhangandRui Zhang 60af5c16ee [Frontend] Add truncation side to OpenAI endpoints (#43260) 
Signed-off-by: Rui Zhang <rza21.bc@gmail.com>
Signed-off-by: Rui Zhang <rui.zhang@globalrelay.net>
Co-authored-by: Rui Zhang <rui.zhang@globalrelay.net>
 2026-05-21 20:32:31 -07:00
Daoyuan Li a6682d1d25 [Bugfix] Warn when renderer_num_workers has no effect on offline LLM (#42905) 
Signed-off-by: Daoyuan Li <94409450+DaoyuanLi2816@users.noreply.github.com>
 2026-05-20 21:35:08 -07:00
Nick HillandBugen Zhao f2ace1d57d [Frontend][RFC] Rust front-end integration (#40848) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
 2026-05-21 12:24:48 +08:00
Ben Browningandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 346cf163a1 [Frontend] Normalize reasoning_content to reasoning for client compatibility (#42664) 
Signed-off-by: Ben Browning <bbrownin@redhat.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-20 21:23:47 -07:00
Aaron HaoandIsotr0py 2d6b3489b9 [R3] Add routed experts to openai entrypoint (#38939) 
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-05-20 09:07:59 -07:00
halleriteandClaude Opus 4.7 cb600d1cdb [Frontend] Forward X-data-parallel-rank header on /inference/v1/generate (#42330) 
Signed-off-by: hallerite <git@hallerite.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 08:58:46 +00:00
pmaybankgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		fadf5d332c add enqueue all option to throughput benchmark (#42975) 
Signed-off-by: Philip Maybank <pmaybank@amd.com>
Signed-off-by: pmaybank <113125070+pmaybank@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-19 20:16:02 -07:00
Sage a78b842d0e [Bugfix] Fix top logprobs token placeholders in /inference/v1/generate (#42887) 
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
 2026-05-19 10:21:49 +00:00
wang.yuqi 301d986473 [Frontend] Consolidate beam search by BeamSearchMixin. (#42946) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-05-19 07:37:40 +00:00
Michael GoinandClaude 239b5ff30c [Frontend] Add --spec-method/--spec-model/--spec-tokens CLI aliases (#42476) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-05-18 17:22:27 -07:00
Jee Jee Ligemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Roger Wang
 		7d5b033782 [LoRA] Support 2D and 3D MoE LoRA adapter at the same time (#42242) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-05-18 15:22:26 +08:00
DustHunterQwen-Coderwang.yuqi
 		39c67d714e fix: add API key authorization to /v2 endpoints (#42594) 
Signed-off-by: DustHunter <dusthunter@126.com>
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-05-16 01:29:27 +00:00
wang.yuqi 75fd68c7a5 [Entrypoints] Split the pooling offline API into PoolingOfflineMixin. (#42267) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-05-15 08:05:57 +00:00
Cyrus Leung 2676ab1e0b [Deprecation] Remove old locations of get_tokenizer and resolve_hf_chat_template (#35024) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-05-15 00:13:32 -07:00
Hongxin XuClaude Opus 4.6arlenxuJunjie Zhang
 		c7560af424 [RFC] Replace shared-memory routed experts with ModelRunnerOutput transfer and HTTP support (#39568) 
Signed-off-by: xhx1022 <1737006628@qq.com>
Signed-off-by: arlenxu <arlenxu@tencent.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: arlenxu <arlenxu@tencent.com>
Co-authored-by: Junjie Zhang <junj.jay.zhang@gmail.com>
 2026-05-14 14:12:30 +00:00
aoshen02 8c79ad6580 Revert "[Core] Replace routing replay with device cache and async D2H pipeline" (#39917) (#42434) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
 2026-05-13 23:49:01 -07:00
Krish Gupta 70c00163ff [Feature] Add instruction support for score/rerank chat templates (#42412) 
Signed-off-by: KrxGu <krishom70@gmail.com>
 2026-05-14 09:41:22 +08:00
Michael Goin 8efd508204 [Quantization] Rework quantization_config to use QuantKey and allow for activation override (#41566) 2026-05-13 16:58:32 -04:00
Walter Beller-Morales 873910d608 [Frontend] add support for thinking_token_budget in completions (#42116) 2026-05-13 16:01:52 -04:00
Ben Browning 0f69128a37 [Bugfix] Handle real-world gpt-oss tool call output in Harmony parsing (#42454) 
Signed-off-by: Ben Browning <bbrownin@redhat.com>
 2026-05-13 17:54:46 +00:00
halleriteandClaude Opus 4.7 97c4317bf5 [Bugfix][Frontend] Default max_tokens server-side on /inference/v1/generate (#42329) 
Signed-off-by: hallerite <git@hallerite.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 11:16:46 +02:00
Taneem Ibrahimgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Wentao Ye
 		503697c9ce [chore] Refactor pooling metadata token ID accessors (#42368) 
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-05-13 06:08:01 +00:00
yzong-rh 6ff7405b81 [Bugfix] [Frontend] Responses API, fix merging of messages (#42189) 
Signed-off-by: Yifan Zong <yzong@redhat.com>
Signed-off-by: Yifan <yzong@redhat.com>
 2026-05-12 16:09:59 +00:00
Wentao Ye a1b2d87498 [Refactor] Clean up pooling models build_tok_params logic (#42341) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-12 15:05:05 +00:00
chazenandChauncey 7f65f84428 [Bugfix] Fix empty channel/recipient in harmony for /v1/responses (#35540) 
Signed-off-by: kg6-sleipnir <christopherhazen42@gmail.com>
Signed-off-by: chazen <45186108+kg6-sleipnir@users.noreply.github.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
 2026-05-12 08:45:51 +00:00
wang.yuqiandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> d37e25ffbe [Frontend] Consolidate Speech to Text entrypoints. (#42370) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-12 07:06:57 +00:00
Chauncey 617239b70c [Frontend]Responses API supports chat_template_kwargs (#42272) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-05-11 11:59:39 +00:00
汪志鹏Wang, Zhipeng | RASIA <zhipeng.wang@rakuten.com>CursorChauncey
 		581b5e9afc [Frontend] Return rendered prompt text in chat completion response (#42052) 
Signed-off-by: Wang, Zhipeng | RASIA <zhipeng.wang@rakuten.com>
Co-authored-by: Wang, Zhipeng | RASIA <zhipeng.wang@rakuten.com>
Co-authored-by: Cursor <cursor@cursor.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
 2026-05-11 13:53:39 +08:00
abdulrahman-cohereCursor AgentNicolò Lucchesi
 		a5d0a5afba [Frontend][Bugfix] Abort ASR engine requests on cancellation (#41266) 
Signed-off-by: abdulrahman-cohere <abdulrahman.abdulrazzag@cohere.com>
Signed-off-by: <>
Co-authored-by: Cursor Agent <cursor-agent@cursor.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-05-09 23:51:11 -07:00
Dao007forever 3f5bd482f5 [Bugfix][KV Transfer][NIXL] Notify P node on pre-admission rejection to free stranded KV blocks (#41269) 2026-05-09 22:52:09 -07:00
Sumanth R Hegde e3b65a5ba0 [feat] Add explicit /start_weight_update and /finish_weight_update APIs for weight transfer (#39212) 2026-05-08 18:03:33 -07:00
Shrinav LokaClaude Opus 4.6wang.yuqimergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		cd58e30872 [Perf] Use numpy zero-copy path for embedding float response serialization (#41681) 
Signed-off-by: Shrinav Loka <lokashrinav@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-07 19:42:21 -07:00
TomerBN-Nvidia 8189a15914 [Core] Replace routing replay with device cache and async D2H pipeline (#39917) 
Signed-off-by: Tomer Barnatan <tbarnatan@nvidia.com>
 2026-05-07 11:24:56 -07:00
Flora Feng 8eb401134e [Refactor] Consolidate required/named tool_choice streaming into DelegatingParser (#41876) 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
 2026-05-07 09:50:59 -07:00
Viktor Pus d5b31c954d [Bugfix] Account for truncate_prompt_tokens when computing max_tokens (#41800) 
Signed-off-by: Viktor Pus <viktorpus@tenstorrent.com>
 2026-05-06 16:10:17 +00:00
jackQwertyJackchaunceyjiang
 		6467213a9f fix(openai): tolerate empty content in forced tool choice (#40148) 
Signed-off-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com>
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com>
Co-authored-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-05-06 07:16:03 -07:00
Chauncey c7aa186d67 [Frontend] Supports resubmitting output items with missing fields in Responses API (#41355) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-05-05 22:21:33 -04:00
Linzhang LiMichael GoinUbospicasfeng33
 		844df54269 feat: update xgrammar==0.2.0 to use structural tags for strict tool calling + reasoning for more models (#40894) 
Signed-off-by: Yuchuan <yuchuan.7streams@gmail.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Ubospica <ubospica@gmail.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Ubospica <ubospica@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
 2026-05-04 12:45:24 -07:00
Bugen Zhaoandwang.yuqi a07642667d [Bugfix] Pass reasoning parser kwargs to structured output (#41199) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-04-30 23:38:02 -07:00
Luis 🚀andAndrew Sansom 14043dfecd feat: Enable prompt_embeds Content Part Support in vLLM Chat Completions API (#40720) 
Signed-off-by: Luis Robaina <luis@protopia.ai>
Signed-off-by: Luis Robaina 🚀 <luisfabian1545@gmail.com>
Signed-off-by: LuisRobaina <luis@protopia.ai>
Co-authored-by: Andrew Sansom <qthequartermasterman@gmail.com>
 2026-05-01 10:05:55 +08:00
Dhruv SingalClaude Opus 4.6Dhruv SingalYour NameOpenCodeSimon Mo
 		cb1b02d0e8 [Frontend] Add VLLM_SKIP_MODEL_NAME_VALIDATION environment variable (#34676) 
Signed-off-by: Dhruv Singal <dhruvsingalabc@gmail.com>
Signed-off-by: Dhruv Singal <dsingal@Dhruvs-MacBook-Pro.local>
Signed-off-by: Your Name <you@example.com>
Signed-off-by: vLLM Assistant <assistant@vllm.ai>
Signed-off-by: Simon Mo <simon.mo@hey.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Dhruv Singal <dsingal@Dhruvs-MacBook-Pro.local>
Co-authored-by: Your Name <you@example.com>
Co-authored-by: OpenCode <noreply@openai.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2026-04-29 23:19:09 -07:00
Jared Wensfeng33Chauncey
 		11b69129e2 [Frontend] Add defer_loading and tool_reference support for Anthropic and OpenAI APIs (#40190) 
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
 2026-04-29 04:35:50 -07:00
Bugen Zhao 33f36d4260 [DSV4] Support max reasoning effort (#40982) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-04-29 11:03:47 +00:00
Chauncey 3885d340a4 [Frontend]Responses API supports Tool/Function calling with streaming with named tool/function (#41110) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-04-29 09:11:27 +00:00
haosdent 4b95e9cec4 [CI] Return HTTP 400 for unsupported chat content part type (#41121) 
Signed-off-by: haosdent <haosdent@gmail.com>
 2026-04-29 10:23:26 +08:00
wang.yuqi a8208e6a81 [Examples] Resettle features examples. (#40995) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-04-28 00:33:41 -07:00