iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
16,247 Commits 3 Branches 0 Tags
2fa1f8ec00cf85b15422cb4c0e8eb3632ee13ea8
Commit Graph
595 Commits
Author SHA1 Message Date
Bugen Zhaoandwang.yuqi a07642667d [Bugfix] Pass reasoning parser kwargs to structured output (#41199) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-04-30 23:38:02 -07:00
Luis 🚀andAndrew Sansom 14043dfecd feat: Enable prompt_embeds Content Part Support in vLLM Chat Completions API (#40720) 
Signed-off-by: Luis Robaina <luis@protopia.ai>
Signed-off-by: Luis Robaina 🚀 <luisfabian1545@gmail.com>
Signed-off-by: LuisRobaina <luis@protopia.ai>
Co-authored-by: Andrew Sansom <qthequartermasterman@gmail.com>
 2026-05-01 10:05:55 +08:00
Aaron HaoJunjie ZhangNick Hill
 		0335316a9b [BUG] Two phase pause to prevent deadlock (#39366) 
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
Signed-off-by: Aaron Hao <ahao@anyscale.com>
Co-authored-by: Junjie Zhang <junj.jay.zhang@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-04-29 17:51:03 -04:00
wang.yuqi a8208e6a81 [Examples] Resettle features examples. (#40995) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-04-28 00:33:41 -07:00
+7
Yifan QiaoYongye ZhuYongye ZhuSimon MoBugen ZhaoGiancarlo DelfinJee Jee LiNick HillRoger WangRoy WangWoosuk KwonyoukaichaoZhewen LiZijing LiukhluuqizixiZhewen Li
 		4d51588e23 [Feat] DeepSeek V4 Rebased (#40860) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: qizixi <zixi@inferact.ai>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <yongye@inferact.ai>
Co-authored-by: Simon Mo <simon@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roy Wang <yasong.wang@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com>
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com>
Co-authored-by: khluu <khluu000@gmail.com>
Co-authored-by: qizixi <zixi@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
 2026-04-26 18:31:08 -07:00
Andreas Karatzas 95995bbef8 [ROCm][Engine] Fix GPU memory leaks in engine shutdown and test workaround for async KV prefix cache reset (#38503) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-04-25 05:25:20 +00:00
shaharmor98Robert Shawmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		e9ba519f45 [DP][Ray] Pin DP control bundle to same node as first GPU bundle (#39167) 
Signed-off-by: Shahar Mor <smor@nvidia.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-04-23 17:21:13 +00:00
Wentao Yeandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 8f87eb4622 [Refactor] Clean up log once scope="local" (#40540) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-04-22 16:42:43 -04:00
Wentao Ye 1842447c09 [Refactor] Remove unused param (#39750) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-04-21 14:59:20 -07:00
Kris HungandClaude Opus 4.6 766cb65d00 feat(multimodal): support externally processed mm_kwargs with cache injection (#39502) 
Signed-off-by: Krish Hung <krishung5@gmail.com>
Signed-off-by: krishung5 <krish@nvidia.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 11:31:09 +00:00
Csrayz 68be0f853e [Metrics] Add request_id to FinishedRequestStats to enable correlation between metrics and requests (#39710) 
Enables external `StatLogger` plugins to correlate per-request metrics
with request-level context. Also, this is a pre-requisite for Prometheus
exemplars in #30972.

Signed-off-by: Csrayz <33659823+Csrayz@users.noreply.github.com>
 2026-04-15 11:24:17 +00:00
Lucas Kabela c08f3b2a62 Measure encoder compile time seperate from llm backbone (#39240) 
Signed-off-by: Lucas Kabela <lucaskabela@meta.com>
 2026-04-14 10:52:49 -04:00
Mark McLoughlinandOr Ozeri d3af8c1831 [Core][Metrics][BugFix] Replace num_cached_tokens/num_external_computed_tokens with PrefillStats (#37460) 
Related to `Counters can only be incremented by non-negative amounts`
error with the `vllm:prompt_tokens_by_source_total` metric.

Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Or Ozeri <or@ozery.com>
 2026-04-14 09:00:45 +01:00
Nick Hill ef076c1b73 [Core] Change max_model_len in EngineCoreReadyResponse to be non-None (#39442) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-04-10 14:34:57 +08:00
Chauncey ecbfbb8d61 [Feature] Add auto-detection for reasoning_config when only reasoning_parser is set (#38214) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-04-10 01:36:26 +00:00
Cyrus Leung 54d244f28f [UX] Improve error message for MM input too long (#39409) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-04-09 13:20:19 +00:00
Nick Hill c8d98f81f6 [Core] Simplify API server handshake (#39364) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-04-09 18:56:15 +08:00
wang.yuqi 66c079ae83 [Frontend][4/n] Improve pooling entrypoints | pooling. (#39153) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-04-09 10:09:45 +00:00
triangleXIVandmgoin 7c94ae16c6 [BugFix] --max-model-len=-1 causes over-limit requests to hang and starve the entire service (#39102) 
Signed-off-by: triangle14 <y1019026570@gmail.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2026-04-08 14:03:17 -07:00
Shengqi ChenJason LiRoger Wang
 		75e01a39a1 [Feature] NUMA binding support for GPU workers (#38635) 
Signed-off-by: Shengqi Chen <harry-chen@outlook.com>
Co-authored-by: Jason Li <jasonlizhengjian@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-04-08 09:55:24 -07:00
haosdent 8904fc4d19 [Bugfix] Fix V1 logprobs empty strings for multi-byte UTF-8 tokens when logprobs > 0 (#34875) 
Signed-off-by: haosdent <haosdent@gmail.com>
 2026-04-08 15:30:00 +00:00
fangyuchuandNick Hill b5e608258e [Refactor] Unify engine process monitoring in engine manager and add Ray backend support (#35862) 
Signed-off-by: fangyuchu <fangyuchu@qq.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-03-30 10:16:09 -07:00
BviciiandCyrus Leung 999dfc1622 [Bugfix] Offload blocking tokenizer ops to shared thread pool to unblock event loop (#34789) 
Signed-off-by: Bvicii <yizhanhuang2002@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-03-26 22:17:00 -07:00
Cyrus Leung ba2f0acc2d [Misc] Reorganize inputs (#35182) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-03-25 10:22:54 -07:00
Sungjae LeeChaunceyHarry Mellor
 		4731884796 [Feature] limit thinking tokens (hard limit) (#20859) 
Signed-off-by: Sungjae Lee <33976427+llsj14@users.noreply.github.com>
Signed-off-by: Sungjae Lee <sung-jae.lee@navercorp.com>
Signed-off-by: Chauncey <chaunceyjiang@gmail.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-24 09:53:07 -07:00
Brandon PelfreyandNick Hill 80b70884eb Add tensor IPC transfer mechanism for multimodal data (#32104) 
Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com>
Signed-off-by: Brandon Pelfrey <brandonpelfrey@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-03-21 20:10:20 +00:00
Itay AlroyandRon Tourgeman c57d38d603 elastic_ep: Fix issues with repeated scale up/down cycles (#37131) 
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
Co-authored-by: Ron Tourgeman <rtourgeman@nvidia.com>
 2026-03-20 23:13:02 +00:00
Itay Alroy ca1ac1a4b4 Fix DP coordinator ZMQ port TOCTOU (#37452) 
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
 2026-03-20 00:58:31 +00:00
Aaron Hao 6accb21f2a [bug] Fix deadlock with pause resume and collective_rpc (#37024) 
Signed-off-by: hao-aaron <ahao@anyscale.com>
 2026-03-19 01:49:02 +00:00
Itay Alroy de1a86b7de elastic_ep: Fix stateless group port races (#36330) 
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
 2026-03-18 14:36:18 +00:00
Nick Hill 54a6db827f [BugFix] Fix "DP Coordinator receives unexpected..." messages (#37008) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-13 23:18:05 +00:00
Mark McLoughlinClaude Sonnet 4.5Nick Hill
 		7afe0faab1 [Frontend][Core] Re-add shutdown timeout - allowing in-flight requests to finish (#36666) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-03-13 12:10:06 -07:00
Itay AlroyYongji WuRon Tourgeman
 		d5af196c18 [2/N] Elastic EP Milestone 2: Integrating NIXL-EP (#35627) 
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
Co-authored-by: Yongji Wu <wuyongji317@gmail.com>
Co-authored-by: Ron Tourgeman <rtourgeman@nvidia.com>
 2026-03-13 09:25:33 -04:00
Kunshang Ji 53ec16a705 [Hardware] Replace torch.cuda.device_count/current_device/set_device API (#36145) 
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-03-12 07:57:47 -07:00
Nick Hill 36735fd772 [BugFix] Fix multiple/duplicate stdout prefixes (#36822) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-12 12:23:21 +08:00
Wentao Ye 35bdca5431 [Refactor] Remove dead code in KV connector (#36424) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-03-11 19:40:17 +00:00
fangyuchu fa0d353acf [Bugfix] Surface exceptions from non-blocking execute_model in UniProcExecutor to avoid DP deadlocks (#35194) 
Signed-off-by: fangyuchu <fangyuchu@qq.com>
 2026-03-11 03:22:21 +00:00
Nick Hill 65b2f405dc [Core] Simplify core kv-cache blocks initialization logic (#36521) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-10 20:20:02 +00:00
Harry Mellor f83b933b84 [CI] Bump mypy version to 1.19.1 (#36104) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-10 09:18:28 -07:00
SoluMilken 409c4e632d [Misc] fix typo: homogenous-> homogeneous (2 lines change) (#36508) 
Signed-off-by: SoluMilken <ypiheyn.imm02g@g2.nctu.edu.tw>
 2026-03-10 06:25:37 -07:00
Mark McLoughlin 234860399b [Frontend][Core] Revert "Add shutdown timeout" (#34730 and #36270) (#36628) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2026-03-10 06:20:41 -07:00
Wentao Ye 7279374f91 [Perf] Compute maxsim in worker side, reducing redundant copies, 2.7% E2E throughput improvement (#36159) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-03-09 20:55:58 -07:00
Matthew Bonanni 77a73458e3 Reapply [Attention] Refactor check_and_update_config (#35122) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-03-09 07:17:14 -07:00
Cyrus Leung f96c3ab08c [Deprecation][1/2] Remove items deprecated in v0.18 (#36470) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-03-09 03:43:23 -07:00
Sage 4497431df6 [Frontend] Add GPU-less render serving path (vllm launch render) (#36166) 2026-03-08 16:35:09 +01:00
Nick HillandMark McLoughlin 6a18d8789b [Core] Fix benign error log during normal shutdown (#36270) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
 2026-03-07 00:39:21 +00:00
Nick Hill 26bd43b52d Revert "[BugFix] Fix engine hanging after KV cache initialization fai… (#36262) 2026-03-06 08:28:09 -08:00
Mark McLoughlinandClaude Sonnet 4.5 27066d1b2b [Frontend][Core] Add shutdown timeout - allowing in-flight requests to finish (#34730) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-03-05 22:04:31 -08:00
Shiyan DengandLu Fang 0a208d1f54 [BugFix] Fix engine hanging after KV cache initialization failure (#35478) 
Signed-off-by: Shiyan Deng <dsy842974287@meta.com>
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
 2026-03-05 20:58:09 -08:00
Jiayi YanandHarry Mellor 6a895197fa [Bugfix][CI] fix typos (#34934) 
Signed-off-by: 1195343015 <1195343015@qq.com>
Signed-off-by: Jiayi Yan <66017932+1195343015@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-05 17:05:46 +00:00