iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
22,991 Commits 3 Branches 0 Tags
c19c263ac8ea3fd6e1d2d689f0e4e725855d89cf
Commit Graph
81 Commits
Author SHA1 Message Date
Aaron Haoaoshen02Claude Opus 5.5
 		fa312090cb [Frontend] Accept each connection on the least loaded API worker (#60723) 
Signed-off-by: hao-aaron <aaronhao@inferact.ai>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 04:40:13 +00:00
Bugen ZhaoOpenAI CodexClaude Opus 5.5
 		f226af0e11 [Bugfix] Eliminate frontend ZMQ port TOCTOU with inherited listeners (#54113) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 06:25:27 +00:00
AlecBugen ZhaoClaude Opus 5.5
 		9d2a6f52b2 [Rust Frontend] Expose gRPC port in Python vllm serve (#59659) 
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Alec Flowers <aflowers@nvidia.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-10-02 08:14:37 +00:00
Louie Tsai 208bb38c80 [Bugfix][Rust Frontend] Fix startup with config-only model (#58643) 
Signed-off-by: louie-tsai <louie.tsai@intel.com>
 2026-09-29 10:11:53 +00:00
Mark McLoughlin a44d7b5151 [Core][Logging] Fix JSON logging process decoration (#57957) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2026-09-25 10:12:28 -06:00
Mark McLoughlinandAI Assistant 45d56c0baf [Core] Model console logging as CLI configuration (#57205) 
Add `--logging-config` CLI argument which can be supplied as
JSON or using dotted arguments. The `--log-level` argument
is provided for convenience, and `--log-config-file` is deprecated
in favor of `--logging-config.pylogging_config_file`.

Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: AI Assistant <noreply@openai.com>
 2026-09-25 11:36:59 +01:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 16:44:29 +00:00
Nick Hill f2aad6aa70 [MRV2] Buffer util simplifications (#56888) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-09-15 20:36:51 +00:00
Chauncey f0fbc98c48 [Frontend] Share max_num_queued_reqs across API server processes (#54746) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-09-15 14:50:26 +00:00
Moe Huzaifa 3e833933dc [Bugfix][Frontend] Keep credentials out of the Rust frontend launch log (#53738) 
Signed-off-by: mhuzaifa3 <mhuzaifa3@outlook.com>
 2026-08-25 16:32:12 +00:00
wang.yuqi eac636a7fa [Frontend] Move api_server.py out openai folder (#52131) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-08-19 09:43:46 +00:00
Connor Carpenterandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> fa722b9f01 [Rust Frontend][gRPC] Add explicit data-parallel rank routing (#51178) 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-08-10 15:05:24 -07:00
Ning Xie 9a7ae4b808 [chore] log process manager shutdown with more details (#49437) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2026-07-31 11:55:09 -07:00
Max HuRoger WangShang Wang
 		5a65ba5f17 [Refactor] Move iteration logging to the frontend (#46647) 
Signed-off-by: maxyanghu <hyoung2991@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Shang Wang <shangw@nvidia.com>
 2026-07-15 17:59:05 -07:00
Harry Mellor e24d1b24fe Fix Transformers modeling backend usage stats (#47472) 2026-07-02 12:51:23 -07:00
Tahsin TunanBugen Zhaomergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		b8cb75b149 [Rust Frontend] Add static HTTPS and mTLS support for HTTP and gRPC (#45890) 
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Tahsin Tunan <tahsintunan@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-06-30 01:45:59 +00:00
Kaiand图灵 9b215ae60b [Rust Frontend] Forward VLLM_ENGINE_READY_TIMEOUT_S via --args-json (#44610) 
Signed-off-by: kai <kai@example.com>
Co-authored-by: 图灵 <tuling.wk@alibaba-inc.com>
 2026-06-25 07:25:08 +00:00
Nick Hill 7df3d7dada [Core] Ensure memory is pinned prior to async h2d copy (#45424) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-20 20:02:24 -07:00
Bugen Zhaoandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> e2c58570ea [Rust Frontend] Support hybrid/external DP LB in Python supervised bootstrap (#45805) 
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-06-17 07:32:40 +00:00
Zachary XiandCursor 3f627ebef7 [Misc] usage_stats: report more engine, spec-decode, and EP config (#44595) 
Signed-off-by: Zach Xi <zachary.xi@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-06-08 15:20:00 -07:00
Wentao Ye 41a4829f22 [Logs Refactor] Optimize shutdown logs, easier to follow and consistent (#43707) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-06-04 14:36:32 -04:00
wang.yuqi d01d0b4646 [Frontend] Consolidate online serving utils. (#44479) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-06-04 06:49:31 +00:00
Wentao Ye be4062fd6c [Bug] Fix tests/distributed/test_elastic_ep.py - assert False (#43813) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-28 11:00:56 -04:00
Vadim GimpelsonandNick Hill d6b48f928f [BugFix] Fix hard-coded timeout for multi-API-server startup (#43768) 
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-05-28 00:09:13 -07:00
Vadim Gimpelsonandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 812e7e7364 [Bugfix][V1] Fix TOCTOU race causing intermittent EADDRINUSE on multi-API-server DP startup (#42585) 
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Signed-off-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-26 14:06:00 -07:00
Andreas Karatzas 5940590855 [ROCm][CI] Stabilize 400 error return code for invalid schema inputs (#43016) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-05-24 10:06:49 +00:00
Nick HillandBugen Zhao f2ace1d57d [Frontend][RFC] Rust front-end integration (#40848) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
 2026-05-21 12:24:48 +08:00
fangyuchuandNick Hill b5e608258e [Refactor] Unify engine process monitoring in engine manager and add Ray backend support (#35862) 
Signed-off-by: fangyuchu <fangyuchu@qq.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-03-30 10:16:09 -07:00
Nick Hill 72cad44d3c [Frontend] Move APIServerProcessManager target server fn (#38115) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-25 18:14:41 +00:00
Brandon PelfreyandNick Hill 80b70884eb Add tensor IPC transfer mechanism for multimodal data (#32104) 
Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com>
Signed-off-by: Brandon Pelfrey <brandonpelfrey@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-03-21 20:10:20 +00:00
Mark McLoughlinClaude Sonnet 4.5Nick Hill
 		7afe0faab1 [Frontend][Core] Re-add shutdown timeout - allowing in-flight requests to finish (#36666) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-03-13 12:10:06 -07:00
Mark McLoughlin 234860399b [Frontend][Core] Revert "Add shutdown timeout" (#34730 and #36270) (#36628) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2026-03-10 06:20:41 -07:00
Mark McLoughlinandClaude Sonnet 4.5 27066d1b2b [Frontend][Core] Add shutdown timeout - allowing in-flight requests to finish (#34730) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
 2026-03-05 22:04:31 -08:00
杨朱 · KikiandClaude Opus 4.5 1a7894dbdf [Misc] Replace Optional[X] with X | None syntax (#33332) 
Signed-off-by: carlory <baofa.fan@daocloud.io>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-30 01:56:59 -08:00
Max HuandMax Hu 6ebe34d6fa [Feature] Add iteration level logging and enhance nvtx marker (#31193) 
Signed-off-by: Max Hu <maxhu@nvidia.com>
Signed-off-by: Max Hu <hyoung2991@gmail.com>
Co-authored-by: Max Hu <maxhu@nvidia.com>
 2026-01-09 00:13:39 +00:00
Nick Hill 80b6080ddc [BugFix] Fix async scheduling + chunked prefill + preemption (#28787) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-17 06:46:46 +08:00
Cyrus Leung 6ebffafbb6 [Misc] Clean up more utils (#27567) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-27 15:30:38 +00:00
Andrew Sansom ff93cc8c84 [CORE] Support Prefix Caching with Prompt Embeds (#27219) 
Signed-off-by: Andrew Sansom <andrew@protopia.ai>
 2025-10-22 22:18:07 -07:00
iAmir97andiAmir97 7a6c8c3fa1 [Chore] Separate out vllm.utils.network_utils (#27164) 
Signed-off-by: iAmir97 <Amir.balwel@embeddedllm.com>
Co-authored-by: iAmir97 <Amir.balwel@embeddedllm.com>
 2025-10-19 03:06:32 -07:00
Tyler Michael Smith 579d2e5458 [WideEP][P/D] Add usage stats for DP+EP and KV Connector (#26836) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2025-10-14 23:51:54 +00:00
Harry Mellor 8fcaaf6a16 Update Optional[x] -> x | None and Union[x, y] to x | y (#26633) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-12 09:51:31 -07:00
Harry Mellor 6c04638214 Fix per file ruff ignores related to line length (#26262) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-06 05:12:40 +00:00
Harry Mellor d6953beb91 Convert formatting to use ruff instead of yapf + isort (#26247) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-05 07:06:22 -07:00
Naman Lalit 4e33a7ea85 [Bugfix] Optimize CpuGpuBuffer initialization (#25447) 
Signed-off-by: Naman Lalit <nl2688@nyu.edu>
 2025-09-27 08:07:36 +08:00
Corey Lowman c85d75cf08 Add VLLM_NVTX_SCOPES_FOR_PROFILING=1 to enable nvtx.annotate scopes (#25501) 
Signed-off-by: Corey Lowman <clowman1993@gmail.com>
 2025-09-23 22:50:09 +00:00
Boyuan FengandHarry Mellor 94e6b2d55f Allow users to specify kv cache memory size (#21489) 
Signed-off-by: Boyuan Feng <boyuan@meta.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-09-11 13:41:07 +00:00
Bangsheng TangandBangsheng Tang 848562bd49 break execute_model in gpu_model_runner into sub-functions for custom scopes (#24265) 
Co-authored-by: Bangsheng Tang <bangsheng@meta.com>
 2025-09-06 14:02:47 -07:00
Andrew Sansom 305a1cc0d2 refactor: Turn GPUModelRunner.inputs_embeds to a CpuGpuBuffer (#24345) 
Signed-off-by: Andrew Sansom <andrew@protopia.ai>
 2025-09-05 23:01:23 -07:00
Woosuk Kwon 04ff1e43fb [Misc] Move CpuGpuBuffer to vllm/v1/utils.py (#23728) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-08-27 03:25:00 -07:00