iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
22,991 Commits 3 Branches 0 Tags
c19c263ac8ea3fd6e1d2d689f0e4e725855d89cf
Commit Graph
183 Commits
Author SHA1 Message Date
Bolin SunClaudeNick Hill
 		04b02fc8a5 [Core] shm tensor arena for efficient cpu->gpu worker broadcast (#51207) 
Signed-off-by: Bolin Sun <bolins@nvidia.com>
Signed-off-by: <bolins@nvidia.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-10-09 16:13:24 -07:00
Nicolò Lucchesi 8733a21d55 [Core] Bound execute_dummy_batch RPC waits by the execute-model timeout (#60398) 
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
 2026-10-08 01:22:04 +00:00
czhu-cohereandaoshen02 6fddc9b21e [Profiler] Add per-session profiling controls to Python and Rust frontends (#57875) 
Signed-off-by: czhu <conway.zhu@cohere.com>
Signed-off-by: Conway Zhu <conway.zhu@cohere.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
 2026-10-06 05:21:19 +00:00
Chandra MouryaandVS Chandra Mourya 8a2364605c [Core] Bound draft-token RPC waits by the execute-model timeout (#58779) 
Signed-off-by: VS Chandra Mourya <219748331+vschandramourya@users.noreply.github.com>
Co-authored-by: VS Chandra Mourya <219748331+vschandramourya@users.noreply.github.com>
 2026-09-26 00:40:22 +00:00
Mark McLoughlinandAI Assistant 45d56c0baf [Core] Model console logging as CLI configuration (#57205) 
Add `--logging-config` CLI argument which can be supplied as
JSON or using dotted arguments. The `--log-level` argument
is provided for convenience, and `--log-config-file` is deprecated
in favor of `--logging-config.pylogging_config_file`.

Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: AI Assistant <noreply@openai.com>
 2026-09-25 11:36:59 +01:00
Hongxin Xuaoshen02OpenAI CodexRoger WangNick HillCodex
 		0b7f11a1ee [AuxOutput] Add block-keyed storage for routed-expert outputs (#45635) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-21 10:25:05 +00:00
Dakai AnAlanFokCoaoshen02OpenAI Codex
 		092bdd6d57 [Frontend][Core] Add release_kv_cache_memory() API (#44890) 
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: AlanFokCo <alanfok2868@gmail.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com>
Co-authored-by: AlanFokCo <alanfok2868@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-09-18 07:12:12 +08:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 16:44:29 +00:00
Wentao Ye 6b15bea080 [Refactor] Remove utils dead code (#53941) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-09-08 11:22:09 -04:00
yzong-rh 680e2177e4 [CI][Ray] Fix flaky multi-node assignment test after placement-group teardown (#53621) 2026-08-29 23:48:22 +00:00
Ace EldeibandCodex 9662ab0835 [Bugfix] Set breakable graph env before Ray actor import (#53293) 
Signed-off-by: Ace Eldeib <aeldeib@coreweave.com>
Co-authored-by: Codex <codex@openai.com>
 2026-08-28 23:11:50 +00:00
Charlie Fu fd98d32f73 [ROCm][CI] Fix test_ray_v2_executor (#54249) 
Signed-off-by: charlifu <charlifu@amd.com>
 2026-08-28 15:11:42 -07:00
aoshen02aoshen02ClaudeOpenAI CodexAndreas Karatzas
 		67e86d1e6f [BugFix] Bind RayExecutorV2 TCPStore before publishing its port (#50969) 
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-08-28 12:27:22 -07:00
Jeffrey Wang 79378fe6e8 [Bugfix] Avoid TCPStore port collision for co-located non-DP Ray engines (#53666) 
Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
 2026-08-27 19:58:14 +00:00
石皮幼鸟andOpenAI Codex 4ca856b0b5 [Bugfix] Release worker RPC payload before next dequeue (#51979) 
Signed-off-by: 石皮幼鸟 <2960474346@qq.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-24 07:27:36 -07:00
Aaron Hao 6cd9713463 [RL] P2P RDT weight sync (#43375) 2026-08-23 09:42:22 -07:00
Lucas Wilkinson 8bdc70ec7b [6/N][KV-Cache Layout Refactor] Standardize KV cache layout (#51718) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-08-21 17:47:49 -07:00
crZhao 3fc2893909 [Bugfix] Account for local DP workers in startup thread allocation (#52385) 
Signed-off-by: real-cpu <zhaochenrui757@gmail.com>
 2026-08-17 12:10:34 -07:00
omerpaz95yewentao256Wentao Ye
 		fa9d67f782 [EC Connector] Added Build Connector Worker Meta for EC Connector (#49585) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Signed-off-by: omerpaz95 <73347585+omerpaz95@users.noreply.github.com>
Co-authored-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-08-15 15:40:38 -07:00
Connor CarpenterandNick Hill f473870ecf [Rust Frontend][gRPC] Add RL lifecycle control (#51316) 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-08-14 11:32:39 -07:00
vllmellmandtjtanaa 436be94e13 [ROCm][Bugfix] Use TCP store when AITER custom all-reduce is enabled (#51635) 
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
 2026-08-10 11:07:14 +00:00
aoshen02Parsa Mazaheriaoshen02Claude
 		11ba93f364 [BugFix] Use file:// rendezvous for single-node executors to eliminate startup port races (#50999) 
Signed-off-by: Parsa Mazaheri <p_mazaheri@apple.com>
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Parsa Mazaheri <p_mazaheri@apple.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-08-09 22:00:04 -04:00
Nick HillandClaude 7635a9002b [Core] Explicitly manage torch CPU threads in workers (#49919) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-08-04 10:06:38 -07:00
aoshen02andNick Hill 42ab184ea7 [MRV2] Enable routed-experts capture (#50721) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-08-03 21:00:13 +00:00
bugkeepandClaude 48aa8d8d75 [Bugfix] Prevent stale multiproc RPC deadlines from becoming unbounded waits (#41357) 
Signed-off-by: bugkeep <1921817430@qq.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-07-29 16:41:34 -07:00
Itay Alroy 35efdf6b34 [Elastic EP] Async preparation (#47288) 
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
 2026-07-28 05:19:13 -07:00
rasmithandAndreas Karatzas 60d443f738 [CI/Build][The Rock][BugFix] Use fork method in test_multiproc_executor_multi_node for py 3.14 compat and fix test_multiproc_executor_shutdown_cleanup (#48655) 
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-07-21 15:40:29 -05:00
Nick CaoandClaude 920469974a [UX] Log worker exit code when process dies unexpectedly (#38641) 
Signed-off-by: Nick Cao <ncao@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-07-07 12:36:29 +00:00
fangyuchuTyler Michael Smithmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		245888ff77 [Feature] Detect all2all peer fault with fault tolerance backend and prevent corrupted output (#43637) 
Signed-off-by: fangyuchu <fangyuchu@qq.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-30 09:00:25 -07:00
Seiji Eicherandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> ded6676458 [Bugfix] Seed RayExecutorV2 TCPStore port by DP rank to avoid collisions (#45960) 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-30 07:37:34 -07:00
Tyler Michael SmithClaudeCodexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>kourosh hakhamaneshi
 		ebfbcfe46a Stop setting CUDA_VISIBLE_DEVICES internally in vLLM, add device_ids arg (#45026) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: kourosh hakhamaneshi <kouroshHakha@users.noreply.github.com>
 2026-06-20 13:38:10 -07:00
Nick Hill cf027b86af [Core] Simplify MRV2 async output handling (#45442) 2026-06-13 18:15:36 -07:00
Ryan Rockandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> aab639c705 [Core][AMD] Propagate shutdown timeout to MultiprocExecutor (#43154) 
Signed-off-by: Ryan Rock <ryan.rock@amd.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-06-12 15:13:31 -05:00
qizixi 96229fa99e [KVConnector][1/N] PP-aware handshake aggregation and intermediate-PP output plumbing (#43720) 
Signed-off-by: zixi-qi <zixi@inferact.ai>
 2026-06-04 22:04:19 -07:00
Wentao Ye 41a4829f22 [Logs Refactor] Optimize shutdown logs, easier to follow and consistent (#43707) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-06-04 14:36:32 -04:00
Nick Hill cab5c9a2a9 [Core] Move max_concurrent_batches to VllmConfig (#44274) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-02 08:57:25 -07:00
Raj JoshiandCursor 9090368b65 [Feat] Add support for per GPU worker RDMA NIC selection (#42083) 
Signed-off-by: Raj Joshi <rajjoshi@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-05-28 12:45:23 -07:00
shaharmor98andmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> f9f770ca0b fix nixl side-channel host selection (#41806) 
Signed-off-by: Shahar Mor <smor@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-11 07:40:37 +00:00
Nick Hill 10ebb40d62 [Core] Avoid using extra thread in UniProcExecutor (#40891) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-05-07 15:33:00 -07:00
wang.yuqi a8208e6a81 [Examples] Resettle features examples. (#40995) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-04-28 00:33:41 -07:00
Wentao Yeandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 8f87eb4622 [Refactor] Clean up log once scope="local" (#40540) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-04-22 16:42:43 -04:00
Li, Jiang d02421a7db [CPU] Refactor CPU affinity and memory management (#39781) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2026-04-17 21:01:08 +08:00
Nikita Shapovalov 692db29cd4 [Bugfix] Fix Ray compiled-DAG SHM channel stalls by detaching zero-copy np.ndarray logprobs buffers (#35736) 
Signed-off-by: Nikita Shapovalov <nikita@poolside.ai>
 2026-04-16 23:49:29 +08:00
Lucas Kabela c08f3b2a62 Measure encoder compile time seperate from llm backbone (#39240) 
Signed-off-by: Lucas Kabela <lucaskabela@meta.com>
 2026-04-14 10:52:49 -04:00
Shengqi ChenJason LiRoger Wang
 		75e01a39a1 [Feature] NUMA binding support for GPU workers (#38635) 
Signed-off-by: Shengqi Chen <harry-chen@outlook.com>
Co-authored-by: Jason Li <jasonlizhengjian@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-04-08 09:55:24 -07:00
Wentao Ye 4ae218c122 [Refactor] Remove unused dead code (#38842) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-04-06 11:52:05 -04:00
Anton Ivanov abebd9323d [CPU] Replace OMP initialization (#36487) 
Signed-off-by: Anton Ivanov <anton.ivanov@cambridgegreys.com>
 2026-04-03 18:42:43 +08:00
Nick Hill 6241521dd2 [BugFix] Fix precommit breakage due to conflicting in-flight merges (#38759) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-04-01 15:35:06 -07:00
Jeffrey Wang de5e6c44c6 [Feat][Executor] Introduce RayExecutorV2 (#36836) 
Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
 2026-04-01 14:34:29 -07:00