iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
22,547 Commits 3 Branches 0 Tags
eb68fb9105eab33888ce252001a269a8c52c19a5
Commit Graph
186 Commits
Author SHA1 Message Date
MOVZXandHinata 036608b012 [Engine] Detect a fail-closed fault or death on any rank 
A fault on a non-zero TP rank could hang the engine: the worker's sentinel pipe
was inherited by a helper process and stayed open, so the monitor never saw
the exit. Earlier releases carry the same monitor and fail-closed exit path.

- Worker exit is detected by pidfd, with the sentinel as fallback, before and
  after READY and for every worker through the whole initialisation window.
- A per-worker fault pipe: ple_fail_closed writes one byte before os._exit and
  the monitor starts the failure ladder at once; the fd stays registered
  through worker teardown.
- worker_main marks inherited pipes close-on-exec.
- Any unexpected busy-loop failure exits 1; the termination ladder uses
  time.monotonic(); forced-exit paths no longer wait on logging.
- Tests on real fork/spawn workers for rank 0 and rank 1, with and without an
  fd-inheriting helper.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-09 20:04:37 +07:00
MOVZXandHinata 6c956ee3bd [Engine] Drain-order shutdown 
- EngineCore survives a late SIGTERM during teardown (bounded deferral) so
  every worker drains its guard telemetry.
- A worker exits right after its orderly shutdown.
- A worker failure racing shutdown stays a failure (exit 1).
- Rank 0 waits for the PLE child's whole thread group, bounded, instead of its
  reap; drain-aware failure ladder with a bounded grace for survivors whose
  shutdown already started.
- tests: the sorted-emit graph arm takes max_len on the host before capture.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-09 20:04:37 +07:00
MOVZXandHinata 23349e43b2 [PLE] Integrate the pull transport on v0.30.0 with upstream UVA refused 
- VLLM_E2_PLE_PULL_TRANSPORT replaces the fork's old switch name, which
  v0.30.0 reuses for its UVA backend with the opposite default; strict parser;
  compile/capture factor.
- The pull transport is reachable on v0.30.0 through a pull embedding class
  that inherits upstream's without editing it; the upstream UVA backend is
  provably off while the pull switch is on.
- resolve_ple_mode()/assert_ple_mode() log and assert the effective PLE mode
  for target and drafter, and refuse a mixed configuration.
- Producer log lines for the PLE backend and weight dtype.
- CPU tests for mode resolution and the mixed-config refusal.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-09 19:52:54 +07:00
MOVZXandHinata 025fd08603 [PLE] Carry the host-mapped pull transport onto v0.30.0 
Restores the fork's PLE offload package (not upstream) and its pull
transport: the offload process holds no CUDA context; the captured op waits on
a device-mapped host flag and copies host->GPU on the model stream; publication
is fenced; releases are synchronous across dummy/capture work; failures close
the rank. Includes the test-only fault-injection cuts for the transport.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-09 19:49:16 +07:00
MOVZXandHinata 17c1c70447 [Spec Decode] Structured output under concurrency on a PP shape 
Ports upstream vLLM PRs #54442 and #56802 (issue #54437): never sample from an
unmasked row, and key draft-token hand-off snapshots by scheduler step so an
alternating PP microbatch cannot overwrite the slot.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-09 19:43:58 +07:00
MOVZXandHinata 66b3d8c108 [Engine] Workers cannot outlive a killed EngineCore 
Workers set PR_SET_PDEATHSIG(SIGKILL) against their EngineCore parent with a
getppid race check. Start methods other than fork/spawn are refused before any
worker is created. Subprocess tests cover fork, spawn, graceful SIGTERM and the
refusal.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-09 19:43:53 +07:00
Chandra MouryaandVS Chandra Mourya 8a2364605c [Core] Bound draft-token RPC waits by the execute-model timeout (#58779) 
Signed-off-by: VS Chandra Mourya <219748331+vschandramourya@users.noreply.github.com>
Co-authored-by: VS Chandra Mourya <219748331+vschandramourya@users.noreply.github.com>
 2026-09-26 00:40:22 +00:00
Mark McLoughlinandAI Assistant 45d56c0baf [Core] Model console logging as CLI configuration (#57205) 
Add `--logging-config` CLI argument which can be supplied as
JSON or using dotted arguments. The `--log-level` argument
is provided for convenience, and `--log-config-file` is deprecated
in favor of `--logging-config.pylogging_config_file`.

Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: AI Assistant <noreply@openai.com>
 2026-09-25 11:36:59 +01:00
Hongxin Xuaoshen02OpenAI CodexRoger WangNick HillCodex
 		0b7f11a1ee [AuxOutput] Add block-keyed storage for routed-expert outputs (#45635) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-21 10:25:05 +00:00
Dakai AnAlanFokCoaoshen02OpenAI Codex
 		092bdd6d57 [Frontend][Core] Add release_kv_cache_memory() API (#44890) 
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: AlanFokCo <alanfok2868@gmail.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com>
Co-authored-by: AlanFokCo <alanfok2868@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-09-18 07:12:12 +08:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 16:44:29 +00:00
Wentao Ye 6b15bea080 [Refactor] Remove utils dead code (#53941) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-09-08 11:22:09 -04:00
yzong-rh 680e2177e4 [CI][Ray] Fix flaky multi-node assignment test after placement-group teardown (#53621) 2026-08-29 23:48:22 +00:00
Ace EldeibandCodex 9662ab0835 [Bugfix] Set breakable graph env before Ray actor import (#53293) 
Signed-off-by: Ace Eldeib <aeldeib@coreweave.com>
Co-authored-by: Codex <codex@openai.com>
 2026-08-28 23:11:50 +00:00
Charlie Fu fd98d32f73 [ROCm][CI] Fix test_ray_v2_executor (#54249) 
Signed-off-by: charlifu <charlifu@amd.com>
 2026-08-28 15:11:42 -07:00
aoshen02aoshen02ClaudeOpenAI CodexAndreas Karatzas
 		67e86d1e6f [BugFix] Bind RayExecutorV2 TCPStore before publishing its port (#50969) 
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-08-28 12:27:22 -07:00
Jeffrey Wang 79378fe6e8 [Bugfix] Avoid TCPStore port collision for co-located non-DP Ray engines (#53666) 
Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
 2026-08-27 19:58:14 +00:00
石皮幼鸟andOpenAI Codex 4ca856b0b5 [Bugfix] Release worker RPC payload before next dequeue (#51979) 
Signed-off-by: 石皮幼鸟 <2960474346@qq.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-24 07:27:36 -07:00
Aaron Hao 6cd9713463 [RL] P2P RDT weight sync (#43375) 2026-08-23 09:42:22 -07:00
Lucas Wilkinson 8bdc70ec7b [6/N][KV-Cache Layout Refactor] Standardize KV cache layout (#51718) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-08-21 17:47:49 -07:00
crZhao 3fc2893909 [Bugfix] Account for local DP workers in startup thread allocation (#52385) 
Signed-off-by: real-cpu <zhaochenrui757@gmail.com>
 2026-08-17 12:10:34 -07:00
omerpaz95yewentao256Wentao Ye
 		fa9d67f782 [EC Connector] Added Build Connector Worker Meta for EC Connector (#49585) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Signed-off-by: omerpaz95 <73347585+omerpaz95@users.noreply.github.com>
Co-authored-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-08-15 15:40:38 -07:00
Connor CarpenterandNick Hill f473870ecf [Rust Frontend][gRPC] Add RL lifecycle control (#51316) 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-08-14 11:32:39 -07:00
vllmellmandtjtanaa 436be94e13 [ROCm][Bugfix] Use TCP store when AITER custom all-reduce is enabled (#51635) 
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
 2026-08-10 11:07:14 +00:00
aoshen02Parsa Mazaheriaoshen02Claude
 		11ba93f364 [BugFix] Use file:// rendezvous for single-node executors to eliminate startup port races (#50999) 
Signed-off-by: Parsa Mazaheri <p_mazaheri@apple.com>
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Parsa Mazaheri <p_mazaheri@apple.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-08-09 22:00:04 -04:00
Nick HillandClaude 7635a9002b [Core] Explicitly manage torch CPU threads in workers (#49919) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-08-04 10:06:38 -07:00
aoshen02andNick Hill 42ab184ea7 [MRV2] Enable routed-experts capture (#50721) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-08-03 21:00:13 +00:00
bugkeepandClaude 48aa8d8d75 [Bugfix] Prevent stale multiproc RPC deadlines from becoming unbounded waits (#41357) 
Signed-off-by: bugkeep <1921817430@qq.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-07-29 16:41:34 -07:00
Itay Alroy 35efdf6b34 [Elastic EP] Async preparation (#47288) 
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
 2026-07-28 05:19:13 -07:00
rasmithandAndreas Karatzas 60d443f738 [CI/Build][The Rock][BugFix] Use fork method in test_multiproc_executor_multi_node for py 3.14 compat and fix test_multiproc_executor_shutdown_cleanup (#48655) 
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-07-21 15:40:29 -05:00
Nick CaoandClaude 920469974a [UX] Log worker exit code when process dies unexpectedly (#38641) 
Signed-off-by: Nick Cao <ncao@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-07-07 12:36:29 +00:00
fangyuchuTyler Michael Smithmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		245888ff77 [Feature] Detect all2all peer fault with fault tolerance backend and prevent corrupted output (#43637) 
Signed-off-by: fangyuchu <fangyuchu@qq.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-30 09:00:25 -07:00
Seiji Eicherandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> ded6676458 [Bugfix] Seed RayExecutorV2 TCPStore port by DP rank to avoid collisions (#45960) 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-30 07:37:34 -07:00
Tyler Michael SmithClaudeCodexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>kourosh hakhamaneshi
 		ebfbcfe46a Stop setting CUDA_VISIBLE_DEVICES internally in vLLM, add device_ids arg (#45026) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: kourosh hakhamaneshi <kouroshHakha@users.noreply.github.com>
 2026-06-20 13:38:10 -07:00
Nick Hill cf027b86af [Core] Simplify MRV2 async output handling (#45442) 2026-06-13 18:15:36 -07:00
Ryan Rockandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> aab639c705 [Core][AMD] Propagate shutdown timeout to MultiprocExecutor (#43154) 
Signed-off-by: Ryan Rock <ryan.rock@amd.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-06-12 15:13:31 -05:00
qizixi 96229fa99e [KVConnector][1/N] PP-aware handshake aggregation and intermediate-PP output plumbing (#43720) 
Signed-off-by: zixi-qi <zixi@inferact.ai>
 2026-06-04 22:04:19 -07:00
Wentao Ye 41a4829f22 [Logs Refactor] Optimize shutdown logs, easier to follow and consistent (#43707) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-06-04 14:36:32 -04:00
Nick Hill cab5c9a2a9 [Core] Move max_concurrent_batches to VllmConfig (#44274) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-02 08:57:25 -07:00
Raj JoshiandCursor 9090368b65 [Feat] Add support for per GPU worker RDMA NIC selection (#42083) 
Signed-off-by: Raj Joshi <rajjoshi@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-05-28 12:45:23 -07:00
shaharmor98andmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> f9f770ca0b fix nixl side-channel host selection (#41806) 
Signed-off-by: Shahar Mor <smor@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-11 07:40:37 +00:00
Nick Hill 10ebb40d62 [Core] Avoid using extra thread in UniProcExecutor (#40891) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-05-07 15:33:00 -07:00
wang.yuqi a8208e6a81 [Examples] Resettle features examples. (#40995) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-04-28 00:33:41 -07:00
Wentao Yeandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 8f87eb4622 [Refactor] Clean up log once scope="local" (#40540) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-04-22 16:42:43 -04:00
Li, Jiang d02421a7db [CPU] Refactor CPU affinity and memory management (#39781) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2026-04-17 21:01:08 +08:00
Nikita Shapovalov 692db29cd4 [Bugfix] Fix Ray compiled-DAG SHM channel stalls by detaching zero-copy np.ndarray logprobs buffers (#35736) 
Signed-off-by: Nikita Shapovalov <nikita@poolside.ai>
 2026-04-16 23:49:29 +08:00
Lucas Kabela c08f3b2a62 Measure encoder compile time seperate from llm backbone (#39240) 
Signed-off-by: Lucas Kabela <lucaskabela@meta.com>
 2026-04-14 10:52:49 -04:00
Shengqi ChenJason LiRoger Wang
 		75e01a39a1 [Feature] NUMA binding support for GPU workers (#38635) 
Signed-off-by: Shengqi Chen <harry-chen@outlook.com>
Co-authored-by: Jason Li <jasonlizhengjian@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-04-08 09:55:24 -07:00
Wentao Ye 4ae218c122 [Refactor] Remove unused dead code (#38842) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-04-06 11:52:05 -04:00