[RL] Support checkpoint-coordinate sparse NCCL weight updates (#53751)

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
This commit is contained in:
Shuolei Wang
2026-08-27 18:19:51 +08:00
committed by GitHub
co-authored by aoshen02 OpenAI Codex
parent 50708f95ae
commit 6c4aa8bec3
5 changed files with 435 additions and 797 deletions
+1 -1
View File
@@ -33,7 +33,7 @@ engine drives on your behalf:
| ------- | --------- | -------- |
| [NCCL](nccl.md) | NCCL broadcast | Separate GPUs for training and inference |
| [IPC](ipc.md) | CUDA IPC handles | Colocated training and inference on same GPU |
| [sparse_nccl](nccl.md#sparse-nccl) | NCCL broadcast | Sparse flat-index weight patches (TP=1/PP=1) |
| [sparse_nccl](nccl.md#sparse-nccl) | NCCL broadcast | Checkpoint-coordinate sparse weight patches |
| [sharded_rdt](sharded_rdt.md) | NIXL / Ray Direct Transport (pull-based) | Very large models where each worker needs only its own slice (MoE with expert parallelism) |
## Quickstart
+27 -21
View File
@@ -132,17 +132,19 @@ pass — this is the invariant to test first.
## Sparse NCCL
Sparse, flat-index weight patches use a separate backend,
`WeightTransferConfig(backend="sparse_nccl")`. It shares only NCCL process-group
initialization with the dense engine; patches are applied directly in place to
existing parameters, with no layerwise reload. The current sparse MVP requires
`TP=1` and `PP=1`, and assumes a single-rank trainer.
Sparse, flat-index weight patches use
`WeightTransferConfig(backend="sparse_nccl")`. Names, full shapes, and flat
indices are interpreted in **checkpoint/Hugging Face coordinates**. Every
inference rank receives the same checkpoint-global patch, then the model's native
`load_weights()` maps it to rank-local TP/EP and packed runtime parameters.
Sparse is a **delta** backend: each round's patches are a fresh set of deltas
from the latest optimizer step, not a stable stream of the model's parameters. So
the engine takes **no `WeightSource`** — passing one raises — and each round's
patches go straight to `send_weights(patches)`. A round with no patches is a
no-op.
Tensor parallelism is supported and does not have to match the trainer's layout.
Pipeline-parallel behavior is not covered by the current sparse NCCL GPU test.
Sparse is a **delta** backend: each call supplies replacement patches rather
than a stable stream of the model's parameters. The engine therefore takes no
`WeightSource`; patches go straight to `send_weights(patches)`, and an empty
patch list is a no-op.
```python
from vllm.distributed.weight_transfer import (
@@ -154,6 +156,7 @@ from vllm.distributed.weight_transfer.sparse_nccl_engine import (
SparseWeightPatch,
)
client = RayVLLMWeightSyncClient(llm)
engine = WeightTransferTrainerFactory.trainer_init(
init_info=SparseNCCLTrainerInitInfo(
master_address=master_address,
@@ -161,7 +164,7 @@ engine = WeightTransferTrainerFactory.trainer_init(
world_size=world_size,
rank=0,
),
client=RayVLLMWeightSyncClient(llm),
client=client,
)
patches = [
@@ -175,21 +178,24 @@ patches = [
engine.send_weights(patches)
```
`SparseNCCLTrainerInitInfo` takes the same rendezvous fields as the dense
backend and no packed params — sparse transfers are never packed.
Each `send_weights(patches)` call owns a complete one-shot `start` / `update` / `finish` lifecycle.
Patches are validated on the trainer *before* `start_weight_update`: int32
indices, 1-D, and matching `indices`/`values` lengths, with `full_shape` set. The
worker checks the same invariants when applying, but by then the broadcasts are
under way, where a size mismatch wedges both sides instead of raising.
RL infrastructure that owns the generic worker lifecycle can keep one logical update open across bounded chunks without adding trainer-side session state:
A trainer can hold a dense engine and a sparse engine at once — a full resync
through the dense engine, cheap sparse deltas in between — as
[`rlhf_sparse_nccl.py`](../../../examples/rl/rlhf_sparse_nccl.py) demonstrates.
```python
client.start_weight_update()
for patches in patch_chunks:
engine.send_weight_chunk(patches)
client.finish_weight_update()
```
Sparse NCCL sends only `O(nnz)` indices and values. Checkpoint application still uses `O(N)` staging through the native loader, and the caller owns export/diff state and restart or reseed after a partial failure.
[`rlhf_sparse_nccl.py`](../../../examples/rl/rlhf_sparse_nccl.py) demonstrates per-expert checkpoint patches with a Qwen3 MoE TP2/EP2 engine.
## Examples
- [RLHF with NCCL weight syncing (`vllm serve`, HTTP)](../../../examples/rl/rlhf_http_nccl.py) - **Start here.** Trainer on one GPU, 2x tensor-parallel fp8 server on two others; HTTP control plane, NCCL data plane. Launches and tears down its own server
- [RLHF with NCCL + FSDP2 and expert parallelism](../../../examples/rl/rlhf_nccl_fsdp_ep.py) - Multi-rank trainer: every FSDP rank builds an engine and joins the `full_tensor()` gather while only rank 0 touches the wire
- [RLHF with sparse NCCL weight syncing (offline, Ray)](../../../examples/rl/rlhf_sparse_nccl.py) - Dense-vs-sparse equivalence demo with a real model on a 2-GPU trainer/inference setup; sparse patches use `backend="sparse_nccl"` and currently require `TP=1` and `PP=1`
- [RLHF with sparse NCCL weight syncing (offline, Ray)](../../../examples/rl/rlhf_sparse_nccl.py) - Qwen3 MoE per-expert checkpoint updates with one trainer GPU and two TP2/EP2 inference GPUs
- [RLHF with async weight syncing (offline, Ray)](../../../examples/rl/rlhf_async_new_apis.py) - Async generation with mid-flight pause, weight sync, resume, and validation against a fresh model; uses an in-process `AsyncLLMEngine` with `RayVLLMWeightSyncClient`