[Docs][RL] Rewrite weight-transfer docs; standardize examples (#51729)
Signed-off-by: haoaaron <ahao@anyscale.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
9b3382685e
commit
793ca6998a
@@ -4,12 +4,28 @@ vLLM provides a pluggable weight transfer system for synchronizing model weights
|
||||
|
||||
## Architecture
|
||||
|
||||
The weight transfer system follows a **four-phase protocol** with a pluggable backend design:
|
||||
Weight transfer has **two engines, one per process**, and they are symmetric:
|
||||
|
||||
1. **Initialization** (`init_weight_transfer_engine`): Establishes the communication channel between the trainer and inference workers. Called once before the training loop begins.
|
||||
2. **Start** (`start_weight_update`): Prepares the inference engine for a weight update.
|
||||
3. **Weight Update** (`update_weights`): Transfers updated weights from the trainer to the inference engine. May be called one or more times (e.g., for chunked transfers).
|
||||
4. **Finish** (`finish_weight_update`): Finalizes the weight update (e.g., runs post-processing for checkpoint-format weights). Called once after all weights have been transferred.
|
||||
| | Trainer process | Inference workers |
|
||||
| --- | --- | --- |
|
||||
| Class | `TrainerWeightTransferEngine` | `WeightTransferEngine` |
|
||||
| Built by | `WeightTransferTrainerFactory.trainer_init(...)` | vLLM, from `WeightTransferConfig` |
|
||||
| Driven by | `send_weights()` | the four-phase protocol below |
|
||||
| Holds | communicators, transfer plans, wire params | communicators, the target model |
|
||||
|
||||
The trainer engine is stateful: it owns its communicators and wire params, pulls
|
||||
weights from a [`WeightSource`](base.md#weightsource), and drives the inference
|
||||
side through a [`VLLMWeightSyncClient`](base.md#vllmweightsyncclient). Trainer
|
||||
code never has to know the transport, and never has to thread transfer state
|
||||
back in on every round — one `send_weights()` call per sync.
|
||||
|
||||
Under the hood every round is the same **four-phase protocol**, which the trainer
|
||||
engine drives on your behalf:
|
||||
|
||||
1. **Initialization** (`init_weight_transfer_engine`): establishes the communication channel between the trainer and inference workers. Called once, from `trainer_init`, before the training loop begins.
|
||||
2. **Start** (`start_weight_update`): prepares the inference engine for a weight update.
|
||||
3. **Weight Update** (`update_weights`): transfers updated weights. May be called one or more times (e.g. for chunked transfers).
|
||||
4. **Finish** (`finish_weight_update`): finalizes the update (e.g. runs post-processing for checkpoint-format weights). Called once after all weights have been transferred.
|
||||
|
||||
## Available Backends
|
||||
|
||||
@@ -19,11 +35,12 @@ The weight transfer system follows a **four-phase protocol** with a pluggable ba
|
||||
| [IPC](ipc.md) | CUDA IPC handles | Colocated training and inference on same GPU |
|
||||
| [sparse_nccl](nccl.md#sparse-nccl) | NCCL broadcast | Sparse flat-index weight patches (TP=1/PP=1) |
|
||||
|
||||
## Configuration
|
||||
## Quickstart
|
||||
|
||||
Specify the weight transfer backend through `WeightTransferConfig`. The backend determines which engine handles the weight synchronization.
|
||||
### Inference Side
|
||||
|
||||
### Programmatic (Offline Inference)
|
||||
The inference side takes only a backend name. Everything else about the transfer
|
||||
is decided by the trainer and shipped over at the init handshake.
|
||||
|
||||
```python
|
||||
from vllm import LLM
|
||||
@@ -31,22 +48,100 @@ from vllm.config import WeightTransferConfig
|
||||
|
||||
llm = LLM(
|
||||
model="my-model",
|
||||
weight_transfer_config=WeightTransferConfig(backend="nccl"), # or "ipc"
|
||||
weight_transfer_config=WeightTransferConfig(backend="nccl"), # or "ipc", "sparse_nccl"
|
||||
)
|
||||
```
|
||||
|
||||
### CLI (Online Serving)
|
||||
Or, for online serving:
|
||||
|
||||
```bash
|
||||
vllm serve my-model \
|
||||
--weight-transfer-config '{"backend": "nccl"}'
|
||||
```
|
||||
|
||||
The `backend` field accepts `"nccl"` (default), `"ipc"`, or `"sparse_nccl"`.
|
||||
### Trainer Side
|
||||
|
||||
Build the engine once, then call `send_weights()` once per sync:
|
||||
|
||||
```python
|
||||
from vllm.distributed.weight_transfer import (
|
||||
ModuleSource,
|
||||
HTTPVLLMWeightSyncClient,
|
||||
WeightTransferTrainerFactory,
|
||||
)
|
||||
from vllm.distributed.weight_transfer.nccl_engine import NCCLTrainerInitInfo
|
||||
|
||||
# Once, before the training loop.
|
||||
engine = WeightTransferTrainerFactory.trainer_init(
|
||||
init_info=NCCLTrainerInitInfo(
|
||||
master_address=master_address,
|
||||
master_port=master_port,
|
||||
world_size=world_size, # trainer + all inference workers
|
||||
rank=0, # this trainer rank; rank 0 is the sender
|
||||
packed=True,
|
||||
),
|
||||
client=HTTPVLLMWeightSyncClient("http://localhost:8000"),
|
||||
source=ModuleSource(model),
|
||||
)
|
||||
|
||||
# Once per weight sync.
|
||||
for step in range(num_steps):
|
||||
train_one_step(model)
|
||||
engine.send_weights()
|
||||
```
|
||||
|
||||
`send_weights()` drives start → update → finish on the inference side *and* the
|
||||
data-plane transfer, including any concurrency the backend needs (NCCL, for
|
||||
example, must run the workers' `update_weights` at the same time as the
|
||||
trainer's broadcast, since both rendezvous inside the same NCCL calls).
|
||||
|
||||
There is no `backend=` argument on the trainer side: each `TrainerInitInfo`
|
||||
subclass declares its own `backend`, and the factory dispatches on it.
|
||||
|
||||
## Where Each Setting Lives
|
||||
|
||||
The trainer is the single source of truth for anything the two sides must agree
|
||||
on. This is the main rule to internalize:
|
||||
|
||||
| Setting | Lives on | Notes |
|
||||
| ------- | -------- | ----- |
|
||||
| Backend, inference side | `WeightTransferConfig(backend=...)` | A plain string selector — that is all this config holds |
|
||||
| Backend, trainer side | `init_info.backend` | A `ClassVar` on each init info; never passed by hand |
|
||||
| Wire params (`packed`, buffer sizes) | the backend's `TrainerInitInfo` | The trainer propagates them to the worker during `trainer_init`, so the two sides *cannot* disagree |
|
||||
| Weights to send | a [`WeightSource`](base.md#weightsource) | `ModuleSource(model)` covers plain and FSDP-sharded modules |
|
||||
| Per-round payload | the backend's update info | Built by the engine (from `source.metadata()`, or from a delta backend's per-round payload); you never construct it |
|
||||
|
||||
!!! note
|
||||
Wire params are deliberately *not* on `WeightTransferConfig` and *not* on the
|
||||
per-round update info. Because the trainer ships them at init and the worker
|
||||
reads what it was told, a mismatched `packed` flag between the two sides is
|
||||
unrepresentable rather than merely discouraged.
|
||||
|
||||
## Multi-Rank Trainers
|
||||
|
||||
For sharded trainers (FSDP, TP/PP/EP), **every trainer rank builds an engine and
|
||||
calls `send_weights()`**. Rank 0 is the sender: only it holds a communicator,
|
||||
talks to the client, and puts bytes on the wire. Non-sender ranks still iterate
|
||||
the `WeightSource`, because materializing a parameter is usually itself a
|
||||
collective (an FSDP `full_tensor()` all-gather, a Megatron export) that would
|
||||
deadlock if some ranks skipped it.
|
||||
|
||||
Pass each process's own `rank` on the init info. It is explicit rather than read
|
||||
from a global process group, which is ambiguous once several groups (FSDP / TP /
|
||||
PP / EP) exist.
|
||||
|
||||
```python
|
||||
engine = WeightTransferTrainerFactory.trainer_init(
|
||||
init_info=NCCLTrainerInitInfo(..., rank=torch.distributed.get_rank()),
|
||||
client=client,
|
||||
source=ModuleSource(model),
|
||||
)
|
||||
engine.send_weights() # called on every rank
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
When running vLLM as an HTTP server, the following endpoints are available for weight transfer:
|
||||
When running vLLM as an HTTP server, the following endpoints are available for weight transfer. `HTTPVLLMWeightSyncClient` speaks the first four for you.
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| -------- | ------ | ----------- |
|
||||
@@ -63,29 +158,9 @@ When running vLLM as an HTTP server, the following endpoints are available for w
|
||||
!!! note
|
||||
The HTTP weight transfer endpoints require `VLLM_SERVER_DEV_MODE=1` to be set.
|
||||
|
||||
## Trainer-Side API
|
||||
|
||||
Both backends provide static methods that the trainer calls to send weights. The general pattern is:
|
||||
|
||||
```python
|
||||
# 1. Initialize the transfer engine (backend-specific)
|
||||
EngineClass.trainer_init(init_info)
|
||||
|
||||
# 2. Start weight update on inference side
|
||||
llm.start_weight_update()
|
||||
|
||||
# 3. Send weights to inference workers
|
||||
EngineClass.trainer_send_weights(
|
||||
iterator=model.named_parameters(),
|
||||
trainer_args=backend_specific_args,
|
||||
)
|
||||
|
||||
# 4. Finish weight update on inference side
|
||||
llm.finish_weight_update(weight_version="step-42")
|
||||
```
|
||||
|
||||
See the [NCCL](nccl.md) and [IPC](ipc.md) pages for backend-specific trainer APIs and full examples.
|
||||
|
||||
## Extending the System
|
||||
|
||||
The weight transfer system is designed to be extensible. You can implement custom backends by subclassing `WeightTransferEngine` and registering them with the factory. See the [Base Class](base.md) page for details.
|
||||
Every piece of the system is replaceable: the weights you send (`WeightSource`),
|
||||
the control-plane transport (`VLLMWeightSyncClient`), and the transports
|
||||
themselves (both engine ABCs, each with its own factory registry). See
|
||||
[Base Classes](base.md).
|
||||
|
||||
Reference in New Issue
Block a user