[Docs][RL] Rewrite weight-transfer docs; standardize examples (#51729)
Signed-off-by: haoaaron <ahao@anyscale.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
9b3382685e
commit
793ca6998a
@@ -8,111 +8,109 @@ The IPC weight transfer engine uses **CUDA IPC** (Inter-Process Communication) h
|
||||
|
||||
## How It Works
|
||||
|
||||
1. The trainer creates CUDA tensors for each weight and generates IPC handles using `torch.multiprocessing.reductions.reduce_tensor`. In multi-GPU setups (e.g. FSDP), each trainer rank must all-gather the full tensor for each layer onto its own GPU before generating the IPC handle.
|
||||
2. IPC handles for each gpu are sent to the inference engine via **Ray**, **HTTP**, or a **custom callable**. Each rank only reads the handle corresponding to its own GPU.
|
||||
1. The trainer creates CUDA tensors for each weight and generates IPC handles using `torch.multiprocessing.reductions.reduce_tensor`. In multi-GPU setups (e.g. FSDP), each trainer rank materializes the full tensor for each parameter onto its own GPU before generating the handle — which `ModuleSource` does for you.
|
||||
2. Every trainer rank contributes its handles to an all-gather; the sender merges them so each payload maps every GPU UUID to its args, and ships the merged handles to the inference engine through the client. Each worker reads only the handle for its own GPU.
|
||||
3. The inference worker reconstructs the tensors from the handles using `rebuild_cuda_tensor`, reading directly from the trainer's GPU memory.
|
||||
|
||||
Unlike NCCL, IPC transfer is straight-line: `update_weights` *is* the transfer,
|
||||
and it rides the client, so there is no concurrent broadcast to overlap with.
|
||||
|
||||
!!! note
|
||||
The handle all-gather in step 2 runs over the **default** process group, so it
|
||||
assumes that group is exactly the set of colocated trainer ranks and that the
|
||||
sender is a member. It is a no-op when no distributed group exists.
|
||||
|
||||
!!! warning
|
||||
IPC handles involve sending serialized Python objects. When using HTTP transport, you must set `VLLM_ALLOW_INSECURE_SERIALIZATION=1` on both the server and client. This is because IPC handles are pickled and base64-encoded for HTTP transmission.
|
||||
|
||||
## Inference Side
|
||||
|
||||
```python
|
||||
from vllm import LLM
|
||||
from vllm.config import WeightTransferConfig
|
||||
|
||||
llm = LLM(model="my-model", weight_transfer_config=WeightTransferConfig(backend="ipc"))
|
||||
```
|
||||
|
||||
```bash
|
||||
vllm serve my-model --weight-transfer-config '{"backend": "ipc"}'
|
||||
```
|
||||
|
||||
IPC needs no data-plane rendezvous, so `init_transfer_engine` opens no channel —
|
||||
it only records the `packed` flag the trainer ships at the handshake, which
|
||||
`receive_weights` then reads. Whether a transfer is packed is therefore never
|
||||
something you configure on the inference side.
|
||||
|
||||
## Trainer Side
|
||||
|
||||
```python
|
||||
from vllm.distributed.weight_transfer import (
|
||||
ModuleSource,
|
||||
HTTPVLLMWeightSyncClient,
|
||||
WeightTransferTrainerFactory,
|
||||
)
|
||||
from vllm.distributed.weight_transfer.ipc_engine import IPCTrainerInitInfo
|
||||
|
||||
engine = WeightTransferTrainerFactory.trainer_init(
|
||||
init_info=IPCTrainerInitInfo(rank=0, packed=False), # rank 0 is the sender
|
||||
client=HTTPVLLMWeightSyncClient("http://localhost:8000"),
|
||||
source=ModuleSource(model),
|
||||
)
|
||||
|
||||
engine.send_weights() # once per sync
|
||||
```
|
||||
|
||||
`send_weights()` drives `start_weight_update`, the transfer itself, and
|
||||
`finish_weight_update`, and holds strong references to the IPC-shared copies
|
||||
until after the post-send barrier — the consumer's views would otherwise dangle.
|
||||
|
||||
Any [`VLLMWeightSyncClient`](base.md#vllmweightsyncclient) works here — the
|
||||
built-in HTTP and Ray clients, or an adapter for your own stack. The engine is
|
||||
identical either way.
|
||||
|
||||
### `IPCTrainerInitInfo`
|
||||
|
||||
| Field | Default | Description |
|
||||
| ----- | ------- | ----------- |
|
||||
| `rank` | — | Keyword-only. This trainer process's rank; **0 is the sender** |
|
||||
| `packed` | `False` | Chunked, bounded-memory transfer (see below) |
|
||||
| `packed_buffer_size_bytes` | 1 GiB | Chunk size when `packed=True` |
|
||||
|
||||
`packed` is a must-agree wire param: `trainer_init` ships it to the worker, which
|
||||
records it and decodes accordingly. It is not a `WeightTransferConfig` field and
|
||||
not a per-round `update_weights` field. `packed_buffer_size_bytes` is
|
||||
producer-only — the consumer rebuilds from the IPC handle plus the per-chunk
|
||||
`tensor_sizes`, so it never needs the buffer size.
|
||||
|
||||
## Packed (Chunked) Transfer
|
||||
|
||||
By default, all weights are sent in a single API call. For large models, this requires the full model to reside in GPU memory on both sides simultaneously. Setting `packed=True` enables **chunked transfer** with bounded GPU memory:
|
||||
By default, all weights are sent in a single `update_weights` call. For large
|
||||
models, this requires the full model to reside in GPU memory on both sides
|
||||
simultaneously. Setting `packed=True` enables **chunked transfer** with bounded
|
||||
GPU memory:
|
||||
|
||||
- Weights are concatenated into fixed-size packed buffers (controlled by `packed_buffer_size_bytes`).
|
||||
- Weights are concatenated into fixed-size packed buffers (`packed_buffer_size_bytes`).
|
||||
- Each chunk is sent as a separate `update_weights` call within a single `start_weight_update` / `finish_weight_update` bracket, so the layerwise reload pass is initialized once at the start and finalized once at the end regardless of chunk count.
|
||||
- After each chunk is consumed, the GPU memory for that chunk can be reclaimed.
|
||||
|
||||
```python
|
||||
trainer_args = IPCTrainerSendWeightsArgs(
|
||||
send_mode="ray",
|
||||
llm_handle=llm_actor_handle,
|
||||
packed=True,
|
||||
packed_buffer_size_bytes=256 * 1024 * 1024, # 256 MB chunks
|
||||
engine = WeightTransferTrainerFactory.trainer_init(
|
||||
init_info=IPCTrainerInitInfo(
|
||||
rank=0,
|
||||
packed=True,
|
||||
packed_buffer_size_bytes=256 * 1024 * 1024, # 256 MB chunks
|
||||
),
|
||||
client=client,
|
||||
source=ModuleSource(model),
|
||||
)
|
||||
```
|
||||
|
||||
## Initialization
|
||||
|
||||
The IPC backend requires no initialization on either side. The `init_transfer_engine` call is a no-op for IPC.
|
||||
|
||||
## Sending Weights
|
||||
|
||||
IPC supports two transport modes for delivering the handles:
|
||||
|
||||
### Ray Mode
|
||||
|
||||
Used when vLLM is running as a Ray actor:
|
||||
|
||||
```python
|
||||
from vllm.distributed.weight_transfer.ipc_engine import (
|
||||
IPCTrainerSendWeightsArgs,
|
||||
IPCWeightTransferEngine,
|
||||
)
|
||||
|
||||
trainer_args = IPCTrainerSendWeightsArgs(
|
||||
send_mode="ray",
|
||||
llm_handle=llm_actor_handle,
|
||||
)
|
||||
# start
|
||||
ray.get(llm_actor_handle.start_weight_update.remote())
|
||||
# send weights
|
||||
IPCWeightTransferEngine.trainer_send_weights(
|
||||
iterator=model.named_parameters(),
|
||||
trainer_args=trainer_args,
|
||||
)
|
||||
# finish
|
||||
ray.get(llm_actor_handle.finish_weight_update.remote())
|
||||
```
|
||||
|
||||
In Ray mode, the engine calls `llm_handle.update_weights.remote(...)` directly, passing the IPC handles via Ray's serialization.
|
||||
|
||||
### HTTP Mode
|
||||
|
||||
Used when vLLM is running as an HTTP server:
|
||||
|
||||
```python
|
||||
trainer_args = IPCTrainerSendWeightsArgs(
|
||||
send_mode="http",
|
||||
url="http://localhost:8000",
|
||||
)
|
||||
|
||||
# start
|
||||
base_url = "http://localhost:8000"
|
||||
url = f"{base_url}/start_weight_update"
|
||||
response = requests.post(url, json={}, timeout=60)
|
||||
response.raise_for_status()
|
||||
# send weights
|
||||
IPCWeightTransferEngine.trainer_send_weights(
|
||||
iterator=model.named_parameters(),
|
||||
trainer_args=trainer_args,
|
||||
)
|
||||
# finish
|
||||
url = f"{base_url}/finish_weight_update"
|
||||
response = requests.post(url, json={}, timeout=60)
|
||||
response.raise_for_status()
|
||||
```
|
||||
|
||||
In HTTP mode, IPC handles are pickled, base64-encoded, and sent as JSON to the `/update_weights` endpoint. Because the worker deserializes the payload via `pickle.loads`, the vLLM server must be started with `VLLM_ALLOW_INSECURE_SERIALIZATION=1`.
|
||||
|
||||
```python
|
||||
def my_custom_sender(update_info: IPCWeightTransferUpdateInfo):
|
||||
# Custom logic to deliver update_info to vLLM
|
||||
...
|
||||
|
||||
trainer_args = IPCTrainerSendWeightsArgs(
|
||||
send_mode=my_custom_sender,
|
||||
)
|
||||
|
||||
IPCWeightTransferEngine.trainer_send_weights(
|
||||
iterator=model.named_parameters(),
|
||||
trainer_args=trainer_args,
|
||||
)
|
||||
```
|
||||
|
||||
See [`IPCTrainerSendWeightsArgs`](https://github.com/vllm-project/vllm/blob/main/vllm/distributed/weight_transfer/ipc_engine.py) for the full list of configurable fields.
|
||||
On a multi-rank trainer the producer reuses one buffer across chunks, so packed
|
||||
mode carries a per-chunk barrier across ranks: without it, a rank could overwrite
|
||||
its buffer while its colocated worker is still reading the current chunk. This is
|
||||
handled inside `send_weights()`.
|
||||
|
||||
## Examples
|
||||
|
||||
- [RLHF with IPC weight syncing (offline, Ray)](../../../examples/rl/rlhf_ipc.py) - Colocated training and inference on a single GPU using Ray placement groups and CUDA IPC handles
|
||||
- [RLHF with IPC weight syncing (online serving, HTTP)](../../../examples/rl/rlhf_http_ipc.py) - Weight transfer with a vLLM HTTP server where both server and trainer share the same GPU
|
||||
- [RLHF with IPC weight syncing (`vllm serve`, HTTP)](../../../examples/rl/rlhf_http_ipc.py) - **Start here.** Server and training model share a single GPU; HTTP control plane, CUDA IPC data plane. Launches and tears down its own server
|
||||
- [RLHF with IPC + FSDP2 and expert parallelism](../../../examples/rl/rlhf_ipc_fsdp_ep.py) - Multi-rank trainer colocated with a `--data-parallel-size 4` server on the same 4 GPUs: every FSDP rank builds an engine and joins the handle all-gather, with packed chunking and sleep/wake around the transfer
|
||||
|
||||
Reference in New Issue
Block a user