[Feat][RL] IPC weight sync optimizations: multigpu support and chunked packed tensors (#37476)
Signed-off-by: ahao-anyscale <ahao@anyscale.com> Signed-off-by: hao-aaron <ahao@anyscale.com>
This commit is contained in:
@@ -1,21 +1,37 @@
|
||||
# IPC Engine
|
||||
|
||||
The IPC weight transfer engine uses **CUDA IPC** (Inter-Process Communication) handles to share GPU memory directly between the trainer and inference workers on the **same node and same GPU**. This avoids any data copying, making it a efficient option when colocating training and inference.
|
||||
The IPC weight transfer engine uses **CUDA IPC** (Inter-Process Communication) handles to share GPU memory directly between the trainer and inference workers on the **same GPU**. This avoids any data copying, making it the most efficient option when colocating training and inference. Multi-GPU setups are supported — weights are all gathered by each GPU and are extracted by the correct colocated process.
|
||||
|
||||
## When to Use IPC
|
||||
|
||||
- Training and inference on the **same GPU** (colocated)
|
||||
- You want to minimize memory overhead by sharing tensors in-place
|
||||
- Training and inference share the **same GPU(s)** (colocated)
|
||||
|
||||
## How It Works
|
||||
|
||||
1. The trainer creates CUDA tensors for each weight and generates IPC handles using `torch.multiprocessing.reductions.reduce_tensor`.
|
||||
2. IPC handles are sent to the inference engine via **Ray.remote()** or **HTTP POST**.
|
||||
3. The inference worker reconstructs the tensors from the handles, reading directly from the trainer's GPU memory.
|
||||
1. The trainer creates CUDA tensors for each weight and generates IPC handles using `torch.multiprocessing.reductions.reduce_tensor`. In multi-GPU setups (e.g. FSDP), each trainer rank must all-gather the full tensor for each layer onto its own GPU before generating the IPC handle.
|
||||
2. IPC handles for each gpu are sent to the inference engine via **Ray**, **HTTP**, or a **custom callable**. Each rank only reads the handle corresponding to its own GPU.
|
||||
3. The inference worker reconstructs the tensors from the handles using `rebuild_cuda_tensor`, reading directly from the trainer's GPU memory.
|
||||
|
||||
!!! warning
|
||||
IPC handles involve sending serialized Python objects. When using HTTP transport, you must set `VLLM_ALLOW_INSECURE_SERIALIZATION=1` on both the server and client. This is because IPC handles are pickled and base64-encoded for HTTP transmission.
|
||||
|
||||
## Packed (Chunked) Transfer
|
||||
|
||||
By default, all weights are sent in a single API call. For large models, this requires the full model to reside in GPU memory on both sides simultaneously. Setting `packed=True` enables **chunked transfer** with bounded GPU memory:
|
||||
|
||||
- Weights are concatenated into fixed-size packed buffers (controlled by `packed_buffer_size_bytes`).
|
||||
- Each chunk is sent as a separate `update_weights` call within a single `start_weight_update` / `finish_weight_update` bracket, so the layerwise reload pass is initialized once at the start and finalized once at the end regardless of chunk count.
|
||||
- After each chunk is consumed, the GPU memory for that chunk can be reclaimed.
|
||||
|
||||
```python
|
||||
trainer_args = IPCTrainerSendWeightsArgs(
|
||||
send_mode="ray",
|
||||
llm_handle=llm_actor_handle,
|
||||
packed=True,
|
||||
packed_buffer_size_bytes=256 * 1024 * 1024, # 256 MB chunks
|
||||
)
|
||||
```
|
||||
|
||||
## Initialization
|
||||
|
||||
The IPC backend requires no initialization on either side. The `init_transfer_engine` call is a no-op for IPC.
|
||||
@@ -35,7 +51,7 @@ from vllm.distributed.weight_transfer.ipc_engine import (
|
||||
)
|
||||
|
||||
trainer_args = IPCTrainerSendWeightsArgs(
|
||||
mode="ray",
|
||||
send_mode="ray",
|
||||
llm_handle=llm_actor_handle,
|
||||
)
|
||||
# start
|
||||
@@ -57,7 +73,7 @@ Used when vLLM is running as an HTTP server:
|
||||
|
||||
```python
|
||||
trainer_args = IPCTrainerSendWeightsArgs(
|
||||
mode="http",
|
||||
send_mode="http",
|
||||
url="http://localhost:8000",
|
||||
)
|
||||
|
||||
@@ -77,7 +93,22 @@ response = requests.post(url, json={}, timeout=60)
|
||||
response.raise_for_status()
|
||||
```
|
||||
|
||||
In HTTP mode, IPC handles are pickled, base64-encoded, and sent as JSON to the `/update_weights` endpoint. As with Ray mode, you must call `start_weight_update` before and `finish_weight_update` after.
|
||||
In HTTP mode, IPC handles are pickled, base64-encoded, and sent as JSON to the `/update_weights` endpoint. Because the worker deserializes the payload via `pickle.loads`, the vLLM server must be started with `VLLM_ALLOW_INSECURE_SERIALIZATION=1`.
|
||||
|
||||
```python
|
||||
def my_custom_sender(update_info: IPCWeightTransferUpdateInfo):
|
||||
# Custom logic to deliver update_info to vLLM
|
||||
...
|
||||
|
||||
trainer_args = IPCTrainerSendWeightsArgs(
|
||||
send_mode=my_custom_sender,
|
||||
)
|
||||
|
||||
IPCWeightTransferEngine.trainer_send_weights(
|
||||
iterator=model.named_parameters(),
|
||||
trainer_args=trainer_args,
|
||||
)
|
||||
```
|
||||
|
||||
See [`IPCTrainerSendWeightsArgs`](https://github.com/vllm-project/vllm/blob/main/vllm/distributed/weight_transfer/ipc_engine.py) for the full list of configurable fields.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user