[docs] Add docs for new RL flows (#36188)
Signed-off-by: ahao-anyscale <ahao@anyscale.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
This commit is contained in:
co-authored by
Harry Mellor
parent
fad09e8a1f
commit
47a1f11bff
@@ -23,15 +23,18 @@ def title(text: str) -> str:
|
||||
# Custom substitutions
|
||||
subs = {
|
||||
"io": "IO",
|
||||
"api": "API",
|
||||
"rl": "RL",
|
||||
"api(s?)": r"API\1",
|
||||
"cli": "CLI",
|
||||
"cpu": "CPU",
|
||||
"ipc": "IPC",
|
||||
"llm": "LLM",
|
||||
"mae": "MAE",
|
||||
"ner": "NER",
|
||||
"tpu": "TPU",
|
||||
"gguf": "GGUF",
|
||||
"lora": "LoRA",
|
||||
"nccl": "NCCL",
|
||||
"rlhf": "RLHF",
|
||||
"vllm": "vLLM",
|
||||
"openai": "OpenAI",
|
||||
@@ -196,6 +199,11 @@ class Example:
|
||||
|
||||
|
||||
def on_startup(command: Literal["build", "gh-deploy", "serve"], dirty: bool):
|
||||
# Monkey-patch dirname_to_title in awesome-nav so that sub-directory names are
|
||||
# title-cased (e.g. "Offline Inference" instead of "Offline inference").
|
||||
import mkdocs_awesome_nav.nav.directory as _nav_dir
|
||||
|
||||
_nav_dir.dirname_to_title = title
|
||||
logger.info("Generating example documentation")
|
||||
logger.debug("Root directory: %s", ROOT_DIR.resolve())
|
||||
logger.debug("Example directory: %s", EXAMPLE_DIR.resolve())
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
# Async Reinforcement Learning
|
||||
|
||||
## Overview
|
||||
|
||||
In a standard RL training loop, generation and training happen sequentially: the policy generates rollouts, then training runs on those rollouts, and the cycle repeats. During generation the training accelerators sit idle, and vice versa.
|
||||
|
||||
The **one-off pipelining** approach separates the generation and training phases into two parallel coroutines, allowing the model to generate new samples while simultaneously training on previously generated data. This can lead to better GPU utilization and greater training throughput.
|
||||
|
||||
However, this overlap introduces a complication: weights must be updated in the inference engine mid-flight, while requests may still be in progress.
|
||||
|
||||
## The Pause and Resume API
|
||||
|
||||
To safely update weights while the inference engine is running, vLLM provides `pause_generation` and `resume_generation` methods. These let the trainer coordinate a clean window for weight synchronization without losing in-flight work.
|
||||
|
||||
### pause_generation
|
||||
|
||||
```python
|
||||
await engine.pause_generation(mode="keep", clear_cache=True)
|
||||
```
|
||||
|
||||
The `mode` parameter controls how in-flight requests are handled:
|
||||
|
||||
| Mode | Behavior |
|
||||
| ---- | -------- |
|
||||
| `"abort"` | Abort all in-flight requests immediately and return partial results (default) |
|
||||
| `"wait"` | Wait for all in-flight requests to finish before pausing |
|
||||
| `"keep"` | Freeze requests in the queue; they resume when `resume_generation` is called |
|
||||
|
||||
The `clear_cache` parameter controls whether to clear the KV cache and prefix cache after pausing.
|
||||
|
||||
### resume_generation
|
||||
|
||||
```python
|
||||
await engine.resume_generation()
|
||||
```
|
||||
|
||||
Resumes the scheduler after a pause. Any requests frozen with `mode="keep"` will continue generating.
|
||||
|
||||
### HTTP Endpoints
|
||||
|
||||
When using the vLLM HTTP server, the same functionality is available via:
|
||||
|
||||
- `POST /pause?mode=keep` - Pause generation
|
||||
- `POST /resume` - Resume generation
|
||||
|
||||
!!! note "Data Parallelism"
|
||||
When using data parallelism with vLLM's **internal load balancer** (i.e. `data_parallel_backend="ray"`), pause and resume are handled automatically across all DP ranks -- a single call is sufficient. When using an **external load balancer** (i.e. multiple independent vLLM instances behind a proxy), you must send pause and resume requests to **every** engine instance individually before and after the weight update.
|
||||
|
||||
## Typical Async RL Flow
|
||||
|
||||
A typical async RL loop with weight syncing looks like this:
|
||||
|
||||
1. Start generating rollouts from the current policy
|
||||
2. Once trainer has new weights to update to, pause generation with `mode="keep"`
|
||||
3. Sync the updated weights from the trainer to the inference engine (see [Weight Transfer](weight_transfer/README.md))
|
||||
4. Resume generation -- in-flight requests continue with the new weights
|
||||
5. Repeat
|
||||
|
||||
The key insight is that requests paused with `mode="keep"` will produce tokens from the **old** weights before the pause and tokens from the **new** weights after resume. The `clear_cache` parameter controls whether the KV cache is invalidated during the pause. When `clear_cache=True`, previously cached key-value entries are discarded, so all tokens generated after resume will be computed entirely with the new weights. When `clear_cache=False`, existing KV cache entries are retained, meaning some tokens in context may still reflect the old weights (stale KV cache).
|
||||
|
||||
## Example
|
||||
|
||||
The [async RLHF example](../examples/rl/rlhf_async_new_apis.md) demonstrates this pattern with `vllm.AsyncLLMEngine`, NCCL weight transfer, and mid-flight pause/resume with validation.
|
||||
@@ -16,11 +16,9 @@ The following open-source RL libraries use vLLM for fast rollouts (sorted alphab
|
||||
- [Unsloth](https://github.com/unslothai/unsloth)
|
||||
- [verl](https://github.com/volcengine/verl)
|
||||
|
||||
See the following basic examples to get started if you don't want to use an existing library:
|
||||
For weight synchronization between training and inference, see the [Weight Transfer](weight_transfer/README.md) documentation, which covers the pluggable backend system with [NCCL](weight_transfer/nccl.md) (multi-GPU) and [IPC](weight_transfer/ipc.md) (same-GPU) engines.
|
||||
|
||||
- [Training and inference processes are located on separate GPUs (inspired by OpenRLHF)](../examples/offline_inference/rlhf.md)
|
||||
- [Training and inference processes are colocated on the same GPUs using Ray](../examples/offline_inference/rlhf_colocate.md)
|
||||
- [Utilities for performing RLHF with vLLM](../examples/offline_inference/rlhf_utils.md)
|
||||
For pipelining generation and training to improve GPU utilization and throughput, see the [Async Reinforcement Learning](async_rl.md) guide, which covers the pause/resume API for safely updating weights mid-flight.
|
||||
|
||||
See the following notebooks showing how to use vLLM for GRPO:
|
||||
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
# Weight Transfer
|
||||
|
||||
vLLM provides a pluggable weight transfer system for synchronizing model weights from a training process to the inference engine during reinforcement learning (RL) workflows. This is essential for RLHF, GRPO, and other online RL methods where the policy model is iteratively updated during training and the updated weights must be reflected in the inference engine for rollout generation.
|
||||
|
||||
## Architecture
|
||||
|
||||
The weight transfer system follows a **two-phase protocol** with a pluggable backend design:
|
||||
|
||||
1. **Initialization** (`init_weight_transfer_engine`): Establishes the communication channel between the trainer and inference workers. Called once before the training loop begins.
|
||||
2. **Weight Update** (`update_weights`): Transfers updated weights from the trainer to the inference engine. Called after each training step (or batch of steps).
|
||||
|
||||
## Available Backends
|
||||
|
||||
| Backend | Transport | Use Case |
|
||||
| ------- | --------- | -------- |
|
||||
| [NCCL](nccl.md) | NCCL broadcast | Separate GPUs for training and inference |
|
||||
| [IPC](ipc.md) | CUDA IPC handles | Colocated training and inference on same GPU |
|
||||
|
||||
## Configuration
|
||||
|
||||
Specify the weight transfer backend through `WeightTransferConfig`. The backend determines which engine handles the weight synchronization.
|
||||
|
||||
### Programmatic (Offline Inference)
|
||||
|
||||
```python
|
||||
from vllm import LLM
|
||||
from vllm.config import WeightTransferConfig
|
||||
|
||||
llm = LLM(
|
||||
model="my-model",
|
||||
weight_transfer_config=WeightTransferConfig(backend="nccl"), # or "ipc"
|
||||
)
|
||||
```
|
||||
|
||||
### CLI (Online Serving)
|
||||
|
||||
```bash
|
||||
vllm serve my-model \
|
||||
--weight-transfer-config '{"backend": "nccl"}'
|
||||
```
|
||||
|
||||
The `backend` field accepts `"nccl"` (default) or `"ipc"`.
|
||||
|
||||
## API Endpoints
|
||||
|
||||
When running vLLM as an HTTP server, the following endpoints are available for weight transfer:
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| -------- | ------ | ----------- |
|
||||
| `/init_weight_transfer_engine` | POST | Initialize the weight transfer engine with backend-specific info |
|
||||
| `/update_weights` | POST | Trigger a weight update with backend-specific metadata |
|
||||
| `/pause` | POST | Pause generation before weight sync to handle inflight requests |
|
||||
| `/resume` | POST | Resume generation after weight sync |
|
||||
| `/get_world_size` | GET | Get the number of inference workers (useful for NCCL world size calculation) |
|
||||
|
||||
!!! note
|
||||
The HTTP weight transfer endpoints require `VLLM_SERVER_DEV_MODE=1` to be set.
|
||||
|
||||
## Trainer-Side API
|
||||
|
||||
Both backends provide static methods that the trainer calls to send weights. The general pattern is:
|
||||
|
||||
```python
|
||||
# 1. Initialize the transfer engine (backend-specific)
|
||||
EngineClass.trainer_init(init_info)
|
||||
|
||||
# 2. Send weights to inference workers
|
||||
EngineClass.trainer_send_weights(
|
||||
iterator=model.named_parameters(),
|
||||
trainer_args=backend_specific_args,
|
||||
)
|
||||
```
|
||||
|
||||
See the [NCCL](nccl.md) and [IPC](ipc.md) pages for backend-specific trainer APIs and full examples.
|
||||
|
||||
## Extending the System
|
||||
|
||||
The weight transfer system is designed to be extensible. You can implement custom backends by subclassing `WeightTransferEngine` and registering them with the factory. See the [Base Class](base.md) page for details.
|
||||
@@ -0,0 +1,162 @@
|
||||
# Base Class and Custom Engines
|
||||
|
||||
The weight transfer system is built on an abstract base class that defines the contract between vLLM's worker infrastructure and the transport backend. You can implement custom backends by subclassing `WeightTransferEngine` and registering them with the `WeightTransferEngineFactory`.
|
||||
|
||||
## WeightTransferEngine
|
||||
|
||||
The `WeightTransferEngine` is a generic abstract class parameterized by two dataclass types:
|
||||
|
||||
- **`TInitInfo`** (extends `WeightTransferInitInfo`): Backend-specific initialization parameters.
|
||||
- **`TUpdateInfo`** (extends `WeightTransferUpdateInfo`): Backend-specific weight update metadata.
|
||||
|
||||
### Abstract Methods
|
||||
|
||||
Subclasses must implement these four methods:
|
||||
|
||||
| Method | Side | Description |
|
||||
| ------ | ---- | ----------- |
|
||||
| `init_transfer_engine(init_info)` | Inference | Initialize the communication channel on each inference worker |
|
||||
| `receive_weights(update_info, load_weights)` | Inference | Receive weights and call `load_weights` incrementally |
|
||||
| `shutdown()` | Inference | Clean up resources |
|
||||
| `trainer_send_weights(iterator, trainer_args)` | Trainer | Static method to send weights from the trainer process |
|
||||
|
||||
### Request Classes
|
||||
|
||||
The API-level request classes provide backend-agnostic serialization using plain dictionaries. The engine's `parse_init_info` and `parse_update_info` methods convert these dictionaries into typed dataclasses.
|
||||
|
||||
```python
|
||||
from vllm.distributed.weight_transfer.base import (
|
||||
WeightTransferInitRequest,
|
||||
WeightTransferUpdateRequest,
|
||||
)
|
||||
|
||||
# Init request (dict is converted to backend-specific TInitInfo)
|
||||
init_request = WeightTransferInitRequest(
|
||||
init_info={"master_address": "10.0.0.1", "master_port": 29500, ...}
|
||||
)
|
||||
|
||||
# Update request (dict is converted to backend-specific TUpdateInfo)
|
||||
update_request = WeightTransferUpdateRequest(
|
||||
update_info={"names": [...], "dtype_names": [...], "shapes": [...]}
|
||||
)
|
||||
```
|
||||
|
||||
### WeightTransferUpdateInfo
|
||||
|
||||
The base `WeightTransferUpdateInfo` includes an `is_checkpoint_format` flag:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class WeightTransferUpdateInfo(ABC):
|
||||
is_checkpoint_format: bool = True
|
||||
```
|
||||
|
||||
When `is_checkpoint_format=True` (the default), vLLM applies layerwise weight processing (repacking, renaming, etc.) on the received weights before loading them. Set to `False` if the trainer has already converted weights to the kernel format expected by the model.
|
||||
|
||||
## Implementing a Custom Engine
|
||||
|
||||
To create a custom weight transfer backend:
|
||||
|
||||
### 1. Define Info Dataclasses
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
from vllm.distributed.weight_transfer.base import (
|
||||
WeightTransferEngine,
|
||||
WeightTransferInitInfo,
|
||||
WeightTransferUpdateInfo,
|
||||
)
|
||||
|
||||
@dataclass
|
||||
class MyInitInfo(WeightTransferInitInfo):
|
||||
endpoint: str
|
||||
token: str
|
||||
|
||||
@dataclass
|
||||
class MyUpdateInfo(WeightTransferUpdateInfo):
|
||||
names: list[str]
|
||||
dtype_names: list[str]
|
||||
shapes: list[list[int]]
|
||||
# Add custom fields as needed
|
||||
```
|
||||
|
||||
### 2. Implement the Engine
|
||||
|
||||
```python
|
||||
from collections.abc import Callable, Iterator
|
||||
from typing import Any
|
||||
import torch
|
||||
|
||||
class MyWeightTransferEngine(WeightTransferEngine[MyInitInfo, MyUpdateInfo]):
|
||||
init_info_cls = MyInitInfo
|
||||
update_info_cls = MyUpdateInfo
|
||||
|
||||
def init_transfer_engine(self, init_info: MyInitInfo) -> None:
|
||||
# Set up connection to trainer using init_info.endpoint, etc.
|
||||
...
|
||||
|
||||
def receive_weights(
|
||||
self,
|
||||
update_info: MyUpdateInfo,
|
||||
load_weights: Callable[[list[tuple[str, torch.Tensor]]], None],
|
||||
) -> None:
|
||||
# Receive each weight and call load_weights incrementally
|
||||
for name, dtype_name, shape in zip(
|
||||
update_info.names, update_info.dtype_names, update_info.shapes
|
||||
):
|
||||
dtype = getattr(torch, dtype_name)
|
||||
weight = self._fetch_weight(name, shape, dtype)
|
||||
load_weights([(name, weight)])
|
||||
|
||||
def shutdown(self) -> None:
|
||||
# Clean up resources
|
||||
...
|
||||
|
||||
@staticmethod
|
||||
def trainer_send_weights(
|
||||
iterator: Iterator[tuple[str, torch.Tensor]],
|
||||
trainer_args: dict[str, Any],
|
||||
) -> None:
|
||||
# Send weights from the trainer process
|
||||
for name, tensor in iterator:
|
||||
# Send tensor via custom transport
|
||||
...
|
||||
```
|
||||
|
||||
!!! important
|
||||
The `load_weights` callable passed to `receive_weights` should be called **incrementally** (one or a few weights at a time) rather than accumulating all weights first. This avoids GPU out-of-memory errors with large models.
|
||||
|
||||
### 3. Register with the Factory
|
||||
|
||||
```python
|
||||
from vllm.distributed.weight_transfer.factory import WeightTransferEngineFactory
|
||||
|
||||
# Option 1: Lazy loading (recommended for built-in engines)
|
||||
WeightTransferEngineFactory.register_engine(
|
||||
"my_backend",
|
||||
"my_package.my_module",
|
||||
"MyWeightTransferEngine",
|
||||
)
|
||||
|
||||
# Option 2: Direct class registration
|
||||
WeightTransferEngineFactory.register_engine(
|
||||
"my_backend",
|
||||
MyWeightTransferEngine,
|
||||
)
|
||||
```
|
||||
|
||||
Once registered, users can select your backend via `WeightTransferConfig(backend="my_backend")`.
|
||||
|
||||
## WeightTransferEngineFactory
|
||||
|
||||
The factory uses a registry pattern with lazy loading. Built-in engines (`nccl` and `ipc`) are registered at import time but their modules are only loaded when the backend is actually requested. This avoids importing heavy dependencies (like NCCL communicators) when they aren't needed.
|
||||
|
||||
```python
|
||||
from vllm.distributed.weight_transfer.factory import WeightTransferEngineFactory
|
||||
|
||||
# Create an engine from config
|
||||
engine = WeightTransferEngineFactory.create_engine(
|
||||
config=weight_transfer_config,
|
||||
parallel_config=parallel_config,
|
||||
)
|
||||
```
|
||||
@@ -0,0 +1,73 @@
|
||||
# IPC Engine
|
||||
|
||||
The IPC weight transfer engine uses **CUDA IPC** (Inter-Process Communication) handles to share GPU memory directly between the trainer and inference workers on the **same node and same GPU**. This avoids any data copying, making it a efficient option when colocating training and inference.
|
||||
|
||||
## When to Use IPC
|
||||
|
||||
- Training and inference on the **same GPU** (colocated)
|
||||
- You want to minimize memory overhead by sharing tensors in-place
|
||||
|
||||
## How It Works
|
||||
|
||||
1. The trainer creates CUDA tensors for each weight and generates IPC handles using `torch.multiprocessing.reductions.reduce_tensor`.
|
||||
2. IPC handles are sent to the inference engine via **Ray.remote()** or **HTTP POST**.
|
||||
3. The inference worker reconstructs the tensors from the handles, reading directly from the trainer's GPU memory.
|
||||
|
||||
!!! warning
|
||||
IPC handles involve sending serialized Python objects. When using HTTP transport, you must set `VLLM_ALLOW_INSECURE_SERIALIZATION=1` on both the server and client. This is because IPC handles are pickled and base64-encoded for HTTP transmission.
|
||||
|
||||
## Initialization
|
||||
|
||||
The IPC backend requires no initialization on either side. The `init_transfer_engine` call is a no-op for IPC.
|
||||
|
||||
## Sending Weights
|
||||
|
||||
IPC supports two transport modes for delivering the handles:
|
||||
|
||||
### Ray Mode
|
||||
|
||||
Used when vLLM is running as a Ray actor:
|
||||
|
||||
```python
|
||||
from vllm.distributed.weight_transfer.ipc_engine import (
|
||||
IPCTrainerSendWeightsArgs,
|
||||
IPCWeightTransferEngine,
|
||||
)
|
||||
|
||||
trainer_args = IPCTrainerSendWeightsArgs(
|
||||
mode="ray",
|
||||
llm_handle=llm_actor_handle,
|
||||
)
|
||||
|
||||
IPCWeightTransferEngine.trainer_send_weights(
|
||||
iterator=model.named_parameters(),
|
||||
trainer_args=trainer_args,
|
||||
)
|
||||
```
|
||||
|
||||
In Ray mode, the engine calls `llm_handle.update_weights.remote(...)` directly, passing the IPC handles via Ray's serialization.
|
||||
|
||||
### HTTP Mode
|
||||
|
||||
Used when vLLM is running as an HTTP server:
|
||||
|
||||
```python
|
||||
trainer_args = IPCTrainerSendWeightsArgs(
|
||||
mode="http",
|
||||
url="http://localhost:8000",
|
||||
)
|
||||
|
||||
IPCWeightTransferEngine.trainer_send_weights(
|
||||
iterator=model.named_parameters(),
|
||||
trainer_args=trainer_args,
|
||||
)
|
||||
```
|
||||
|
||||
In HTTP mode, IPC handles are pickled, base64-encoded, and sent as JSON to the `/update_weights` endpoint.
|
||||
|
||||
See [`IPCTrainerSendWeightsArgs`](https://github.com/vllm-project/vllm/blob/main/vllm/distributed/weight_transfer/ipc_engine.py) for the full list of configurable fields.
|
||||
|
||||
## Examples
|
||||
|
||||
- [RLHF with IPC weight syncing (offline, Ray)](../../examples/rl/rlhf_ipc.md) - Colocated training and inference on a single GPU using Ray placement groups and CUDA IPC handles
|
||||
- [RLHF with IPC weight syncing (online serving, HTTP)](../../examples/rl/rlhf_http_ipc.md) - Weight transfer with a vLLM HTTP server where both server and trainer share the same GPU
|
||||
@@ -0,0 +1,110 @@
|
||||
# NCCL Engine
|
||||
|
||||
The NCCL weight transfer engine uses [NCCL](https://developer.nvidia.com/nccl) broadcast operations to transfer weights from the trainer to inference workers. It supports **multi-node** and **multi-GPU** setups where the trainer and inference engine run on separate GPUs.
|
||||
|
||||
## When to Use NCCL
|
||||
|
||||
- Training and inference on **separate GPUs** (possibly across nodes)
|
||||
- **Tensor-parallel** inference with multiple workers that all need the updated weights
|
||||
- You need high-bandwidth, low-latency weight transfer over NVLink or InfiniBand
|
||||
|
||||
## How It Works
|
||||
|
||||
1. The trainer and all inference workers join a shared NCCL process group using `StatelessProcessGroup` (vLLM's torch.distributed-independent group abstraction).
|
||||
2. The trainer broadcasts weights to all workers simultaneously. Each worker receives and loads weights incrementally.
|
||||
3. Optionally, **packed tensor broadcasting** batches multiple small tensors into larger buffers with double/triple buffering and CUDA stream overlap for higher throughput. This implementation is based on [NeMo-RL's packed tensor](https://github.com/NVIDIA-NeMo/RL/blob/main/nemo_rl/utils/packed_tensor.py).
|
||||
|
||||
## Initialization
|
||||
|
||||
NCCL requires explicit process group setup. The trainer and inference workers must agree on a master address, port, and world size.
|
||||
|
||||
### Inference Side
|
||||
|
||||
```python
|
||||
from vllm.distributed.weight_transfer.base import WeightTransferInitRequest
|
||||
|
||||
# rank_offset accounts for the trainer occupying rank 0
|
||||
llm.init_weight_transfer_engine(
|
||||
WeightTransferInitRequest(
|
||||
init_info=dict(
|
||||
master_address=master_address,
|
||||
master_port=master_port,
|
||||
rank_offset=1,
|
||||
world_size=world_size, # trainer + all inference workers
|
||||
)
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
### Trainer Side
|
||||
|
||||
```python
|
||||
from vllm.distributed.weight_transfer.nccl_engine import (
|
||||
NCCLWeightTransferEngine,
|
||||
)
|
||||
|
||||
group = NCCLWeightTransferEngine.trainer_init(
|
||||
dict(
|
||||
master_address=master_address,
|
||||
master_port=master_port,
|
||||
world_size=world_size,
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
!!! note
|
||||
`trainer_init` always assigns the trainer to rank 0. Inference workers start at `rank_offset` (typically 1).
|
||||
|
||||
## Sending Weights
|
||||
|
||||
```python
|
||||
from vllm.distributed.weight_transfer.nccl_engine import (
|
||||
NCCLTrainerSendWeightsArgs,
|
||||
NCCLWeightTransferEngine,
|
||||
)
|
||||
|
||||
trainer_args = NCCLTrainerSendWeightsArgs(
|
||||
group=group,
|
||||
packed=True, # use packed broadcasting for efficiency
|
||||
)
|
||||
|
||||
NCCLWeightTransferEngine.trainer_send_weights(
|
||||
iterator=model.named_parameters(),
|
||||
trainer_args=trainer_args,
|
||||
)
|
||||
```
|
||||
|
||||
See [`NCCLTrainerSendWeightsArgs`](https://github.com/vllm-project/vllm/blob/main/vllm/distributed/weight_transfer/nccl_engine.py) for the full list of configurable fields.
|
||||
|
||||
### Packed Tensor Broadcasting
|
||||
|
||||
When `packed=True`, multiple weight tensors are packed into large contiguous buffers before broadcasting. This reduces the number of NCCL operations and uses double/triple buffering with dedicated CUDA streams for overlap between packing, broadcasting, and unpacking.
|
||||
|
||||
Both the trainer (`NCCLTrainerSendWeightsArgs`) and inference side (`NCCLWeightTransferUpdateInfo`) must use matching `packed_buffer_size_bytes` and `packed_num_buffers` values.
|
||||
|
||||
## Receiving Weights (Inference Side)
|
||||
|
||||
The inference side triggers weight reception by calling `update_weights`:
|
||||
|
||||
```python
|
||||
from vllm.distributed.weight_transfer.base import WeightTransferUpdateRequest
|
||||
|
||||
llm.update_weights(
|
||||
WeightTransferUpdateRequest(
|
||||
update_info=dict(
|
||||
names=names,
|
||||
dtype_names=dtype_names,
|
||||
shapes=shapes,
|
||||
packed=True,
|
||||
)
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
The `names`, `dtype_names`, and `shapes` lists describe each parameter. These must match the order in which the trainer iterates over its parameters.
|
||||
|
||||
## Examples
|
||||
|
||||
- [RLHF with NCCL weight syncing (offline, Ray)](../../examples/rl/rlhf_nccl.md) - Trainer on one GPU, 2x tensor-parallel vLLM engine on two others, with packed NCCL weight broadcast
|
||||
- [RLHF with async weight syncing (offline, Ray)](../../examples/rl/rlhf_async_new_apis.md) - Async generation with mid-flight pause, weight sync, resume, and validation against a fresh model
|
||||
- [RLHF with NCCL weight syncing (online serving, HTTP)](../../examples/rl/rlhf_http_nccl.md) - Weight transfer with a running vLLM HTTP server using HTTP control plane and NCCL data plane
|
||||
Reference in New Issue
Block a user