[feat] Add explicit /start_weight_update and /finish_weight_update APIs for weight transfer (#39212)
This commit is contained in:
@@ -4,10 +4,12 @@ vLLM provides a pluggable weight transfer system for synchronizing model weights
|
||||
|
||||
## Architecture
|
||||
|
||||
The weight transfer system follows a **two-phase protocol** with a pluggable backend design:
|
||||
The weight transfer system follows a **four-phase protocol** with a pluggable backend design:
|
||||
|
||||
1. **Initialization** (`init_weight_transfer_engine`): Establishes the communication channel between the trainer and inference workers. Called once before the training loop begins.
|
||||
2. **Weight Update** (`update_weights`): Transfers updated weights from the trainer to the inference engine. Called after each training step (or batch of steps).
|
||||
2. **Start** (`start_weight_update`): Prepares the inference engine for a weight update.
|
||||
3. **Weight Update** (`update_weights`): Transfers updated weights from the trainer to the inference engine. May be called one or more times (e.g., for chunked transfers).
|
||||
4. **Finish** (`finish_weight_update`): Finalizes the weight update (e.g., runs post-processing for checkpoint-format weights). Called once after all weights have been transferred.
|
||||
|
||||
## Available Backends
|
||||
|
||||
@@ -48,7 +50,9 @@ When running vLLM as an HTTP server, the following endpoints are available for w
|
||||
| Endpoint | Method | Description |
|
||||
| -------- | ------ | ----------- |
|
||||
| `/init_weight_transfer_engine` | POST | Initialize the weight transfer engine with backend-specific info |
|
||||
| `/update_weights` | POST | Trigger a weight update with backend-specific metadata |
|
||||
| `/start_weight_update` | POST | Start a weight update |
|
||||
| `/update_weights` | POST | Transfer a batch of weights with backend-specific metadata |
|
||||
| `/finish_weight_update` | POST | Finish the weight update and run post-processing |
|
||||
| `/pause` | POST | Pause generation before weight sync to handle inflight requests |
|
||||
| `/resume` | POST | Resume generation after weight sync |
|
||||
| `/get_world_size` | GET | Get the number of inference workers (useful for NCCL world size calculation) |
|
||||
@@ -64,11 +68,17 @@ Both backends provide static methods that the trainer calls to send weights. The
|
||||
# 1. Initialize the transfer engine (backend-specific)
|
||||
EngineClass.trainer_init(init_info)
|
||||
|
||||
# 2. Send weights to inference workers
|
||||
# 2. Start weight update on inference side
|
||||
llm.start_weight_update(is_checkpoint_format=True)
|
||||
|
||||
# 3. Send weights to inference workers
|
||||
EngineClass.trainer_send_weights(
|
||||
iterator=model.named_parameters(),
|
||||
trainer_args=backend_specific_args,
|
||||
)
|
||||
|
||||
# 4. Finish weight update on inference side
|
||||
llm.finish_weight_update()
|
||||
```
|
||||
|
||||
See the [NCCL](nccl.md) and [IPC](ipc.md) pages for backend-specific trainer APIs and full examples.
|
||||
|
||||
@@ -43,16 +43,14 @@ update_request = WeightTransferUpdateRequest(
|
||||
|
||||
### WeightTransferUpdateInfo
|
||||
|
||||
The base `WeightTransferUpdateInfo` includes an `is_checkpoint_format` flag:
|
||||
The base `WeightTransferUpdateInfo` is a marker class for backend-specific update info:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class WeightTransferUpdateInfo(ABC):
|
||||
is_checkpoint_format: bool = True
|
||||
pass
|
||||
```
|
||||
|
||||
When `is_checkpoint_format=True` (the default), vLLM applies layerwise weight processing (repacking, renaming, etc.) on the received weights before loading them. Set to `False` if the trainer has already converted weights to the kernel format expected by the model.
|
||||
|
||||
## Implementing a Custom Engine
|
||||
|
||||
To create a custom weight transfer backend:
|
||||
|
||||
@@ -38,11 +38,15 @@ trainer_args = IPCTrainerSendWeightsArgs(
|
||||
mode="ray",
|
||||
llm_handle=llm_actor_handle,
|
||||
)
|
||||
|
||||
# start
|
||||
ray.get(llm_actor_handle.start_weight_update.remote(is_checkpoint_format=True))
|
||||
# send weights
|
||||
IPCWeightTransferEngine.trainer_send_weights(
|
||||
iterator=model.named_parameters(),
|
||||
trainer_args=trainer_args,
|
||||
)
|
||||
# finish
|
||||
ray.get(llm_actor_handle.finish_weight_update.remote())
|
||||
```
|
||||
|
||||
In Ray mode, the engine calls `llm_handle.update_weights.remote(...)` directly, passing the IPC handles via Ray's serialization.
|
||||
@@ -57,13 +61,23 @@ trainer_args = IPCTrainerSendWeightsArgs(
|
||||
url="http://localhost:8000",
|
||||
)
|
||||
|
||||
# start
|
||||
base_url = "http://localhost:8000"
|
||||
url = f"{base_url}/start_weight_update"
|
||||
response = requests.post(url, json={"is_checkpoint_format": True}, timeout=60)
|
||||
response.raise_for_status()
|
||||
# send weights
|
||||
IPCWeightTransferEngine.trainer_send_weights(
|
||||
iterator=model.named_parameters(),
|
||||
trainer_args=trainer_args,
|
||||
)
|
||||
# finish
|
||||
url = f"{base_url}/finish_weight_update"
|
||||
response = requests.post(url, json={}, timeout=60)
|
||||
response.raise_for_status()
|
||||
```
|
||||
|
||||
In HTTP mode, IPC handles are pickled, base64-encoded, and sent as JSON to the `/update_weights` endpoint.
|
||||
In HTTP mode, IPC handles are pickled, base64-encoded, and sent as JSON to the `/update_weights` endpoint. As with Ray mode, you must call `start_weight_update` before and `finish_weight_update` after.
|
||||
|
||||
See [`IPCTrainerSendWeightsArgs`](https://github.com/vllm-project/vllm/blob/main/vllm/distributed/weight_transfer/ipc_engine.py) for the full list of configurable fields.
|
||||
|
||||
|
||||
@@ -84,11 +84,15 @@ Both the trainer (`NCCLTrainerSendWeightsArgs`) and inference side (`NCCLWeightT
|
||||
|
||||
## Receiving Weights (Inference Side)
|
||||
|
||||
The inference side triggers weight reception by calling `update_weights`:
|
||||
The inference side triggers weight reception using the four-phase protocol — `init_weight_transfer_engine`, `start_weight_update`, `update_weights`, `finish_weight_update`. The init phase is shown [above](#initialization). The remaining three steps are:
|
||||
|
||||
```python
|
||||
from vllm.distributed.weight_transfer.base import WeightTransferUpdateRequest
|
||||
|
||||
# 1. Start the weight update
|
||||
llm.start_weight_update(is_checkpoint_format=True)
|
||||
|
||||
# 2. Receive weights (can be called multiple times for chunked transfers)
|
||||
llm.update_weights(
|
||||
WeightTransferUpdateRequest(
|
||||
update_info=dict(
|
||||
@@ -99,10 +103,15 @@ llm.update_weights(
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
# 3. Finish the weight update
|
||||
llm.finish_weight_update()
|
||||
```
|
||||
|
||||
The `names`, `dtype_names`, and `shapes` lists describe each parameter. These must match the order in which the trainer iterates over its parameters.
|
||||
|
||||
`start_weight_update` must be called before `update_weights`, and `finish_weight_update` must be called after all weight chunks have been transferred. The `is_checkpoint_format` flag controls whether layerwise reload processing is applied (`True` for checkpoint-format weights, `False` for pre-processed kernel-format weights).
|
||||
|
||||
## Examples
|
||||
|
||||
- [RLHF with NCCL weight syncing (offline, Ray)](../../examples/rl/rlhf_nccl.md) - Trainer on one GPU, 2x tensor-parallel vLLM engine on two others, with packed NCCL weight broadcast
|
||||
|
||||
Reference in New Issue
Block a user