[feat] Add explicit /start_weight_update and /finish_weight_update APIs for weight transfer (#39212)

This commit is contained in:
Sumanth R Hegde
2026-05-08 18:03:33 -07:00
committed by GitHub
parent 30f519e947
commit e3b65a5ba0
19 changed files with 287 additions and 53 deletions
+14 -4
View File
@@ -4,10 +4,12 @@ vLLM provides a pluggable weight transfer system for synchronizing model weights
## Architecture
The weight transfer system follows a **two-phase protocol** with a pluggable backend design:
The weight transfer system follows a **four-phase protocol** with a pluggable backend design:
1. **Initialization** (`init_weight_transfer_engine`): Establishes the communication channel between the trainer and inference workers. Called once before the training loop begins.
2. **Weight Update** (`update_weights`): Transfers updated weights from the trainer to the inference engine. Called after each training step (or batch of steps).
2. **Start** (`start_weight_update`): Prepares the inference engine for a weight update.
3. **Weight Update** (`update_weights`): Transfers updated weights from the trainer to the inference engine. May be called one or more times (e.g., for chunked transfers).
4. **Finish** (`finish_weight_update`): Finalizes the weight update (e.g., runs post-processing for checkpoint-format weights). Called once after all weights have been transferred.
## Available Backends
@@ -48,7 +50,9 @@ When running vLLM as an HTTP server, the following endpoints are available for w
| Endpoint | Method | Description |
| -------- | ------ | ----------- |
| `/init_weight_transfer_engine` | POST | Initialize the weight transfer engine with backend-specific info |
| `/update_weights` | POST | Trigger a weight update with backend-specific metadata |
| `/start_weight_update` | POST | Start a weight update |
| `/update_weights` | POST | Transfer a batch of weights with backend-specific metadata |
| `/finish_weight_update` | POST | Finish the weight update and run post-processing |
| `/pause` | POST | Pause generation before weight sync to handle inflight requests |
| `/resume` | POST | Resume generation after weight sync |
| `/get_world_size` | GET | Get the number of inference workers (useful for NCCL world size calculation) |
@@ -64,11 +68,17 @@ Both backends provide static methods that the trainer calls to send weights. The
# 1. Initialize the transfer engine (backend-specific)
EngineClass.trainer_init(init_info)
# 2. Send weights to inference workers
# 2. Start weight update on inference side
llm.start_weight_update(is_checkpoint_format=True)
# 3. Send weights to inference workers
EngineClass.trainer_send_weights(
iterator=model.named_parameters(),
trainer_args=backend_specific_args,
)
# 4. Finish weight update on inference side
llm.finish_weight_update()
```
See the [NCCL](nccl.md) and [IPC](ipc.md) pages for backend-specific trainer APIs and full examples.