[Doc][RL] Document the weight-transfer HTTP protocol for trainers without vLLM (#62267)
Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
1b2a666391
commit
2365177ddd
@@ -145,7 +145,7 @@ engine.send_weights() # called on every rank
|
||||
|
||||
## API Endpoints
|
||||
|
||||
When running vLLM as an HTTP server, the following endpoints are available for weight transfer. `HTTPVLLMWeightSyncClient` speaks the first four for you.
|
||||
When running vLLM as an HTTP server, the following endpoints are available for weight transfer. `HTTPVLLMWeightSyncClient` speaks the first five for you. A trainer may also implement the protocol directly without importing vLLM: see the wire contracts in [NCCL](nccl.md#trainer-without-vllm-over-http) and [IPC](ipc.md#trainer-without-vllm-over-http). This keeps the trainer decoupled from vLLM: it needs no vLLM install or version pin, only this documented HTTP and wire contract, so trainer and server can upgrade independently (non-PyTorch trainers such as JAX can use it too).
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| -------- | ------ | ----------- |
|
||||
@@ -153,6 +153,7 @@ When running vLLM as an HTTP server, the following endpoints are available for w
|
||||
| `/start_weight_update` | POST | Start a weight update |
|
||||
| `/update_weights` | POST | Transfer a batch of weights with backend-specific metadata |
|
||||
| `/finish_weight_update` | POST | Finish the update and optionally commit its `weight_version` |
|
||||
| `/shutdown_weight_transfer_engine` | POST | Release the transfer channel without stopping inference |
|
||||
| `/update_weight_version` | POST | Update `weight_version` without changing model weights |
|
||||
| `/weight_info` | GET | Get the latest committed weight version |
|
||||
| `/pause` | POST | Pause generation before weight sync to handle inflight requests |
|
||||
@@ -162,6 +163,12 @@ When running vLLM as an HTTP server, the following endpoints are available for w
|
||||
!!! note
|
||||
The HTTP weight transfer endpoints require `VLLM_SERVER_DEV_MODE=1` to be set.
|
||||
|
||||
!!! warning
|
||||
If `/update_weights` fails, the model is left mid-reload while `/health`
|
||||
stays 200, and `/finish_weight_update` is rejected. Before generating again,
|
||||
resend a full `/start_weight_update` → `/update_weights` →
|
||||
`/finish_weight_update` (see [#57854](https://github.com/vllm-project/vllm/pull/57854)).
|
||||
|
||||
The Rust frontend's optional gRPC `Control` service exposes the same pause, sleep, weight-transfer, and weight-version lifecycle for trusted sidecars. The `ServerInfo.rl_capabilities` response reports whether weight transfer and sleep mode were configured. Backend-specific `init_info` and `update_info` remain JSON metadata; model tensors continue to move over the configured NCCL, IPC, sparse-NCCL, or sharded-RDT transport.
|
||||
|
||||
## Extending the System
|
||||
|
||||
Reference in New Issue
Block a user