[Doc][RL] Document the weight-transfer HTTP protocol for trainers without vLLM (#62267)

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
aoshen02
2026-10-18 09:26:56 +00:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 1b2a666391
commit 2365177ddd
6 changed files with 435 additions and 1 deletions
+46
View File
@@ -130,6 +130,51 @@ pass — this is the invariant to test first.
IPC does not read `metadata()` at all: it derives the update info from
iteration as it goes, so it cannot observe a divergence.
## Trainer Without vLLM over HTTP
A trainer can speak the [HTTP endpoints](README.md#api-endpoints) directly, with
only PyTorch and an HTTP client. It mints the `ncclUniqueId` itself and is NCCL
rank 0. Reference: [`rlhf_http_nccl_no_vllm.py`](../../../examples/rl/rlhf_http_nccl_no_vllm.py).
`POST /init_weight_transfer_engine` body `{"init_info": {...}}`:
| Field | Default | Meaning |
| ----- | ------- | ------- |
| `nccl_unique_id_b64` | — | Standard (not URL-safe) base64 of the 128-byte `ncclUniqueId`. Replaces `master_address` / `master_port` |
| `rank_offset` | — | First worker rank; `1` for one trainer rank |
| `world_size` | — | Trainer ranks + `GET /get_world_size` |
| `packed` | `false` | Packed broadcasting |
| `packed_buffer_size_bytes` | 1 GiB | Must match the trainer's chunking |
| `packed_num_buffers` | 2 | Worker-side receive buffers |
- Worker rank = `rank_offset + dp_index * (TP * PP) + rank`.
- `/init_weight_transfer_engine` returns only after all ranks join: send it in
the background while the trainer calls `ncclCommInitRank(world_size, uid, 0)`.
There is no timeout; if the trainer never joins, the workers wait forever.
- Right after init every worker issues a warm-up `all_reduce` (sum) of
`torch.zeros(1)` (one float32 element). The trainer must issue the same one
before any other collective.
Each sync: `POST /start_weight_update`, then `POST /update_weights` with
`{"update_info": {"names", "dtype_names", "shapes"}}` **in the background**
while rank 0 broadcasts in `names` order, then `POST /finish_weight_update`.
`dtype_names` are `torch` attribute names (`"bfloat16"`); `shapes` are full
checkpoint shapes.
- Unpacked: one broadcast per tensor.
- Packed: concatenate `t.contiguous().view(-1).view(torch.uint8)` in `names`
order; broadcast a chunk as soon as its size **exceeds**
`packed_buffer_size_bytes` (the crossing tensor stays in the chunk), then the
remainder.
Trainer-side communicators without vLLM:
| Communicator | Status |
| ------------ | ------ |
| `torch.distributed.ProcessGroupNCCL` on a private `TCPStore` + `eager_connect_single_device` | Works; the unique id is read from store key `"0"`, a torch implementation detail (used by the example) |
| `ctypes` against torch's bundled `libnccl.so.2` | Works |
| `torch.cuda.nccl.init_rank` | Fails on Python 3.10+ (`PY_SSIZE_T_CLEAN` `SystemError`) |
## Sparse NCCL
Sparse, flat-index weight patches use
@@ -196,6 +241,7 @@ Sparse NCCL sends only `O(nnz)` indices and values. Checkpoint application still
## Examples
- [RLHF with NCCL weight syncing (`vllm serve`, HTTP)](../../../examples/rl/rlhf_http_nccl.py) - **Start here.** Trainer on one GPU, 2x tensor-parallel fp8 server on two others; HTTP control plane, NCCL data plane. Launches and tears down its own server
- [NCCL weight syncing from a trainer that does not import vLLM](../../../examples/rl/rlhf_http_nccl_no_vllm.py) - `torch` + `requests` only, `nccl_unique_id_b64` rendezvous; see [Trainer Without vLLM over HTTP](#trainer-without-vllm-over-http)
- [RLHF with NCCL + FSDP2 and expert parallelism](../../../examples/rl/rlhf_nccl_fsdp_ep.py) - Multi-rank trainer: every FSDP rank builds an engine and joins the `full_tensor()` gather while only rank 0 touches the wire
- [RLHF with sparse NCCL weight syncing (offline, Ray)](../../../examples/rl/rlhf_sparse_nccl.py) - Qwen3 MoE per-expert checkpoint updates with one trainer GPU and two TP2/EP2 inference GPUs
- [RLHF with async weight syncing (offline, Ray)](../../../examples/rl/rlhf_async_new_apis.py) - Async generation with mid-flight pause, weight sync, resume, and validation against a fresh model; uses an in-process `AsyncLLMEngine` with `RayVLLMWeightSyncClient`