[RL] P2P RDT weight sync (#43375)

This commit is contained in:
Aaron Hao
2026-08-23 09:42:22 -07:00
committed by GitHub
parent 39eba6ac36
commit 6cd9713463
24 changed files with 8409 additions and 29 deletions
+3 -2
View File
@@ -34,6 +34,7 @@ engine drives on your behalf:
| [NCCL](nccl.md) | NCCL broadcast | Separate GPUs for training and inference |
| [IPC](ipc.md) | CUDA IPC handles | Colocated training and inference on same GPU |
| [sparse_nccl](nccl.md#sparse-nccl) | NCCL broadcast | Sparse flat-index weight patches (TP=1/PP=1) |
| [sharded_rdt](sharded_rdt.md) | NIXL / Ray Direct Transport (pull-based) | Very large models where each worker needs only its own slice (MoE with expert parallelism) |
## Quickstart
@@ -48,7 +49,7 @@ from vllm.config import WeightTransferConfig
llm = LLM(
model="my-model",
weight_transfer_config=WeightTransferConfig(backend="nccl"), # or "ipc", "sparse_nccl"
weight_transfer_config=WeightTransferConfig(backend="nccl"), # or "ipc", "sparse_nccl", "sharded_rdt"
)
```
@@ -158,7 +159,7 @@ When running vLLM as an HTTP server, the following endpoints are available for w
!!! note
The HTTP weight transfer endpoints require `VLLM_SERVER_DEV_MODE=1` to be set.
The Rust frontend's optional gRPC `Control` service exposes the same pause, sleep, weight-transfer, and weight-version lifecycle for trusted sidecars. The `ServerInfo.rl_capabilities` response reports whether weight transfer and sleep mode were configured. Backend-specific `init_info` and `update_info` remain JSON metadata; model tensors continue to move over the configured NCCL, IPC, or sparse-NCCL transport.
The Rust frontend's optional gRPC `Control` service exposes the same pause, sleep, weight-transfer, and weight-version lifecycle for trusted sidecars. The `ServerInfo.rl_capabilities` response reports whether weight transfer and sleep mode were configured. Backend-specific `init_info` and `update_info` remain JSON metadata; model tensors continue to move over the configured NCCL, IPC, sparse-NCCL, or sharded-RDT transport.
## Extending the System
+136 -15
View File
@@ -20,8 +20,14 @@ vice versa, so the registries stay independent.
### WeightSource
A `WeightSource` is a **re-iterable** source of the trainer's weights with two
channels:
**This is the adapter for whatever shape your trainer's weights are in.**
`WeightSource` defines how you extract your weights for your specific framework.
What an engine gets is always the same: HF-format parameter names, and tensors
already materialized to their full (unsharded) shape. Whatever gathering,
re-fusing, dequantizing or renaming that takes belongs inside the source.
A `WeightSource` is **re-iterable** and has two required channels:
- **`metadata() -> list[ParamMeta]`** — the name, wire dtype, and full shape of
every parameter, *without transferring anything*. Cheap when shapes are known
@@ -51,9 +57,7 @@ class ParamMeta:
Materializing is typically a collective, so **every trainer rank must iterate the
same source in the same order, in lockstep**, or ranks deadlock. `metadata()` can
itself be a collective for custom producers, so it too runs on every rank — only
the sender ships the result. Under pipeline parallelism a rank may not own a
parameter at all; iterating still drives the collective, and the yielded tensor
is only meaningful on the sender.
the sender ships the result.
`iter(source)` must yield a *fresh* pass each round.
@@ -70,35 +74,133 @@ from vllm.distributed.weight_transfer import ModuleSource
source = ModuleSource(model)
```
#### Custom Sources
#### Custom sources
Subclass `WeightSource` when the weights you want to send require additional processing to convert to a HF compatible format.
Subclass `WeightSource` when the weights need work to reach HF format — a
framework-specific export, a re-fusing step, a dtype cast.
```python
from vllm.distributed.weight_transfer import ParamMeta, WeightSource
from vllm.distributed.weight_transfer.base import materialize_full_tensor
class MyExportSource(WeightSource):
def __init__(self, model):
self._model = model
class MegatronBridgeSource(WeightSource):
"""Megatron model -> HF names, via a bridge that gathers TP/PP/EP internally
and returns full tensors on every rank."""
def __init__(self, bridge, module, dtype):
self._bridge, self._module, self._dtype = bridge, module, dtype
self._meta: list[ParamMeta] | None = None
def _export(self):
return self._bridge.export_hf_weights(self._module)
def metadata(self) -> list[ParamMeta]:
# Cache: for producers that must materialize to learn shapes, this is
# the expensive channel. Runs on every rank (it may be a collective).
if self._meta is None:
self._meta = [
ParamMeta(name, t.dtype, tuple(t.shape)) for name, t in self._export()
ParamMeta(name, self._dtype, tuple(t.shape))
for name, t in self._export()
]
return self._meta
def __iter__(self):
# Must yield exactly what metadata() declared, in the same order.
for name, tensor in self._export():
yield name, materialize_full_tensor(tensor)
yield name, tensor.to(self._dtype).detach().contiguous()
```
#### `held_names()`: partial ownership
By default every rank is assumed to be able to produce every parameter — which is
what the source above does, since its bridge gathers across all parallelism before
yielding. That is simple and always correct, but it means the gather cost is paid
in full on every rank.
Optionally, you can override `held_names()` when the ranks are split so each holds only part of the
model. It returns the
parameter names this rank holds (or `None`, the default, for all of them):
```python
def held_names(self):
# This pipeline stage's layers, and within them only this EP rank's experts.
return self._my_stage_names - self._foreign_expert_names
```
This covers various trainer layouts — pipeline stages (a rank holds some layers),
expert parallelism (a rank holds some experts), both at once, or a shape that fits
neither. Backends that can route per parameter (see
[sharded RDT](sharded_rdt.md)) then pull each name from a rank that actually
holds it.
Three requirements come with overriding it:
- **`metadata()` must still describe the whole model on every rank.** Only the
sender's metadata reaches the inference side, so a rank that reported just its
own share would leave the rest silently un-transferred. Sharded RDT cross-checks
this across ranks at init.
- **Every name must be held by at least one rank**, or it can never be served.
The engine raises at init naming the first orphan.
- **Iteration yields `None` for a name this rank does not hold.** The name still
appears, in metadata order, so the order check stays aligned across ranks — only
the data is absent. Claiming a name and then yielding `None` for it is an error
the engine reports by name.
!!! warning "Partial ownership only works with sharded RDT"
Only a backend that routes per parameter honours `held_names()`.
Broadcast backends ignore it and send every name from every rank, so
declaring partial ownership there changes nothing.
#### Gather groups
Some backends transfer a layer at a time rather than a model at a time, so they
partition `metadata()` into **gather groups**. `layerwise_groups` keys each name
on the **outermost index segment** it contains, so **a group is one decoder
layer**, with runs of un-indexed names (the embeddings, the final norm,
`lm_head`) forming groups of their own where they appear:
```text
group 0 model.embed_tokens.weight
group 1 model.layers.0.* <- one decoder layer
group 2 model.layers.1.*
...
group N+1 model.norm.weight, lm_head.weight
```
Keying on the index rather than a literal prefix means no per-architecture table:
`model.layers.0.`, `model.language_model.layers.0.` (recent Qwen text
checkpoints), `transformer.h.0.` (GPT-2, Falcon), `backbone.layers.0.` (Mamba)
and a vision tower's `visual.blocks.0.` all partition the same way. The index
taken is the *outermost* one, which keeps a MoE layer whole — a per-expert name
like `model.layers.3.mlp.experts.7.w1` keys on the layer, not the expert.
Group index *g* means the same layer on every rank and every consumer, because
every side derives it from one rank's `metadata()`. That agreement is what lets a
backend bound its buffers to one layer and free a layer once everyone is done
with it.
!!! warning "A leaf module's sources must all fall in one group"
The sharded-RDT engine frees a group as soon as its last chunk lands, so a
module split across groups would park a pull until the stall watchdog fires.
The default partition guarantees this; a `groups()` override must preserve
it.
Two hooks follow from this, both with working defaults:
- **`groups()`** — this rank's groups, in metadata order. The default is
`layerwise_groups(metadata())` restricted to the groups holding at least one
held name. A group with nothing held here is skipped entirely.
- **`iter_groups()`** — the same stream, batched one group at a time. The default
drives `__iter__` and batches its output, checking that names arrive in metadata
order. Override it when your framework can produce a whole group in one step:
materializing is usually a collective, and driving it per group rather than per
tensor turns ~37k generator resumes into ~95 on a per-expert MoE model.
Because `metadata()` order defines the partition, **all names sharing a layer
index must be contiguous in it**. A source whose natural export order interleaves
layers — bucketing all the MoE experts together, say — has to reorder before
returning.
### VLLMWeightSyncClient
**This is the adapter for however your RL stack reaches vLLM.** Many RL frameworks wrap
@@ -372,7 +474,7 @@ Subclasses must implement five methods:
The base class provides:
1. `__init__`, taking `config` (`WeightTransferConfig`), `vllm_config` (`VllmConfig`), `device` (`torch.device`), and `model` (`nn.Module`).
2. `update_weights(update_info_dict)`, a thin wrapper for `receive_weights`: it parses the dict into the typed dataclass, calls `receive_weights`, and synchronizes the device.
2. `update_weights(update_info_dict)`, a thin wrapper for `receive_weights`: it parses the dict into the typed dataclass, calls `receive_weights`, and synchronizes the device — unless the engine sets `defers_processing`, below.
3. `parse_init_info` / `parse_update_info`, which convert API-level dicts into the typed dataclasses and raise `ValueError` on a bad payload.
4. `set_weight_update_target` / `reset_weight_update_target`, used to retarget an update at the speculative draft model.
@@ -383,6 +485,25 @@ The base class provides:
update info carries only per-round metadata. This is what makes a
trainer/worker mismatch unrepresentable.
!!! note "`defers_processing`: when a returned update means queued, not applied"
An engine that pipelines its GPU post-processing onto background threads
cannot let `update_weights` synchronize the device — that would block on
those threads and serialize the pipeline. Such an engine sets the class
attribute `defers_processing = True`, omits the per-update sync, and
guarantees completion in `finish_weight_update` instead.
Callers that go through `finish_weight_update` need do nothing; the engine
drains there. A caller that drives the tail itself — running its own
`finalize_layerwise_reload`, say — must check the flag and call
`drain_pending()` first, because with it set a returned `update_weights`
means *queued*, not *applied*. `drain_pending()` is idempotent, and a no-op
on an engine that processes synchronously, so it is always safe to call.
[Sharded RDT](sharded_rdt.md) is the built-in engine that sets it: it
scatters and quantizes on background threads with their own CUDA streams, so
its `drain_pending()` joins both queues and syncs both streams before
`finalize_layerwise_reload` runs.
### Request Classes
The API-level request classes provide backend-agnostic serialization using plain dictionaries.
@@ -497,7 +618,7 @@ Once registered, users select your backend via `WeightTransferConfig(backend="my
### WeightTransferEngineFactory
The factory uses a registry pattern with lazy loading. Built-in engines (`nccl`, `ipc`, and `sparse_nccl`) are registered at import time but their modules are only loaded when the backend is actually requested. This avoids importing heavy dependencies (like NCCL communicators) when they aren't needed.
The factory uses a registry pattern with lazy loading. Built-in engines (`nccl`, `ipc`, `sparse_nccl` and `sharded_rdt`) are registered at import time but their modules are only loaded when the backend is actually requested. This avoids importing heavy dependencies (like NCCL communicators) when they aren't needed.
```python
from vllm.distributed.weight_transfer import WeightTransferEngineFactory
@@ -0,0 +1,126 @@
# Sharded RDT Engine
The sharded RDT weight transfer engine moves weights point-to-point over [NIXL](https://github.com/ai-dynamo/nixl) using Ray Direct Transport (RDT), Ray's zero-copy tensor transport between actors. It is **pull-based**: the inference workers initiate every transfer and each one asks only for the *slice* it consumes under tensor and expert parallelism. A large MoE model therefore moves roughly `total_bytes / num_workers` per worker instead of the `total_bytes` a broadcast costs.
## When to Use Sharded RDT
- Very large models where broadcasting whole parameters is the bottleneck — typically an MoE served with expert parallelism, where each worker owns a small fraction of the experts
- Trainer and inference on **separate GPUs**, over a fabric NIXL supports (InfiniBand, RoCE, EFA)
- Trainers that are themselves sharded, including **pipeline-parallel** ones where a rank holds only part of the model
Requirements:
- `distributed_executor_backend="ray"` — the workers must be Ray actors
- Ray `>= 2.56.0` on both the trainer and the workers
- `nixl` installed in the environment shared by trainer and workers
- Weight loaders that stay inside the supported op set (below)
- EPLB (`enable_eplb=true`) is rejected: it rearranges experts at runtime, which invalidates the recorded plan
## How It Works
### Slices are tracked through vLLM's own weight loaders
A weight loader normally receives a full HF-format tensor and slices out the part this worker needs. Instead of sending it one, the engine hands it a `FakeRDTTensor`: a zero-storage tensor that answers `.shape` / `.dtype` / `.size()` but holds no data. Every view or slice op the loader calls returns a new fake with that op appended to a recorded chain, and `copy_` is the sink that ends it.
That chain is the wire format. `("model.layers.0.w", (("narrow", (0, 512, 512), ()), ("t", (), ())))` tells the trainer: take this tensor, `narrow` it, transpose it, send the result. The trainer replays it with `getattr(tensor, op)(*args, **kwargs)`.
Discovery is expensive, so it happens **once**, at `init_transfer_engine`, as a dry run over `model.load_weights` with every parameter on meta. Nothing is transferred; the engine just records, per leaf module, which slice feeds which destination region. Every later sync is pure replay.
Anything a loader does that needs real data — arithmetic, `.to()`, `.float()`, `.item()`, `.data`, bool-mask indexing — falls outside the allowlist and raises at init. That is deliberate: failing loudly during setup beats silently transferring the wrong bytes. `SUPPORTED_OPS` in `sharded_rdt_common.py` is the single table both sides derive from, so the recorder and the replayer cannot drift.
### Received slices land directly in the layerwise reload buffers
The engine drives [layerwise reload](../layerwise.md) itself, in `start_weight_update` / `finish_weight_update`. Because the dry run already recorded each destination as an `as_strided` region of its parameter, an arriving slice is copied straight into the layer being reloaded — no full HF tensor is ever materialized on the worker, and no second pass over `load_weights` runs. Each layer is quantized and copied into its persistent kernel storage as soon as its last slice lands.
### Gathers and pulls are pipelined, which is what `gather_lookahead` bounds
The trainer usually cannot serve its parameters as they sit: FSDP shards them, and even an EP-split trainer has to assemble a whole expert. So each sync still runs gather collectives — but a layer at a time, not a model at a time.
**A gather group is one decoder layer.** The parameter list is keyed on the outermost index segment of each name, which leaves runs of un-indexed names — the embeddings before the first layer, the final norm and `lm_head` after the last — as groups of their own:
```text
group 0 model.embed_tokens.weight
group 1 model.layers.0.* <- one decoder layer
group 2 model.layers.1.*
...
group N+1 model.norm.weight, lm_head.weight
```
The layer is the unit of everything that follows: the trainer gathers a layer, publishes it (immediately pullable), and moves on to the next while the consumers pull the one it just published. Once every consumer has signalled that it is done with a layer, the trainer drops it and gains a credit to gather another. `gather_lookahead` is how far ahead of the consumers that loop may run, so at most `gather_lookahead + 1` layers are resident on the trainer at a time. The default of 1 keeps the next layer gathered and pullable while the current one is being pulled — enough to hide the handoff without doubling trainer memory. Raise it only if one layer's gather is slower than its pulls.
Because a layer is also the unit the consumers free and the unit the receive buffers are sized against, it is what keeps memory bounded on both sides: without it the whole model would be one transfer, and both sides would have to hold their full share of it at once. Keying on the index rather than a fixed `model.layers.` prefix is what makes that hold across naming conventions — a VLM's `model.language_model.layers.`, GPT-2's `transformer.h.`, a vision tower's `visual.blocks.`. Sources can control the partition; see [gather groups](base.md#gather-groups).
### Ownership
A trainer rank need not hold the whole model. Each one declares what it holds through [`WeightSource.held_names()`](base.md#held_names-partial-ownership), the fleet all-gathers those declarations at `trainer_init`, and the consumers route each pull to a rank that actually holds the name. Pipeline stages, expert parallelism, and combinations of the two are all the same declaration. Consumers spread their pulls across the ranks that hold a name, so no single trainer NIC becomes the bottleneck.
## Inference Side
```python
from vllm import LLM
from vllm.config import WeightTransferConfig
llm = LLM(
model="my-model",
weight_transfer_config=WeightTransferConfig(backend="sharded_rdt"),
distributed_executor_backend="ray",
)
```
```bash
vllm serve my-model \
--distributed-executor-backend ray \
--weight-transfer-config '{"backend": "sharded_rdt"}'
```
Everything else — which producers exist, how the model splits into layer groups, the ownership table — arrives from the trainer at the init handshake.
!!! warning "Size the receive buffers before choosing `gpu_memory_utilization`"
Each worker holds `num_rdt_buffers` receive buffers, each large enough for the biggest single slice batch it pulls. Like NCCL and NIXL internals, they do **not** count against `gpu_memory_utilization`, so a fraction that leaves no headroom OOMs at the first sync even though the engine came up healthy. The buffer size is driven by the largest atomic slice — for an untied vocab matrix on a worker that holds it unsliced, that is the whole embedding.
## Trainer Side
```python
from vllm.distributed.weight_transfer import (
ModuleSource,
HTTPVLLMWeightSyncClient,
WeightTransferTrainerFactory,
)
from vllm.distributed.weight_transfer.sharded_rdt_trainer import (
ShardedRDTTrainerInitInfo,
)
engine = WeightTransferTrainerFactory.trainer_init(
init_info=ShardedRDTTrainerInitInfo(
rank=rank, # rank 0 is the sender
num_consumers=8, # inference workers, fleet-wide
trainer_actor_namespace="my_namespace", # must be visible to the workers
),
client=HTTPVLLMWeightSyncClient("http://localhost:8000"),
source=ModuleSource(model),
)
engine.send_weights() # once per sync, on every trainer rank
```
`trainer_init` and `send_weights` run on **every** trainer rank: each one owns a serve actor and takes part in the gathers, while only rank 0 drives the inference-side handshake. Any [`VLLMWeightSyncClient`](base.md#vllmweightsyncclient) works.
To adapt a trainer that is not a plain `nn.Module` — a Megatron export, a raw sharded checkpoint — subclass [`WeightSource`](base.md#weightsource).
### `ShardedRDTTrainerInitInfo`
| Field | Default | Description |
| ----- | ------- | ----------- |
| `rank` | — | Keyword-only. This trainer rank; **0 is the sender** |
| `num_consumers` | — | Inference workers across the whole fleet (TP × DP) |
| `trainer_actor_namespace` | `None` | Ray namespace for the serve actors; the workers resolve them by name here |
| `num_rdt_buffers` | 2 | Ring depth on both sides |
| `buffer_presize_gb` | 0.0 | Pre-size each buffer slot, in GiB. Set it to cover the largest atomic slice |
| `gather_lookahead` | 1 | Gathered-but-unfreed layers the gather loop runs ahead by |
| `stall_timeout_s` | 300.0 | Seconds without progress before the sync fails. A liveness backstop for a consumer that dies mid-sync, not a latency target |
## Examples
- [Small MoE on 4 GPUs](../../../examples/rl/rlhf_sharded_rdt_small_ep.py) — 2 FSDP2 trainer ranks → 2 vLLM DP ranks with expert parallelism, one node. It pairs a trainer fleet with a separate inference fleet, the only arrangement this backend supports, and asserts that the sync moved the weights and that a second sync leaves generation unchanged — so it runs unattended in CI
It keeps the trainer deliberately small — just enough FSDP2 to make the weights real — so the file stays about the weight sync rather than about the trainer. For a full RL trainer, SkyRL integrates this backend with Megatron (PP-local gathering and expert-stack fusion for MoE) alongside FSDP: [NovaSky-AI/SkyRL#1753](https://github.com/NovaSky-AI/SkyRL/pull/1753).