[RL] P2P RDT weight sync (#43375)
This commit is contained in:
@@ -34,6 +34,7 @@ engine drives on your behalf:
|
||||
| [NCCL](nccl.md) | NCCL broadcast | Separate GPUs for training and inference |
|
||||
| [IPC](ipc.md) | CUDA IPC handles | Colocated training and inference on same GPU |
|
||||
| [sparse_nccl](nccl.md#sparse-nccl) | NCCL broadcast | Sparse flat-index weight patches (TP=1/PP=1) |
|
||||
| [sharded_rdt](sharded_rdt.md) | NIXL / Ray Direct Transport (pull-based) | Very large models where each worker needs only its own slice (MoE with expert parallelism) |
|
||||
|
||||
## Quickstart
|
||||
|
||||
@@ -48,7 +49,7 @@ from vllm.config import WeightTransferConfig
|
||||
|
||||
llm = LLM(
|
||||
model="my-model",
|
||||
weight_transfer_config=WeightTransferConfig(backend="nccl"), # or "ipc", "sparse_nccl"
|
||||
weight_transfer_config=WeightTransferConfig(backend="nccl"), # or "ipc", "sparse_nccl", "sharded_rdt"
|
||||
)
|
||||
```
|
||||
|
||||
@@ -158,7 +159,7 @@ When running vLLM as an HTTP server, the following endpoints are available for w
|
||||
!!! note
|
||||
The HTTP weight transfer endpoints require `VLLM_SERVER_DEV_MODE=1` to be set.
|
||||
|
||||
The Rust frontend's optional gRPC `Control` service exposes the same pause, sleep, weight-transfer, and weight-version lifecycle for trusted sidecars. The `ServerInfo.rl_capabilities` response reports whether weight transfer and sleep mode were configured. Backend-specific `init_info` and `update_info` remain JSON metadata; model tensors continue to move over the configured NCCL, IPC, or sparse-NCCL transport.
|
||||
The Rust frontend's optional gRPC `Control` service exposes the same pause, sleep, weight-transfer, and weight-version lifecycle for trusted sidecars. The `ServerInfo.rl_capabilities` response reports whether weight transfer and sleep mode were configured. Backend-specific `init_info` and `update_info` remain JSON metadata; model tensors continue to move over the configured NCCL, IPC, sparse-NCCL, or sharded-RDT transport.
|
||||
|
||||
## Extending the System
|
||||
|
||||
|
||||
@@ -20,8 +20,14 @@ vice versa, so the registries stay independent.
|
||||
|
||||
### WeightSource
|
||||
|
||||
A `WeightSource` is a **re-iterable** source of the trainer's weights with two
|
||||
channels:
|
||||
**This is the adapter for whatever shape your trainer's weights are in.**
|
||||
`WeightSource` defines how you extract your weights for your specific framework.
|
||||
|
||||
What an engine gets is always the same: HF-format parameter names, and tensors
|
||||
already materialized to their full (unsharded) shape. Whatever gathering,
|
||||
re-fusing, dequantizing or renaming that takes belongs inside the source.
|
||||
|
||||
A `WeightSource` is **re-iterable** and has two required channels:
|
||||
|
||||
- **`metadata() -> list[ParamMeta]`** — the name, wire dtype, and full shape of
|
||||
every parameter, *without transferring anything*. Cheap when shapes are known
|
||||
@@ -51,9 +57,7 @@ class ParamMeta:
|
||||
Materializing is typically a collective, so **every trainer rank must iterate the
|
||||
same source in the same order, in lockstep**, or ranks deadlock. `metadata()` can
|
||||
itself be a collective for custom producers, so it too runs on every rank — only
|
||||
the sender ships the result. Under pipeline parallelism a rank may not own a
|
||||
parameter at all; iterating still drives the collective, and the yielded tensor
|
||||
is only meaningful on the sender.
|
||||
the sender ships the result.
|
||||
|
||||
`iter(source)` must yield a *fresh* pass each round.
|
||||
|
||||
@@ -70,35 +74,133 @@ from vllm.distributed.weight_transfer import ModuleSource
|
||||
source = ModuleSource(model)
|
||||
```
|
||||
|
||||
#### Custom Sources
|
||||
#### Custom sources
|
||||
|
||||
Subclass `WeightSource` when the weights you want to send require additional processing to convert to a HF compatible format.
|
||||
Subclass `WeightSource` when the weights need work to reach HF format — a
|
||||
framework-specific export, a re-fusing step, a dtype cast.
|
||||
|
||||
```python
|
||||
from vllm.distributed.weight_transfer import ParamMeta, WeightSource
|
||||
from vllm.distributed.weight_transfer.base import materialize_full_tensor
|
||||
|
||||
|
||||
class MyExportSource(WeightSource):
|
||||
def __init__(self, model):
|
||||
self._model = model
|
||||
class MegatronBridgeSource(WeightSource):
|
||||
"""Megatron model -> HF names, via a bridge that gathers TP/PP/EP internally
|
||||
and returns full tensors on every rank."""
|
||||
|
||||
def __init__(self, bridge, module, dtype):
|
||||
self._bridge, self._module, self._dtype = bridge, module, dtype
|
||||
self._meta: list[ParamMeta] | None = None
|
||||
|
||||
def _export(self):
|
||||
return self._bridge.export_hf_weights(self._module)
|
||||
|
||||
def metadata(self) -> list[ParamMeta]:
|
||||
# Cache: for producers that must materialize to learn shapes, this is
|
||||
# the expensive channel. Runs on every rank (it may be a collective).
|
||||
if self._meta is None:
|
||||
self._meta = [
|
||||
ParamMeta(name, t.dtype, tuple(t.shape)) for name, t in self._export()
|
||||
ParamMeta(name, self._dtype, tuple(t.shape))
|
||||
for name, t in self._export()
|
||||
]
|
||||
return self._meta
|
||||
|
||||
def __iter__(self):
|
||||
# Must yield exactly what metadata() declared, in the same order.
|
||||
for name, tensor in self._export():
|
||||
yield name, materialize_full_tensor(tensor)
|
||||
yield name, tensor.to(self._dtype).detach().contiguous()
|
||||
```
|
||||
|
||||
#### `held_names()`: partial ownership
|
||||
|
||||
By default every rank is assumed to be able to produce every parameter — which is
|
||||
what the source above does, since its bridge gathers across all parallelism before
|
||||
yielding. That is simple and always correct, but it means the gather cost is paid
|
||||
in full on every rank.
|
||||
|
||||
Optionally, you can override `held_names()` when the ranks are split so each holds only part of the
|
||||
model. It returns the
|
||||
parameter names this rank holds (or `None`, the default, for all of them):
|
||||
|
||||
```python
|
||||
def held_names(self):
|
||||
# This pipeline stage's layers, and within them only this EP rank's experts.
|
||||
return self._my_stage_names - self._foreign_expert_names
|
||||
```
|
||||
|
||||
This covers various trainer layouts — pipeline stages (a rank holds some layers),
|
||||
expert parallelism (a rank holds some experts), both at once, or a shape that fits
|
||||
neither. Backends that can route per parameter (see
|
||||
[sharded RDT](sharded_rdt.md)) then pull each name from a rank that actually
|
||||
holds it.
|
||||
|
||||
Three requirements come with overriding it:
|
||||
|
||||
- **`metadata()` must still describe the whole model on every rank.** Only the
|
||||
sender's metadata reaches the inference side, so a rank that reported just its
|
||||
own share would leave the rest silently un-transferred. Sharded RDT cross-checks
|
||||
this across ranks at init.
|
||||
- **Every name must be held by at least one rank**, or it can never be served.
|
||||
The engine raises at init naming the first orphan.
|
||||
- **Iteration yields `None` for a name this rank does not hold.** The name still
|
||||
appears, in metadata order, so the order check stays aligned across ranks — only
|
||||
the data is absent. Claiming a name and then yielding `None` for it is an error
|
||||
the engine reports by name.
|
||||
|
||||
!!! warning "Partial ownership only works with sharded RDT"
|
||||
Only a backend that routes per parameter honours `held_names()`.
|
||||
Broadcast backends ignore it and send every name from every rank, so
|
||||
declaring partial ownership there changes nothing.
|
||||
|
||||
#### Gather groups
|
||||
|
||||
Some backends transfer a layer at a time rather than a model at a time, so they
|
||||
partition `metadata()` into **gather groups**. `layerwise_groups` keys each name
|
||||
on the **outermost index segment** it contains, so **a group is one decoder
|
||||
layer**, with runs of un-indexed names (the embeddings, the final norm,
|
||||
`lm_head`) forming groups of their own where they appear:
|
||||
|
||||
```text
|
||||
group 0 model.embed_tokens.weight
|
||||
group 1 model.layers.0.* <- one decoder layer
|
||||
group 2 model.layers.1.*
|
||||
...
|
||||
group N+1 model.norm.weight, lm_head.weight
|
||||
```
|
||||
|
||||
Keying on the index rather than a literal prefix means no per-architecture table:
|
||||
`model.layers.0.`, `model.language_model.layers.0.` (recent Qwen text
|
||||
checkpoints), `transformer.h.0.` (GPT-2, Falcon), `backbone.layers.0.` (Mamba)
|
||||
and a vision tower's `visual.blocks.0.` all partition the same way. The index
|
||||
taken is the *outermost* one, which keeps a MoE layer whole — a per-expert name
|
||||
like `model.layers.3.mlp.experts.7.w1` keys on the layer, not the expert.
|
||||
|
||||
Group index *g* means the same layer on every rank and every consumer, because
|
||||
every side derives it from one rank's `metadata()`. That agreement is what lets a
|
||||
backend bound its buffers to one layer and free a layer once everyone is done
|
||||
with it.
|
||||
|
||||
!!! warning "A leaf module's sources must all fall in one group"
|
||||
The sharded-RDT engine frees a group as soon as its last chunk lands, so a
|
||||
module split across groups would park a pull until the stall watchdog fires.
|
||||
The default partition guarantees this; a `groups()` override must preserve
|
||||
it.
|
||||
|
||||
Two hooks follow from this, both with working defaults:
|
||||
|
||||
- **`groups()`** — this rank's groups, in metadata order. The default is
|
||||
`layerwise_groups(metadata())` restricted to the groups holding at least one
|
||||
held name. A group with nothing held here is skipped entirely.
|
||||
- **`iter_groups()`** — the same stream, batched one group at a time. The default
|
||||
drives `__iter__` and batches its output, checking that names arrive in metadata
|
||||
order. Override it when your framework can produce a whole group in one step:
|
||||
materializing is usually a collective, and driving it per group rather than per
|
||||
tensor turns ~37k generator resumes into ~95 on a per-expert MoE model.
|
||||
|
||||
Because `metadata()` order defines the partition, **all names sharing a layer
|
||||
index must be contiguous in it**. A source whose natural export order interleaves
|
||||
layers — bucketing all the MoE experts together, say — has to reorder before
|
||||
returning.
|
||||
|
||||
### VLLMWeightSyncClient
|
||||
|
||||
**This is the adapter for however your RL stack reaches vLLM.** Many RL frameworks wrap
|
||||
@@ -372,7 +474,7 @@ Subclasses must implement five methods:
|
||||
The base class provides:
|
||||
|
||||
1. `__init__`, taking `config` (`WeightTransferConfig`), `vllm_config` (`VllmConfig`), `device` (`torch.device`), and `model` (`nn.Module`).
|
||||
2. `update_weights(update_info_dict)`, a thin wrapper for `receive_weights`: it parses the dict into the typed dataclass, calls `receive_weights`, and synchronizes the device.
|
||||
2. `update_weights(update_info_dict)`, a thin wrapper for `receive_weights`: it parses the dict into the typed dataclass, calls `receive_weights`, and synchronizes the device — unless the engine sets `defers_processing`, below.
|
||||
3. `parse_init_info` / `parse_update_info`, which convert API-level dicts into the typed dataclasses and raise `ValueError` on a bad payload.
|
||||
4. `set_weight_update_target` / `reset_weight_update_target`, used to retarget an update at the speculative draft model.
|
||||
|
||||
@@ -383,6 +485,25 @@ The base class provides:
|
||||
update info carries only per-round metadata. This is what makes a
|
||||
trainer/worker mismatch unrepresentable.
|
||||
|
||||
!!! note "`defers_processing`: when a returned update means queued, not applied"
|
||||
An engine that pipelines its GPU post-processing onto background threads
|
||||
cannot let `update_weights` synchronize the device — that would block on
|
||||
those threads and serialize the pipeline. Such an engine sets the class
|
||||
attribute `defers_processing = True`, omits the per-update sync, and
|
||||
guarantees completion in `finish_weight_update` instead.
|
||||
|
||||
Callers that go through `finish_weight_update` need do nothing; the engine
|
||||
drains there. A caller that drives the tail itself — running its own
|
||||
`finalize_layerwise_reload`, say — must check the flag and call
|
||||
`drain_pending()` first, because with it set a returned `update_weights`
|
||||
means *queued*, not *applied*. `drain_pending()` is idempotent, and a no-op
|
||||
on an engine that processes synchronously, so it is always safe to call.
|
||||
|
||||
[Sharded RDT](sharded_rdt.md) is the built-in engine that sets it: it
|
||||
scatters and quantizes on background threads with their own CUDA streams, so
|
||||
its `drain_pending()` joins both queues and syncs both streams before
|
||||
`finalize_layerwise_reload` runs.
|
||||
|
||||
### Request Classes
|
||||
|
||||
The API-level request classes provide backend-agnostic serialization using plain dictionaries.
|
||||
@@ -497,7 +618,7 @@ Once registered, users select your backend via `WeightTransferConfig(backend="my
|
||||
|
||||
### WeightTransferEngineFactory
|
||||
|
||||
The factory uses a registry pattern with lazy loading. Built-in engines (`nccl`, `ipc`, and `sparse_nccl`) are registered at import time but their modules are only loaded when the backend is actually requested. This avoids importing heavy dependencies (like NCCL communicators) when they aren't needed.
|
||||
The factory uses a registry pattern with lazy loading. Built-in engines (`nccl`, `ipc`, `sparse_nccl` and `sharded_rdt`) are registered at import time but their modules are only loaded when the backend is actually requested. This avoids importing heavy dependencies (like NCCL communicators) when they aren't needed.
|
||||
|
||||
```python
|
||||
from vllm.distributed.weight_transfer import WeightTransferEngineFactory
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
# Sharded RDT Engine
|
||||
|
||||
The sharded RDT weight transfer engine moves weights point-to-point over [NIXL](https://github.com/ai-dynamo/nixl) using Ray Direct Transport (RDT), Ray's zero-copy tensor transport between actors. It is **pull-based**: the inference workers initiate every transfer and each one asks only for the *slice* it consumes under tensor and expert parallelism. A large MoE model therefore moves roughly `total_bytes / num_workers` per worker instead of the `total_bytes` a broadcast costs.
|
||||
|
||||
## When to Use Sharded RDT
|
||||
|
||||
- Very large models where broadcasting whole parameters is the bottleneck — typically an MoE served with expert parallelism, where each worker owns a small fraction of the experts
|
||||
- Trainer and inference on **separate GPUs**, over a fabric NIXL supports (InfiniBand, RoCE, EFA)
|
||||
- Trainers that are themselves sharded, including **pipeline-parallel** ones where a rank holds only part of the model
|
||||
|
||||
Requirements:
|
||||
|
||||
- `distributed_executor_backend="ray"` — the workers must be Ray actors
|
||||
- Ray `>= 2.56.0` on both the trainer and the workers
|
||||
- `nixl` installed in the environment shared by trainer and workers
|
||||
- Weight loaders that stay inside the supported op set (below)
|
||||
- EPLB (`enable_eplb=true`) is rejected: it rearranges experts at runtime, which invalidates the recorded plan
|
||||
|
||||
## How It Works
|
||||
|
||||
### Slices are tracked through vLLM's own weight loaders
|
||||
|
||||
A weight loader normally receives a full HF-format tensor and slices out the part this worker needs. Instead of sending it one, the engine hands it a `FakeRDTTensor`: a zero-storage tensor that answers `.shape` / `.dtype` / `.size()` but holds no data. Every view or slice op the loader calls returns a new fake with that op appended to a recorded chain, and `copy_` is the sink that ends it.
|
||||
|
||||
That chain is the wire format. `("model.layers.0.w", (("narrow", (0, 512, 512), ()), ("t", (), ())))` tells the trainer: take this tensor, `narrow` it, transpose it, send the result. The trainer replays it with `getattr(tensor, op)(*args, **kwargs)`.
|
||||
|
||||
Discovery is expensive, so it happens **once**, at `init_transfer_engine`, as a dry run over `model.load_weights` with every parameter on meta. Nothing is transferred; the engine just records, per leaf module, which slice feeds which destination region. Every later sync is pure replay.
|
||||
|
||||
Anything a loader does that needs real data — arithmetic, `.to()`, `.float()`, `.item()`, `.data`, bool-mask indexing — falls outside the allowlist and raises at init. That is deliberate: failing loudly during setup beats silently transferring the wrong bytes. `SUPPORTED_OPS` in `sharded_rdt_common.py` is the single table both sides derive from, so the recorder and the replayer cannot drift.
|
||||
|
||||
### Received slices land directly in the layerwise reload buffers
|
||||
|
||||
The engine drives [layerwise reload](../layerwise.md) itself, in `start_weight_update` / `finish_weight_update`. Because the dry run already recorded each destination as an `as_strided` region of its parameter, an arriving slice is copied straight into the layer being reloaded — no full HF tensor is ever materialized on the worker, and no second pass over `load_weights` runs. Each layer is quantized and copied into its persistent kernel storage as soon as its last slice lands.
|
||||
|
||||
### Gathers and pulls are pipelined, which is what `gather_lookahead` bounds
|
||||
|
||||
The trainer usually cannot serve its parameters as they sit: FSDP shards them, and even an EP-split trainer has to assemble a whole expert. So each sync still runs gather collectives — but a layer at a time, not a model at a time.
|
||||
|
||||
**A gather group is one decoder layer.** The parameter list is keyed on the outermost index segment of each name, which leaves runs of un-indexed names — the embeddings before the first layer, the final norm and `lm_head` after the last — as groups of their own:
|
||||
|
||||
```text
|
||||
group 0 model.embed_tokens.weight
|
||||
group 1 model.layers.0.* <- one decoder layer
|
||||
group 2 model.layers.1.*
|
||||
...
|
||||
group N+1 model.norm.weight, lm_head.weight
|
||||
```
|
||||
|
||||
The layer is the unit of everything that follows: the trainer gathers a layer, publishes it (immediately pullable), and moves on to the next while the consumers pull the one it just published. Once every consumer has signalled that it is done with a layer, the trainer drops it and gains a credit to gather another. `gather_lookahead` is how far ahead of the consumers that loop may run, so at most `gather_lookahead + 1` layers are resident on the trainer at a time. The default of 1 keeps the next layer gathered and pullable while the current one is being pulled — enough to hide the handoff without doubling trainer memory. Raise it only if one layer's gather is slower than its pulls.
|
||||
|
||||
Because a layer is also the unit the consumers free and the unit the receive buffers are sized against, it is what keeps memory bounded on both sides: without it the whole model would be one transfer, and both sides would have to hold their full share of it at once. Keying on the index rather than a fixed `model.layers.` prefix is what makes that hold across naming conventions — a VLM's `model.language_model.layers.`, GPT-2's `transformer.h.`, a vision tower's `visual.blocks.`. Sources can control the partition; see [gather groups](base.md#gather-groups).
|
||||
|
||||
### Ownership
|
||||
|
||||
A trainer rank need not hold the whole model. Each one declares what it holds through [`WeightSource.held_names()`](base.md#held_names-partial-ownership), the fleet all-gathers those declarations at `trainer_init`, and the consumers route each pull to a rank that actually holds the name. Pipeline stages, expert parallelism, and combinations of the two are all the same declaration. Consumers spread their pulls across the ranks that hold a name, so no single trainer NIC becomes the bottleneck.
|
||||
|
||||
## Inference Side
|
||||
|
||||
```python
|
||||
from vllm import LLM
|
||||
from vllm.config import WeightTransferConfig
|
||||
|
||||
llm = LLM(
|
||||
model="my-model",
|
||||
weight_transfer_config=WeightTransferConfig(backend="sharded_rdt"),
|
||||
distributed_executor_backend="ray",
|
||||
)
|
||||
```
|
||||
|
||||
```bash
|
||||
vllm serve my-model \
|
||||
--distributed-executor-backend ray \
|
||||
--weight-transfer-config '{"backend": "sharded_rdt"}'
|
||||
```
|
||||
|
||||
Everything else — which producers exist, how the model splits into layer groups, the ownership table — arrives from the trainer at the init handshake.
|
||||
|
||||
!!! warning "Size the receive buffers before choosing `gpu_memory_utilization`"
|
||||
Each worker holds `num_rdt_buffers` receive buffers, each large enough for the biggest single slice batch it pulls. Like NCCL and NIXL internals, they do **not** count against `gpu_memory_utilization`, so a fraction that leaves no headroom OOMs at the first sync even though the engine came up healthy. The buffer size is driven by the largest atomic slice — for an untied vocab matrix on a worker that holds it unsliced, that is the whole embedding.
|
||||
|
||||
## Trainer Side
|
||||
|
||||
```python
|
||||
from vllm.distributed.weight_transfer import (
|
||||
ModuleSource,
|
||||
HTTPVLLMWeightSyncClient,
|
||||
WeightTransferTrainerFactory,
|
||||
)
|
||||
from vllm.distributed.weight_transfer.sharded_rdt_trainer import (
|
||||
ShardedRDTTrainerInitInfo,
|
||||
)
|
||||
|
||||
engine = WeightTransferTrainerFactory.trainer_init(
|
||||
init_info=ShardedRDTTrainerInitInfo(
|
||||
rank=rank, # rank 0 is the sender
|
||||
num_consumers=8, # inference workers, fleet-wide
|
||||
trainer_actor_namespace="my_namespace", # must be visible to the workers
|
||||
),
|
||||
client=HTTPVLLMWeightSyncClient("http://localhost:8000"),
|
||||
source=ModuleSource(model),
|
||||
)
|
||||
|
||||
engine.send_weights() # once per sync, on every trainer rank
|
||||
```
|
||||
|
||||
`trainer_init` and `send_weights` run on **every** trainer rank: each one owns a serve actor and takes part in the gathers, while only rank 0 drives the inference-side handshake. Any [`VLLMWeightSyncClient`](base.md#vllmweightsyncclient) works.
|
||||
|
||||
To adapt a trainer that is not a plain `nn.Module` — a Megatron export, a raw sharded checkpoint — subclass [`WeightSource`](base.md#weightsource).
|
||||
|
||||
### `ShardedRDTTrainerInitInfo`
|
||||
|
||||
| Field | Default | Description |
|
||||
| ----- | ------- | ----------- |
|
||||
| `rank` | — | Keyword-only. This trainer rank; **0 is the sender** |
|
||||
| `num_consumers` | — | Inference workers across the whole fleet (TP × DP) |
|
||||
| `trainer_actor_namespace` | `None` | Ray namespace for the serve actors; the workers resolve them by name here |
|
||||
| `num_rdt_buffers` | 2 | Ring depth on both sides |
|
||||
| `buffer_presize_gb` | 0.0 | Pre-size each buffer slot, in GiB. Set it to cover the largest atomic slice |
|
||||
| `gather_lookahead` | 1 | Gathered-but-unfreed layers the gather loop runs ahead by |
|
||||
| `stall_timeout_s` | 300.0 | Seconds without progress before the sync fails. A liveness backstop for a consumer that dies mid-sync, not a latency target |
|
||||
|
||||
## Examples
|
||||
|
||||
- [Small MoE on 4 GPUs](../../../examples/rl/rlhf_sharded_rdt_small_ep.py) — 2 FSDP2 trainer ranks → 2 vLLM DP ranks with expert parallelism, one node. It pairs a trainer fleet with a separate inference fleet, the only arrangement this backend supports, and asserts that the sync moved the weights and that a second sync leaves generation unchanged — so it runs unattended in CI
|
||||
|
||||
It keeps the trainer deliberately small — just enough FSDP2 to make the weights real — so the file stays about the weight sync rather than about the trainer. For a full RL trainer, SkyRL integrates this backend with Megatron (PP-local gathering and expert-stack fusion for MoE) alongside FSDP: [NovaSky-AI/SkyRL#1753](https://github.com/NovaSky-AI/SkyRL/pull/1753).
|
||||
Reference in New Issue
Block a user