Merge branch 'main' into patches
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
@@ -466,6 +466,31 @@ This approach is more robust than index-based access (`messages[0]`, `messages[1
|
||||
|
||||
Example template file: [examples/pooling/score/template/nemotron-rerank.jinja](../../../examples/pooling/score/template/nemotron-rerank.jinja)
|
||||
|
||||
#### Templates declared by Sentence Transformers checkpoints
|
||||
|
||||
vLLM does not apply a checkpoint's chat template by default, since many rerankers inherit an unrelated template from their base LLM. The exception is a Sentence Transformers (v5.4+) `CrossEncoder` with a single `sequence-classification` Transformer module whose `sentence_bert_config.json` declares the `message` modality. Sentence Transformers formats the pairs of such checkpoints as `query`/`document` messages, so vLLM does the same with the saved template:
|
||||
|
||||
```json
|
||||
{
|
||||
"transformer_task": "sequence-classification",
|
||||
"modality_config": {
|
||||
"text": {"method": "forward", "method_output_name": "logits"},
|
||||
"message": {"method": "forward", "method_output_name": "logits", "format": "flat"}
|
||||
},
|
||||
"module_output_name": "scores",
|
||||
"processing_kwargs": {"chat_template": {"add_generation_prompt": true}}
|
||||
}
|
||||
```
|
||||
|
||||
- The default chat template is used, unless `processing_kwargs.chat_template.chat_template` names another template saved with the tokenizer (e.g. `additional_chat_templates/score.jinja` for `"score"`).
|
||||
- The other `processing_kwargs.chat_template` settings, such as `add_generation_prompt`, are passed to the template; the request's `chat_template_kwargs` override them.
|
||||
- Message contents are passed as strings, as with `--chat-template`.
|
||||
- vLLM does not restore the template suffix after truncation (`restore_suffix`). Use `max_tokens_per_doc` rather than `truncate_prompt_tokens` to truncate long documents.
|
||||
- `--chat-template` (or the `chat_template` argument of `LLM.score`) takes precedence over the saved template and its settings.
|
||||
- Models with a built-in score template keep using it.
|
||||
|
||||
The template in use is logged at startup.
|
||||
|
||||
### Enable/disable activation
|
||||
|
||||
You can enable or disable activation via `use_activation` only works for cross-encoder models.
|
||||
|
||||
@@ -313,14 +313,12 @@ Start the server:
|
||||
```shell
|
||||
vllm serve nvidia/llama-nemotron-rerank-vl-1b-v2 \
|
||||
--runner pooling \
|
||||
--trust-remote-code \
|
||||
--chat-template examples/pooling/score/template/nemotron-vl-rerank.jinja
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
!!! note
|
||||
The chat template bundled with this checkpoint's tokenizer is not suitable
|
||||
for the Score/Rerank APIs. Use the provided override template when serving:
|
||||
`examples/pooling/score/template/nemotron-vl-rerank.jinja`.
|
||||
The checkpoint declares its score template in its Sentence Transformers config, so vLLM applies it automatically (see [Templates declared by Sentence Transformers checkpoints](scoring.md#templates-declared-by-sentence-transformers-checkpoints)).
|
||||
Revisions older than October 2026 need `--chat-template examples/pooling/score/template/nemotron-vl-rerank.jinja`.
|
||||
|
||||
Score a text query against an image document:
|
||||
|
||||
|
||||
@@ -1,8 +1,50 @@
|
||||
# Tokens In <> Tokens Out API
|
||||
|
||||
`/inference/v1/generate` takes prompt token IDs and returns generated token IDs. It is the generation step of the `render → generate → derender` pipeline used in disaggregated serving (see the [Renderer APIs](renderer.md) and [Derenderer APIs](derenderer.md)).
|
||||
`POST /inference/v1/generate` is the token-in/token-out (TITO) HTTP endpoint: it takes prompt token IDs and returns generated token IDs, logprobs keyed by integer token ID, and `finish_reason`/`stop_reason`, with no detokenization. Use it when the caller does its own tokenization and chat templating, such as an RL rollout worker. It is also the generation step of the `render → generate → derender` pipeline used in disaggregated serving (see the [Renderer APIs](renderer.md) and [Derenderer APIs](derenderer.md)).
|
||||
|
||||
The endpoint is registered on `vllm serve` with `--enable-scale-out`, and always with `--tokens-only`. `--tokens-only` also skips loading the tokenizer, so the server can't turn token IDs back into text.
|
||||
## Enabling the endpoint (`--enable-scale-out`, `--tokens-only`)
|
||||
|
||||
- `--enable-scale-out` registers it, with `/render`, `/derender` and `/inference/v1/abort_requests`.
|
||||
- `--tokens-only` always registers it, skips loading the tokenizer and forces `detokenize=False`. Stop strings (`sampling_params.stop`) are then rejected with a 400; use `stop_token_ids`.
|
||||
- With neither flag, it returns 403 `Scale-out endpoints are disabled. Restart vllm serve with --enable-scale-out to enable them.`
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-0.6B --enable-scale-out
|
||||
|
||||
curl -X POST http://localhost:8000/inference/v1/generate \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"token_ids": [151644, 872, 198], "sampling_params": {"max_tokens": 32, "logprobs": 1}}'
|
||||
```
|
||||
|
||||
A full client is in [examples/scale_out/token_generation_client.py](../../../examples/scale_out/token_generation_client.py).
|
||||
|
||||
## Request fields
|
||||
|
||||
| Field | Notes |
|
||||
| --- | --- |
|
||||
| `token_ids` | Required `list[int]`: the already-templated prompt. Non-empty, no negative IDs. |
|
||||
| `sampling_params` | Required. Any [SamplingParams](../../api/README.md#inference-parameters) field, e.g. `max_tokens`, `temperature`, `logprobs`, `stop_token_ids`. Without `max_tokens`, the server applies its own default (as on the OpenAI endpoints), not 16. |
|
||||
| `output_mode` | `"tokens"` (default) or `"text"`, see [Output modes](#output-modes). |
|
||||
| `stream`, `stream_options` | SSE chunks ending with `data: [DONE]`. |
|
||||
| `return_token_ids` | Echo the final prompt IDs as `prompt_token_ids`. |
|
||||
| `features` / `content_parts` | Multimodal input, mutually exclusive. |
|
||||
| `request_id`, `model`, `cache_salt`, `priority`, `kv_transfer_params`, `ec_transfer_params` | Optional. |
|
||||
|
||||
## Response fields
|
||||
|
||||
| `choices[i]` field | Notes |
|
||||
| --- | --- |
|
||||
| `token_ids` | Generated token IDs. |
|
||||
| `finish_reason` | `stop`, `length` or `abort`. |
|
||||
| `stop_reason` | Stop token ID or string; `null` on EOS, length and abort. |
|
||||
| `logprobs` | `{"content": [{"token_id", "logprob", "rank", "top_logprobs": [...]}]}`, one entry per token, sampled token first. `token_id` is an integer, not the older `"token_id:<id>"` string. Needs `sampling_params.logprobs`. |
|
||||
| `weight_versions`, `routed_experts`, `sampling_mask` | Optional, see [Weight versions](#weight-versions); the last two need `--enable-return-routed-experts` / `--return-sampling-mask`. |
|
||||
|
||||
The top level adds `request_id`, `usage`, `prompt_token_ids` and `prompt_logprobs`. Stream chunks carry the same per-choice fields for the new tokens.
|
||||
|
||||
## Targeting a data-parallel rank (`X-data-parallel-rank`)
|
||||
|
||||
Send the header `X-data-parallel-rank: <int>` to pin a request to one data-parallel (DP) engine rank; there is no body field. Without the header, or with a non-integer value, the request is load balanced. A rank outside `[0, data_parallel_size)` is rejected.
|
||||
|
||||
## Output modes
|
||||
|
||||
|
||||
@@ -18,6 +18,8 @@ The following open-source RL libraries use vLLM for fast rollouts (sorted alphab
|
||||
|
||||
For weight synchronization between training and inference, see the [Weight Transfer](weight_transfer/README.md) documentation, which covers the pluggable backend system with [NCCL](weight_transfer/nccl.md) (multi-GPU) and [IPC](weight_transfer/ipc.md) (same-GPU) engines. To verify that an update landed on every rank, see the [Weight Checker](weight_checker.md).
|
||||
|
||||
For MoE policies, see [Routed Experts Return](routed_experts.md) (`--enable-return-routed-experts`) to get the per-token expert routing of each rollout for routing replay (R3) in the trainer.
|
||||
|
||||
For pipelining generation and training to improve GPU utilization and throughput, see the [Async Reinforcement Learning](async_rl.md) guide, which covers the pause/resume API for safely updating weights mid-flight.
|
||||
|
||||
See the following notebooks showing how to use vLLM for GRPO:
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
# Routed Experts Return (MoE Routing Replay / R3)
|
||||
|
||||
`--enable-return-routed-experts` makes vLLM return the routed experts (the expert IDs each MoE layer's router selected) for every token of a request. RL trainers use it for MoE routing replay (R3): the training forward pass reuses the rollout's routing.
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-30B-A3B --enable-return-routed-experts
|
||||
# offline: LLM("Qwen/Qwen3-30B-A3B", enable_return_routed_experts=True)
|
||||
```
|
||||
|
||||
The flag sets `AuxOutputConfig.enable_return_routed_experts` (full form: `--aux-output-config '{"enable_return_routed_experts": true, "max_bytes": N}'`). There is no per-request on/off switch.
|
||||
|
||||
## Requirements
|
||||
|
||||
Startup fails with a `ValueError` unless:
|
||||
|
||||
- **Model Runner V2** is used (`VLLM_USE_V2_MODEL_RUNNER=1` if vLLM falls back to V1);
|
||||
- the runner is `generate` and the model is **MoE**;
|
||||
- **prefix caching is enabled** (the default). `--no-enable-prefix-caching` fails; the old v0.21 advice to disable it is wrong.
|
||||
|
||||
Rejected at startup: adaptive speculative verification, PP > 1, DCP/PCP > 1, and the `NixlConnector`, `NixlPullConnector`, `NixlPushConnector`, `MoRIIOConnector`, `MooncakeConnector` KV connectors. Regular speculative decoding works.
|
||||
|
||||
### How it works: aux output connector
|
||||
|
||||
Routing is stored by the aux output connector (`vllm/distributed/aux_output_connector/`, configured via `--aux-output-config`), keyed by KV block hash and shared with prefix caching, which is why prefix caching is required. `max_bytes` is its host-memory LRU capacity (default: derived from KV cache capacity). If a prefix-cache hit needs evicted rows, the worker raises `BlockObjectStoreError`; increase `max_bytes`.
|
||||
|
||||
## Output format
|
||||
|
||||
| API | Field |
|
||||
| --- | --- |
|
||||
| Offline (`LLM`, `AsyncLLM`) | `CompletionOutput.routed_experts`: `np.ndarray` or `None` |
|
||||
| `/v1/completions`, `/v1/chat/completions` | `choices[i].routed_experts`: base64 `.npy`, **non-streaming only** |
|
||||
| `/inference/v1/generate` | `routed_experts` per choice (base64 `.npy`); with `stream: true`, on the final chunk |
|
||||
|
||||
- **Shape:** `(num_tokens - 1 - routed_experts_prompt_start, num_moe_layers, num_experts_per_tok)`, `num_tokens` = prompt + output. The last sampled token has no row.
|
||||
- **Rows:** row `i` is position `routed_experts_prompt_start + i` of `prompt_token_ids + output_token_ids`.
|
||||
- **dtype:** `uint8` for at most 256 experts, else `uint16`; logical expert IDs.
|
||||
- Filled only when the request finishes. `None` if the flag is off or the request aborted before any forward pass.
|
||||
|
||||
```python
|
||||
import base64, io
|
||||
import numpy as np
|
||||
|
||||
experts = np.load(io.BytesIO(base64.b64decode(choice["routed_experts"])))
|
||||
```
|
||||
|
||||
## `routed_experts_prompt_start`
|
||||
|
||||
Per-request, default `0`: drops the first N prompt-token rows (e.g. routing already received in earlier turns). Must be `0 <= value <= prompt_len`, else HTTP 400. Pass it top-level on `/v1/*completions`, inside `sampling_params` on `/inference/v1/generate`, or as `SamplingParams(routed_experts_prompt_start=...)`.
|
||||
|
||||
## Limitations
|
||||
|
||||
- The Rust HTTP frontend never returns `routed_experts` and always sends `routed_experts_prompt_start = 0`. Use the Python API server.
|
||||
|
||||
Example: [examples/rl/routed_experts_e2e.py](../../examples/rl/routed_experts_e2e.py).
|
||||
@@ -139,8 +139,9 @@ Each position then carries the sampled token's logprob and the top-128
|
||||
drop them. The finite entries are the sampling mask when it has at most `k`
|
||||
tokens.
|
||||
- The sampled token is always the first entry and may repeat inside the top-k.
|
||||
- `/inference/v1/generate` encodes IDs as `token_id:<id>`; the OpenAI
|
||||
endpoints need `return_tokens_as_token_ids`.
|
||||
- The `token_id:<id>` strings come from the OpenAI endpoints with
|
||||
`return_tokens_as_token_ids`; `/inference/v1/generate` returns an integer
|
||||
`token_id` instead.
|
||||
|
||||
## Limitations
|
||||
|
||||
|
||||
@@ -145,7 +145,7 @@ engine.send_weights() # called on every rank
|
||||
|
||||
## API Endpoints
|
||||
|
||||
When running vLLM as an HTTP server, the following endpoints are available for weight transfer. `HTTPVLLMWeightSyncClient` speaks the first four for you.
|
||||
When running vLLM as an HTTP server, the following endpoints are available for weight transfer. `HTTPVLLMWeightSyncClient` speaks the first five for you. A trainer may also implement the protocol directly without importing vLLM: see the wire contracts in [NCCL](nccl.md#trainer-without-vllm-over-http) and [IPC](ipc.md#trainer-without-vllm-over-http). This keeps the trainer decoupled from vLLM: it needs no vLLM install or version pin, only this documented HTTP and wire contract, so trainer and server can upgrade independently (non-PyTorch trainers such as JAX can use it too).
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| -------- | ------ | ----------- |
|
||||
@@ -153,6 +153,7 @@ When running vLLM as an HTTP server, the following endpoints are available for w
|
||||
| `/start_weight_update` | POST | Start a weight update |
|
||||
| `/update_weights` | POST | Transfer a batch of weights with backend-specific metadata |
|
||||
| `/finish_weight_update` | POST | Finish the update and optionally commit its `weight_version` |
|
||||
| `/shutdown_weight_transfer_engine` | POST | Release the transfer channel without stopping inference |
|
||||
| `/update_weight_version` | POST | Update `weight_version` without changing model weights |
|
||||
| `/weight_info` | GET | Get the latest committed weight version |
|
||||
| `/pause` | POST | Pause generation before weight sync to handle inflight requests |
|
||||
@@ -162,6 +163,12 @@ When running vLLM as an HTTP server, the following endpoints are available for w
|
||||
!!! note
|
||||
The HTTP weight transfer endpoints require `VLLM_SERVER_DEV_MODE=1` to be set.
|
||||
|
||||
!!! warning
|
||||
If `/update_weights` fails, the model is left mid-reload while `/health`
|
||||
stays 200, and `/finish_weight_update` is rejected. Before generating again,
|
||||
resend a full `/start_weight_update` → `/update_weights` →
|
||||
`/finish_weight_update` (see [#57854](https://github.com/vllm-project/vllm/pull/57854)).
|
||||
|
||||
The Rust frontend's optional gRPC `Control` service exposes the same pause, sleep, weight-transfer, and weight-version lifecycle for trusted sidecars. The `ServerInfo.rl_capabilities` response reports whether weight transfer and sleep mode were configured. Backend-specific `init_info` and `update_info` remain JSON metadata; model tensors continue to move over the configured NCCL, IPC, sparse-NCCL, or sharded-RDT transport.
|
||||
|
||||
## Extending the System
|
||||
|
||||
@@ -110,6 +110,43 @@ mode carries a per-chunk barrier across ranks: without it, a rank could overwrit
|
||||
its buffer while its colocated worker is still reading the current chunk. This is
|
||||
handled inside `send_weights()`.
|
||||
|
||||
## Trainer Without vLLM over HTTP
|
||||
|
||||
A colocated trainer can speak the [HTTP endpoints](README.md#api-endpoints)
|
||||
directly, with only PyTorch and an HTTP client. The server needs
|
||||
`VLLM_ALLOW_INSECURE_SERIALIZATION=1`; without it, init and start return 200 and
|
||||
the first `/update_weights` returns 500. Reference:
|
||||
[`rlhf_http_ipc_no_vllm.py`](../../../examples/rl/rlhf_http_ipc_no_vllm.py).
|
||||
|
||||
`POST /init_weight_transfer_engine` body: `{"init_info": {"packed": false}}`.
|
||||
|
||||
`POST /update_weights` body `{"update_info": {...}}`:
|
||||
|
||||
| Field | Meaning |
|
||||
| ----- | ------- |
|
||||
| `names`, `dtype_names`, `shapes` | Checkpoint names, `torch` dtype attribute names (`"bfloat16"`), full shapes |
|
||||
| `ipc_handles_pickled` | `base64(pickle.dumps(handles))`. Unpacked: a list with one dict per name. Packed: one dict for the shared buffer |
|
||||
| `tensor_sizes` | Packed only: byte size of each tensor in the chunk, in `names` order |
|
||||
|
||||
- A handle dict is `{gpu_uuid: reduce_tensor(t)[1]}` with one entry per GPU the
|
||||
server uses (`torch.multiprocessing.reductions.reduce_tensor`).
|
||||
`gpu_uuid = str(torch.cuda.get_device_properties(i).uuid)`, **without** the
|
||||
`GPU-` prefix that `nvidia-smi` prints. The worker overwrites `args[6]`
|
||||
(device index) with its own.
|
||||
- `ipc_handles` (unpickled) is not JSON-serializable; over HTTP use
|
||||
`ipc_handles_pickled`.
|
||||
- Packed: export one `uint8` buffer once and reuse it for every chunk. Copy
|
||||
`t.contiguous().view(-1).view(torch.uint8)` in `names` order and close a chunk
|
||||
**before** the next tensor would overflow it (unlike NCCL, which closes after).
|
||||
A tensor larger than the buffer is an error. Synchronize before each
|
||||
`/update_weights`; do not overwrite the buffer until the call returns.
|
||||
- Keep every exported tensor alive until `/finish_weight_update` returns, then
|
||||
drop the references and call `torch.cuda.ipc_collect()`.
|
||||
- Multi-rank trainers: each rank exports handles for its own GPU, the ranks
|
||||
gather the `{uuid: args}` dicts to rank 0 over their own `torch.distributed`
|
||||
group and merge them per index, and rank 0 sends all HTTP. In packed mode,
|
||||
barrier before and after each `/update_weights`.
|
||||
|
||||
## Rank-Local Updates
|
||||
|
||||
For different parameter subsets per worker, `update_info` may be a list indexed
|
||||
@@ -119,4 +156,5 @@ worker to participate.
|
||||
## Examples
|
||||
|
||||
- [RLHF with IPC weight syncing (`vllm serve`, HTTP)](../../../examples/rl/rlhf_http_ipc.py) - **Start here.** Server and training model share a single GPU; HTTP control plane, CUDA IPC data plane. Launches and tears down its own server
|
||||
- [IPC weight syncing from a trainer that does not import vLLM](../../../examples/rl/rlhf_http_ipc_no_vllm.py) - `torch` + `requests` only, `ipc_handles_pickled` over HTTP; see [Trainer Without vLLM over HTTP](#trainer-without-vllm-over-http)
|
||||
- [RLHF with IPC + FSDP2 and expert parallelism](../../../examples/rl/rlhf_ipc_fsdp_ep.py) - Multi-rank trainer colocated with a `--data-parallel-size 4` server on the same 4 GPUs: every FSDP rank builds an engine and joins the handle all-gather, with packed chunking and sleep/wake around the transfer
|
||||
|
||||
@@ -130,6 +130,51 @@ pass — this is the invariant to test first.
|
||||
IPC does not read `metadata()` at all: it derives the update info from
|
||||
iteration as it goes, so it cannot observe a divergence.
|
||||
|
||||
## Trainer Without vLLM over HTTP
|
||||
|
||||
A trainer can speak the [HTTP endpoints](README.md#api-endpoints) directly, with
|
||||
only PyTorch and an HTTP client. It mints the `ncclUniqueId` itself and is NCCL
|
||||
rank 0. Reference: [`rlhf_http_nccl_no_vllm.py`](../../../examples/rl/rlhf_http_nccl_no_vllm.py).
|
||||
|
||||
`POST /init_weight_transfer_engine` body `{"init_info": {...}}`:
|
||||
|
||||
| Field | Default | Meaning |
|
||||
| ----- | ------- | ------- |
|
||||
| `nccl_unique_id_b64` | — | Standard (not URL-safe) base64 of the 128-byte `ncclUniqueId`. Replaces `master_address` / `master_port` |
|
||||
| `rank_offset` | — | First worker rank; `1` for one trainer rank |
|
||||
| `world_size` | — | Trainer ranks + `GET /get_world_size` |
|
||||
| `packed` | `false` | Packed broadcasting |
|
||||
| `packed_buffer_size_bytes` | 1 GiB | Must match the trainer's chunking |
|
||||
| `packed_num_buffers` | 2 | Worker-side receive buffers |
|
||||
|
||||
- Worker rank = `rank_offset + dp_index * (TP * PP) + rank`.
|
||||
- `/init_weight_transfer_engine` returns only after all ranks join: send it in
|
||||
the background while the trainer calls `ncclCommInitRank(world_size, uid, 0)`.
|
||||
There is no timeout; if the trainer never joins, the workers wait forever.
|
||||
- Right after init every worker issues a warm-up `all_reduce` (sum) of
|
||||
`torch.zeros(1)` (one float32 element). The trainer must issue the same one
|
||||
before any other collective.
|
||||
|
||||
Each sync: `POST /start_weight_update`, then `POST /update_weights` with
|
||||
`{"update_info": {"names", "dtype_names", "shapes"}}` **in the background**
|
||||
while rank 0 broadcasts in `names` order, then `POST /finish_weight_update`.
|
||||
`dtype_names` are `torch` attribute names (`"bfloat16"`); `shapes` are full
|
||||
checkpoint shapes.
|
||||
|
||||
- Unpacked: one broadcast per tensor.
|
||||
- Packed: concatenate `t.contiguous().view(-1).view(torch.uint8)` in `names`
|
||||
order; broadcast a chunk as soon as its size **exceeds**
|
||||
`packed_buffer_size_bytes` (the crossing tensor stays in the chunk), then the
|
||||
remainder.
|
||||
|
||||
Trainer-side communicators without vLLM:
|
||||
|
||||
| Communicator | Status |
|
||||
| ------------ | ------ |
|
||||
| `torch.distributed.ProcessGroupNCCL` on a private `TCPStore` + `eager_connect_single_device` | Works; the unique id is read from store key `"0"`, a torch implementation detail (used by the example) |
|
||||
| `ctypes` against torch's bundled `libnccl.so.2` | Works |
|
||||
| `torch.cuda.nccl.init_rank` | Fails on Python 3.10+ (`PY_SSIZE_T_CLEAN` `SystemError`) |
|
||||
|
||||
## Sparse NCCL
|
||||
|
||||
Sparse, flat-index weight patches use
|
||||
@@ -196,6 +241,7 @@ Sparse NCCL sends only `O(nnz)` indices and values. Checkpoint application still
|
||||
## Examples
|
||||
|
||||
- [RLHF with NCCL weight syncing (`vllm serve`, HTTP)](../../../examples/rl/rlhf_http_nccl.py) - **Start here.** Trainer on one GPU, 2x tensor-parallel fp8 server on two others; HTTP control plane, NCCL data plane. Launches and tears down its own server
|
||||
- [NCCL weight syncing from a trainer that does not import vLLM](../../../examples/rl/rlhf_http_nccl_no_vllm.py) - `torch` + `requests` only, `nccl_unique_id_b64` rendezvous; see [Trainer Without vLLM over HTTP](#trainer-without-vllm-over-http)
|
||||
- [RLHF with NCCL + FSDP2 and expert parallelism](../../../examples/rl/rlhf_nccl_fsdp_ep.py) - Multi-rank trainer: every FSDP rank builds an engine and joins the `full_tensor()` gather while only rank 0 touches the wire
|
||||
- [RLHF with sparse NCCL weight syncing (offline, Ray)](../../../examples/rl/rlhf_sparse_nccl.py) - Qwen3 MoE per-expert checkpoint updates with one trainer GPU and two TP2/EP2 inference GPUs
|
||||
- [RLHF with async weight syncing (offline, Ray)](../../../examples/rl/rlhf_async_new_apis.py) - Async generation with mid-flight pause, weight sync, resume, and validation against a fresh model; uses an in-process `AsyncLLMEngine` with `RayVLLMWeightSyncClient`
|
||||
|
||||
Reference in New Issue
Block a user