[Doc] Document ECMooncakeConnector for EPD disaggregation (#59196)
Signed-off-by: Teng Ma <stmatengss@gmail.com> Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
This commit is contained in:
@@ -45,9 +45,10 @@ The following scripts demonstrate the workflow with **ExampleConnector**:
|
||||
### ECMooncakeConnector
|
||||
|
||||
**ECMooncakeConnector** transfers encoder outputs using the Mooncake TransferEngine.
|
||||
See the [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh)
|
||||
for a complete 1 Encoder instance + 1 PD instance setup, including the producer and
|
||||
consumer `--ec-transfer-config` settings and proxy configuration.
|
||||
See the [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md) for
|
||||
producer/consumer `--ec-transfer-config`, proxy `--ec-consumer-zmq-addrs`, and
|
||||
transport options. The [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh)
|
||||
is a complete 1 Encoder instance + 1 PD instance setup.
|
||||
|
||||
With vLLM and Mooncake installed, run the example from the repository root:
|
||||
|
||||
|
||||
@@ -27,7 +27,7 @@ Now supports 9 types of connectors:
|
||||
--kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both", "kv_buffer_device":"cuda", "kv_connector_extra_config":{"backends":["UCX", "GDS"]}}'
|
||||
```
|
||||
|
||||
- **MooncakeConnector**: refer to [examples/disaggregated/mooncake_connector/run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh) for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md).
|
||||
- **MooncakeConnector**: refer to [examples/disaggregated/mooncake_connector/run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh) for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md). For multimodal encoder-cache transfer, see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
|
||||
- **MoRIIOConnector** (ROCm only): see [MoRI-IO Usage Guide](moriio_connector_usage.md) for example usage and detailed documentation.
|
||||
- **MultiConnector**: take advantage of the kv_connector_extra_config: dict[str, Any] already present in KVTransferConfig to stash all the connectors we want in an ordered list of kwargs.such as:
|
||||
|
||||
|
||||
@@ -38,6 +38,9 @@ python examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py --p
|
||||
|
||||
Now you can send requests to the proxy server through port 8000.
|
||||
|
||||
For multimodal Encode-Prefill-Decode (encoder embeddings rather than KV
|
||||
cache), see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
|
||||
|
||||
## Environment Variables
|
||||
|
||||
- `VLLM_MOONCAKE_BOOTSTRAP_PORT`: Port for Mooncake bootstrap server
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
# ECMooncakeConnector Usage Guide
|
||||
|
||||
`ECMooncakeConnector` moves multimodal **encoder embeddings** (encoder cache)
|
||||
between a disaggregated encoder instance and a language-model instance over
|
||||
[Mooncake Transfer Engine](https://github.com/kvcache-ai/Mooncake). Use it
|
||||
when the vision encoder should scale independently of prefill and decode.
|
||||
|
||||
For KV-cache transfer between prefiller and decoder, see
|
||||
[MooncakeConnector Usage Guide](mooncake_connector_usage.md). That path uses
|
||||
`MooncakeConnector`, not `ECMooncakeConnector`.
|
||||
|
||||
Added in [PR #41567](https://github.com/vllm-project/vllm/pull/41567).
|
||||
Overview of disaggregated encoding:
|
||||
[Disaggregated Encoder](disagg_encoder.md).
|
||||
|
||||
!!! important
|
||||
Static proxy flags (`--ec-consumer-zmq-addrs`) support Mooncake EC only in
|
||||
**1E + 1PD** mode (`--prefill-servers-urls disable`). For 1E + 1P + 1D,
|
||||
register `ec_zmq_addrs` on the **prefill** consumer through the proxy's
|
||||
dynamic registration API instead of the static CLI flag. See
|
||||
`examples/disaggregated/disaggregated_encoder/README.md`.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install Mooncake Transfer Engine:
|
||||
|
||||
```bash
|
||||
# CUDA 13 (vLLM default)
|
||||
uv pip install mooncake-transfer-engine-cuda13
|
||||
|
||||
# CUDA 12
|
||||
uv pip install mooncake-transfer-engine
|
||||
```
|
||||
|
||||
The two wheels are the same release built against different CUDA majors.
|
||||
Importing the wrong one fails with
|
||||
`libcudart.so.<major>: cannot open shared object file`.
|
||||
|
||||
## 1E + 1PD
|
||||
|
||||
One encoder producer and one combined prefill/decode consumer. The consumer
|
||||
listens on `ec_ip` + `ec_port` (ZMQ control). The proxy must advertise that
|
||||
address with `--ec-consumer-zmq-addrs`.
|
||||
|
||||
### Encoder (producer)
|
||||
|
||||
The producer must run with `tensor_parallel_size=1`,
|
||||
`pipeline_parallel_size=1`, and `data_parallel_size=1`.
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
|
||||
--port 19534 \
|
||||
--gpu-memory-utilization 0.35 \
|
||||
--enable-request-id-headers \
|
||||
--no-enable-prefix-caching \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--allowed-local-media-path /path/to/media \
|
||||
--ec-transfer-config '{
|
||||
"ec_connector": "ECMooncakeConnector",
|
||||
"ec_role": "ec_producer",
|
||||
"ec_connector_extra_config": {
|
||||
"mooncake_protocol": "rdma"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### Prefill+Decode (consumer)
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \
|
||||
--port 19537 \
|
||||
--gpu-memory-utilization 0.75 \
|
||||
--enable-mm-embeds \
|
||||
--enable-request-id-headers \
|
||||
--no-enable-prefix-caching \
|
||||
--allowed-local-media-path /path/to/media \
|
||||
--ec-transfer-config '{
|
||||
"ec_connector": "ECMooncakeConnector",
|
||||
"ec_role": "ec_consumer",
|
||||
"ec_ip": "127.0.0.1",
|
||||
"ec_port": 19019,
|
||||
"ec_connector_extra_config": {
|
||||
"mooncake_protocol": "rdma"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
`--enable-mm-embeds` lets the PD instance inject transferred embeddings
|
||||
instead of re-running the vision encoder.
|
||||
|
||||
### Proxy
|
||||
|
||||
```bash
|
||||
python examples/disaggregated/disaggregated_encoder/disagg_epd_proxy.py \
|
||||
--host 0.0.0.0 \
|
||||
--port 10002 \
|
||||
--encode-servers-urls http://127.0.0.1:19534 \
|
||||
--prefill-servers-urls disable \
|
||||
--decode-servers-urls http://127.0.0.1:19537 \
|
||||
--ec-consumer-zmq-addrs tcp://127.0.0.1:19019
|
||||
```
|
||||
|
||||
`--ec-consumer-zmq-addrs` must align with `--decode-servers-urls` (the
|
||||
combined PD instances). For data-parallel consumers, list each replica
|
||||
consecutively and set `--ec-consumer-dp-size`.
|
||||
|
||||
### Correctness script
|
||||
|
||||
```bash
|
||||
# TCP (no RDMA device required; CI default)
|
||||
MOONCAKE_EC_PROTOCOL=tcp \
|
||||
bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh
|
||||
|
||||
# RDMA
|
||||
MOONCAKE_EC_PROTOCOL=rdma \
|
||||
bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh
|
||||
```
|
||||
|
||||
The TCP path sets `MC_FORCE_TCP=1` so Transfer Engine does not auto-select
|
||||
RDMA when an HCA is present.
|
||||
|
||||
## Configuration
|
||||
|
||||
Producer and consumer must use the same `mooncake_protocol`.
|
||||
|
||||
| Setting | Where | Default | Notes |
|
||||
| --------- | ------- | --------- | ------- |
|
||||
| `mooncake_protocol` | `ec_connector_extra_config` | `rdma` | Set `tcp` when no verbs device is available. With TCP, also export `MC_FORCE_TCP=1`. |
|
||||
| `ec_ip` / `ec_port` | consumer `ECTransferConfig` | `127.0.0.1` / `14579` | First ZMQ control port. Tensor-parallel rank `r` listens on `ec_port + dp_index * tp_size + r`. |
|
||||
| `ec_buffer_size` | `ECTransferConfig` | `1e9` | Registered receive arena on the consumer, in bytes. |
|
||||
| `control_timeout_s` | `ec_connector_extra_config` | `30` | Reservation RPC timeout. |
|
||||
| `push_wait_timeout_s` | `ec_connector_extra_config` | `60` | How long the consumer waits for a matching encoder push. |
|
||||
| `WITH_NVIDIA_PEERMEM` | process environment | `1` | Read by Mooncake. Set `0` on hosts without `nvidia-peermem` (DMA-BUF path). |
|
||||
| `--ec-consumer-zmq-addrs` | proxy | empty | `tcp://<ec_ip>:<ec_port>` for each PD replica. Required for Mooncake EC. |
|
||||
|
||||
## 1E + 1P + 1D
|
||||
|
||||
The filesystem example is `examples/disaggregated/disaggregated_encoder/disagg_1e1p1d_example.sh`
|
||||
(`ECExampleConnector` for encoder cache, `NixlConnector` for KV). P→D KV can
|
||||
use `MooncakeConnector` instead; see
|
||||
[MooncakeConnector Usage Guide](mooncake_connector_usage.md).
|
||||
|
||||
Do **not** pass `--ec-consumer-zmq-addrs` together with independent prefill
|
||||
URLs: the static proxy flag is E+PD only. For Mooncake EC with a dedicated
|
||||
prefill, use dynamic registration and set `ec_zmq_addrs` on the prefill
|
||||
consumer as described in
|
||||
`examples/disaggregated/disaggregated_encoder/README.md`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely cause |
|
||||
| --------- | ---------------- |
|
||||
| `Install mooncake-transfer-engine` / `libcudart.so.*` | Wrong or missing Mooncake wheel for this CUDA major. |
|
||||
| `Failed to register memory ... Bad address [14]` | GPUDirect registration failed. Set `WITH_NVIDIA_PEERMEM=0`. |
|
||||
| Proxy startup error about E+PD only | `--ec-consumer-zmq-addrs` used with a real `--prefill-servers-urls`. |
|
||||
| Consumer waits then errors; encoder never finishes the push | `--ec-consumer-zmq-addrs` does not match consumer `ec_ip`/`ec_port`, or producer/consumer protocols differ. |
|
||||
| TCP transfers stall on a host that also has RDMA | Export `MC_FORCE_TCP=1`. |
|
||||
| Producer rejected at startup | Encoder used tensor, pipeline, or data parallelism. Keep the producer unsharded. |
|
||||
@@ -57,7 +57,12 @@ The vllm instances and `disagg_encoder_proxy` supports local URIs with ```{"url"
|
||||
|
||||
## EC connector and KV transfer
|
||||
|
||||
The `ECExampleonnector` is used to store the encoder cache on local disk and facilitate transfer. To enable the encoder disaggregation feature, add the following configuration:
|
||||
`ECExampleConnector` stores encoder cache on local disk. For RDMA/TCP transfer
|
||||
without a shared filesystem, use `ECMooncakeConnector` (1E + 1PD only through
|
||||
this proxy). See
|
||||
[ECMooncakeConnector Usage Guide](../../../docs/features/mooncake_ec_connector_usage.md).
|
||||
|
||||
To enable filesystem encoder disaggregation, add:
|
||||
|
||||
```bash
|
||||
# Add to encoder instance:
|
||||
@@ -104,6 +109,8 @@ If you enable prefill instance (`--prefill-servers-urls` not disabled), you will
|
||||
| `--encode-servers-urls` | Comma-separated list of encoder endpoints. Every multimodal item extracted from the request is fanned out to one of these URLs in a round-robin fashion. |
|
||||
| `--prefill-servers-urls` | Comma-separated list of prefill endpoints. Set to `disable`, `none`, or `""` to skip the dedicated prefill phase and run E+PD (encoder + combined prefill/decode). |
|
||||
| `--decode-servers-urls` | Comma-separated list of decode endpoints. Non-stream and stream paths both round-robin over this list. |
|
||||
| `--ec-consumer-zmq-addrs` | Mooncake EC consumer control addresses, aligned with `--decode-servers-urls`. Required for `ECMooncakeConnector`. Supported only in E+PD mode (`--prefill-servers-urls disable`). |
|
||||
| `--ec-consumer-dp-size` | Data-parallel replicas per EC consumer (default 1). |
|
||||
| `--host`, `--port` | Bind address for the proxy itself (defaults: `0.0.0.0:8000`). |
|
||||
|
||||
### Dynamic registration
|
||||
|
||||
Reference in New Issue
Block a user