diff --git a/docs/features/disagg_encoder.md b/docs/features/disagg_encoder.md index ac304345fd..84debbacaa 100644 --- a/docs/features/disagg_encoder.md +++ b/docs/features/disagg_encoder.md @@ -45,9 +45,10 @@ The following scripts demonstrate the workflow with **ExampleConnector**: ### ECMooncakeConnector **ECMooncakeConnector** transfers encoder outputs using the Mooncake TransferEngine. -See the [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh) -for a complete 1 Encoder instance + 1 PD instance setup, including the producer and -consumer `--ec-transfer-config` settings and proxy configuration. +See the [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md) for +producer/consumer `--ec-transfer-config`, proxy `--ec-consumer-zmq-addrs`, and +transport options. The [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh) +is a complete 1 Encoder instance + 1 PD instance setup. With vLLM and Mooncake installed, run the example from the repository root: diff --git a/docs/features/disagg_prefill.md b/docs/features/disagg_prefill.md index e6a5797f26..013d57fa87 100644 --- a/docs/features/disagg_prefill.md +++ b/docs/features/disagg_prefill.md @@ -27,7 +27,7 @@ Now supports 9 types of connectors: --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_both", "kv_buffer_device":"cuda", "kv_connector_extra_config":{"backends":["UCX", "GDS"]}}' ``` -- **MooncakeConnector**: refer to [examples/disaggregated/mooncake_connector/run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh) for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md). +- **MooncakeConnector**: refer to [examples/disaggregated/mooncake_connector/run_mooncake_connector.sh](../../examples/disaggregated/mooncake_connector/run_mooncake_connector.sh) for the example usage of MooncakeConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md). For multimodal encoder-cache transfer, see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md). - **MoRIIOConnector** (ROCm only): see [MoRI-IO Usage Guide](moriio_connector_usage.md) for example usage and detailed documentation. - **MultiConnector**: take advantage of the kv_connector_extra_config: dict[str, Any] already present in KVTransferConfig to stash all the connectors we want in an ordered list of kwargs.such as: diff --git a/docs/features/mooncake_connector_usage.md b/docs/features/mooncake_connector_usage.md index 701a1d0049..866d6d6bd4 100644 --- a/docs/features/mooncake_connector_usage.md +++ b/docs/features/mooncake_connector_usage.md @@ -38,6 +38,9 @@ python examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py --p Now you can send requests to the proxy server through port 8000. +For multimodal Encode-Prefill-Decode (encoder embeddings rather than KV +cache), see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md). + ## Environment Variables - `VLLM_MOONCAKE_BOOTSTRAP_PORT`: Port for Mooncake bootstrap server diff --git a/docs/features/mooncake_ec_connector_usage.md b/docs/features/mooncake_ec_connector_usage.md new file mode 100644 index 0000000000..a710cbdfb6 --- /dev/null +++ b/docs/features/mooncake_ec_connector_usage.md @@ -0,0 +1,158 @@ +# ECMooncakeConnector Usage Guide + +`ECMooncakeConnector` moves multimodal **encoder embeddings** (encoder cache) +between a disaggregated encoder instance and a language-model instance over +[Mooncake Transfer Engine](https://github.com/kvcache-ai/Mooncake). Use it +when the vision encoder should scale independently of prefill and decode. + +For KV-cache transfer between prefiller and decoder, see +[MooncakeConnector Usage Guide](mooncake_connector_usage.md). That path uses +`MooncakeConnector`, not `ECMooncakeConnector`. + +Added in [PR #41567](https://github.com/vllm-project/vllm/pull/41567). +Overview of disaggregated encoding: +[Disaggregated Encoder](disagg_encoder.md). + +!!! important + Static proxy flags (`--ec-consumer-zmq-addrs`) support Mooncake EC only in + **1E + 1PD** mode (`--prefill-servers-urls disable`). For 1E + 1P + 1D, + register `ec_zmq_addrs` on the **prefill** consumer through the proxy's + dynamic registration API instead of the static CLI flag. See + `examples/disaggregated/disaggregated_encoder/README.md`. + +## Prerequisites + +Install Mooncake Transfer Engine: + +```bash +# CUDA 13 (vLLM default) +uv pip install mooncake-transfer-engine-cuda13 + +# CUDA 12 +uv pip install mooncake-transfer-engine +``` + +The two wheels are the same release built against different CUDA majors. +Importing the wrong one fails with +`libcudart.so.: cannot open shared object file`. + +## 1E + 1PD + +One encoder producer and one combined prefill/decode consumer. The consumer +listens on `ec_ip` + `ec_port` (ZMQ control). The proxy must advertise that +address with `--ec-consumer-zmq-addrs`. + +### Encoder (producer) + +The producer must run with `tensor_parallel_size=1`, +`pipeline_parallel_size=1`, and `data_parallel_size=1`. + +```bash +CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \ + --port 19534 \ + --gpu-memory-utilization 0.35 \ + --enable-request-id-headers \ + --no-enable-prefix-caching \ + --max-num-batched-tokens 16384 \ + --allowed-local-media-path /path/to/media \ + --ec-transfer-config '{ + "ec_connector": "ECMooncakeConnector", + "ec_role": "ec_producer", + "ec_connector_extra_config": { + "mooncake_protocol": "rdma" + } + }' +``` + +### Prefill+Decode (consumer) + +```bash +CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen2.5-VL-3B-Instruct \ + --port 19537 \ + --gpu-memory-utilization 0.75 \ + --enable-mm-embeds \ + --enable-request-id-headers \ + --no-enable-prefix-caching \ + --allowed-local-media-path /path/to/media \ + --ec-transfer-config '{ + "ec_connector": "ECMooncakeConnector", + "ec_role": "ec_consumer", + "ec_ip": "127.0.0.1", + "ec_port": 19019, + "ec_connector_extra_config": { + "mooncake_protocol": "rdma" + } + }' +``` + +`--enable-mm-embeds` lets the PD instance inject transferred embeddings +instead of re-running the vision encoder. + +### Proxy + +```bash +python examples/disaggregated/disaggregated_encoder/disagg_epd_proxy.py \ + --host 0.0.0.0 \ + --port 10002 \ + --encode-servers-urls http://127.0.0.1:19534 \ + --prefill-servers-urls disable \ + --decode-servers-urls http://127.0.0.1:19537 \ + --ec-consumer-zmq-addrs tcp://127.0.0.1:19019 +``` + +`--ec-consumer-zmq-addrs` must align with `--decode-servers-urls` (the +combined PD instances). For data-parallel consumers, list each replica +consecutively and set `--ec-consumer-dp-size`. + +### Correctness script + +```bash +# TCP (no RDMA device required; CI default) +MOONCAKE_EC_PROTOCOL=tcp \ + bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh + +# RDMA +MOONCAKE_EC_PROTOCOL=rdma \ + bash tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh +``` + +The TCP path sets `MC_FORCE_TCP=1` so Transfer Engine does not auto-select +RDMA when an HCA is present. + +## Configuration + +Producer and consumer must use the same `mooncake_protocol`. + +| Setting | Where | Default | Notes | +| --------- | ------- | --------- | ------- | +| `mooncake_protocol` | `ec_connector_extra_config` | `rdma` | Set `tcp` when no verbs device is available. With TCP, also export `MC_FORCE_TCP=1`. | +| `ec_ip` / `ec_port` | consumer `ECTransferConfig` | `127.0.0.1` / `14579` | First ZMQ control port. Tensor-parallel rank `r` listens on `ec_port + dp_index * tp_size + r`. | +| `ec_buffer_size` | `ECTransferConfig` | `1e9` | Registered receive arena on the consumer, in bytes. | +| `control_timeout_s` | `ec_connector_extra_config` | `30` | Reservation RPC timeout. | +| `push_wait_timeout_s` | `ec_connector_extra_config` | `60` | How long the consumer waits for a matching encoder push. | +| `WITH_NVIDIA_PEERMEM` | process environment | `1` | Read by Mooncake. Set `0` on hosts without `nvidia-peermem` (DMA-BUF path). | +| `--ec-consumer-zmq-addrs` | proxy | empty | `tcp://:` for each PD replica. Required for Mooncake EC. | + +## 1E + 1P + 1D + +The filesystem example is `examples/disaggregated/disaggregated_encoder/disagg_1e1p1d_example.sh` +(`ECExampleConnector` for encoder cache, `NixlConnector` for KV). P→D KV can +use `MooncakeConnector` instead; see +[MooncakeConnector Usage Guide](mooncake_connector_usage.md). + +Do **not** pass `--ec-consumer-zmq-addrs` together with independent prefill +URLs: the static proxy flag is E+PD only. For Mooncake EC with a dedicated +prefill, use dynamic registration and set `ec_zmq_addrs` on the prefill +consumer as described in +`examples/disaggregated/disaggregated_encoder/README.md`. + +## Troubleshooting + +| Symptom | Likely cause | +| --------- | ---------------- | +| `Install mooncake-transfer-engine` / `libcudart.so.*` | Wrong or missing Mooncake wheel for this CUDA major. | +| `Failed to register memory ... Bad address [14]` | GPUDirect registration failed. Set `WITH_NVIDIA_PEERMEM=0`. | +| Proxy startup error about E+PD only | `--ec-consumer-zmq-addrs` used with a real `--prefill-servers-urls`. | +| Consumer waits then errors; encoder never finishes the push | `--ec-consumer-zmq-addrs` does not match consumer `ec_ip`/`ec_port`, or producer/consumer protocols differ. | +| TCP transfers stall on a host that also has RDMA | Export `MC_FORCE_TCP=1`. | +| Producer rejected at startup | Encoder used tensor, pipeline, or data parallelism. Keep the producer unsharded. | diff --git a/examples/disaggregated/disaggregated_encoder/README.md b/examples/disaggregated/disaggregated_encoder/README.md index ec10f23fc0..b2679d6cd1 100644 --- a/examples/disaggregated/disaggregated_encoder/README.md +++ b/examples/disaggregated/disaggregated_encoder/README.md @@ -57,7 +57,12 @@ The vllm instances and `disagg_encoder_proxy` supports local URIs with ```{"url" ## EC connector and KV transfer -The `ECExampleonnector` is used to store the encoder cache on local disk and facilitate transfer. To enable the encoder disaggregation feature, add the following configuration: +`ECExampleConnector` stores encoder cache on local disk. For RDMA/TCP transfer +without a shared filesystem, use `ECMooncakeConnector` (1E + 1PD only through +this proxy). See +[ECMooncakeConnector Usage Guide](../../../docs/features/mooncake_ec_connector_usage.md). + +To enable filesystem encoder disaggregation, add: ```bash # Add to encoder instance: @@ -104,6 +109,8 @@ If you enable prefill instance (`--prefill-servers-urls` not disabled), you will | `--encode-servers-urls` | Comma-separated list of encoder endpoints. Every multimodal item extracted from the request is fanned out to one of these URLs in a round-robin fashion. | | `--prefill-servers-urls` | Comma-separated list of prefill endpoints. Set to `disable`, `none`, or `""` to skip the dedicated prefill phase and run E+PD (encoder + combined prefill/decode). | | `--decode-servers-urls` | Comma-separated list of decode endpoints. Non-stream and stream paths both round-robin over this list. | +| `--ec-consumer-zmq-addrs` | Mooncake EC consumer control addresses, aligned with `--decode-servers-urls`. Required for `ECMooncakeConnector`. Supported only in E+PD mode (`--prefill-servers-urls disable`). | +| `--ec-consumer-dp-size` | Data-parallel replicas per EC consumer (default 1). | | `--host`, `--port` | Bind address for the proxy itself (defaults: `0.0.0.0:8000`). | ### Dynamic registration