[Feature][EPD] Support dynamic EPD (EC-connector) proxy (#54176)

Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
This commit is contained in:
Tianyu Guo
2026-09-20 12:24:41 +08:00
committed by GitHub
co-authored by OpenAI Codex OpenAI Codex
parent 1596fa5f88
commit 70fc7bd695
8 changed files with 1168 additions and 72 deletions
@@ -106,6 +106,82 @@ If you enable prefill instance (`--prefill-servers-urls` not disabled), you will
| `--decode-servers-urls` | Comma-separated list of decode endpoints. Non-stream and stream paths both round-robin over this list. |
| `--host`, `--port` | Bind address for the proxy itself (defaults: `0.0.0.0:8000`). |
### Dynamic registration
Alternatively, let the external launcher register ready instances over HTTP.
No vLLM configuration changes or worker registration threads are required.
Set `ADMIN_API_KEY` on the proxy and supply it as `X-API-Key` for registration
and removal. These are trusted control-plane APIs; do not expose them publicly.
```bash
export ADMIN_API_KEY="your-admin-key"
python disagg_epd_proxy.py --port 8000 --dynamic-registration
```
Omit the static server URL flags and register every stage. The roles determine
the topology: `encode` + `prefill_decode` gives E+PD; `encode` + `prefill` +
`decode` gives E+P+D. Standalone `decode` always requires an available `prefill`,
even when D registers first or all P instances go offline; otherwise requests
return `503`. The proxy rejects mixing combined PD with standalone P/D,
including unhealthy instances still in the registry. Explicitly remove the
old topology's P/D or PD registrations before switching topologies.
After each instance is ready, the launcher registers its reachable HTTP URL:
```bash
curl --fail-with-body http://proxy-host:8000/instances \
-H "X-API-Key: $ADMIN_API_KEY" -H 'Content-Type: application/json' \
-d '{"role":"encode","url":"http://e-host:8001"}'
curl --fail-with-body http://proxy-host:8000/instances \
-H "X-API-Key: $ADMIN_API_KEY" -H 'Content-Type: application/json' \
-d '{"role":"prefill_decode","url":"http://pd-host:8002"}'
```
For E+P+D, register P with `role: "prefill"` and D with `role: "decode"`.
Example and NIXL EC connectors need no additional registration fields.
For Mooncake, register `ec_zmq_addrs` on the **EC consumer** (`prefill_decode`
or `prefill`), using its configured `ec_ip` and `ec_port`. Keep the fixed-port
layout: supply the TP-rank-0 address for each DP replica, in DP-rank order;
replica `r` uses `ec_port + r * tensor_parallel_size`. The connector discovers
the remaining TP ranks itself. For example, DP=2, TP=2, `ec_port=19019`:
```json
{
"role": "prefill_decode",
"url": "http://pd-host:8002",
"dp_size": 2,
"ec_zmq_addrs": ["tcp://pd-host:19019", "tcp://pd-host:19021"]
}
```
The proxy selects one consumer replica and uses it for both the encoder push
and the consumer HTTP request. Standalone D does not need EC control addresses.
Port allocation and avoiding collisions remain the launcher's responsibility.
Inspect or remove instances without restarting the proxy:
```bash
curl http://proxy-host:8000/instances
curl --fail-with-body -X DELETE \
'http://proxy-host:8000/instances?url=http://e-host:8001' \
-H "X-API-Key: $ADMIN_API_KEY"
```
Registration is idempotent. The proxy probes registered instances every
`--probe-interval` seconds (default 5), with `--probe-timeout` seconds per probe
(default 2). After `--fail-threshold` consecutive failures (default 3), it stops
sending new requests to that instance. Healthy instances rejoin automatically;
unreachable ones are forgotten after `--evicted-ttl` seconds (default 900;
`0` retains them indefinitely). Removal stops new routing; already routed
requests retain their selected endpoints, so drain requests before stopping
the instance. The launcher must re-register instances after a proxy restart.
The example launch scripts also support this flow: export `ADMIN_API_KEY` and
set `DYNAMIC_REGISTRATION=1` when running `disagg_1e1pd_example.sh` or
`disagg_1e1p1d_example.sh`. Their default remains static routing.
### Static configuration
The proxy batches images from the same user request assigned to the same encoder.
Set the proxy environment variable `ENCODER_MAX_BATCH_SIZE` to limit the number
of images per encoder subrequest. It defaults to `0` (unlimited); `1` sends each
@@ -137,6 +213,6 @@ For E + P + D setup:
```bash
$ python disagg_encoder_proxy.py \
--encode-servers-urls "http://e1:8001,http://e2:8001" \
--prefill-servers-urls "http://p1:8003,http://p2:8004" \
--prefill-servers-urls "http://p1:8003,http://p2:8004" \
--decode-servers-urls "http://d1:8005,http://d2:8006"
```
@@ -189,17 +189,34 @@ wait_for_server "$DECODE_PORT"
###############################################################################
# Proxy
###############################################################################
PROXY_ARGS=()
if [[ "${DYNAMIC_REGISTRATION:-0}" == "1" ]]; then
: "${ADMIN_API_KEY:?Set ADMIN_API_KEY for dynamic registration}"
export ADMIN_API_KEY
PROXY_ARGS=(--dynamic-registration)
else
PROXY_ARGS=(
--encode-servers-urls "http://localhost:$ENCODE_PORT"
--prefill-servers-urls "http://localhost:$PREFILL_PORT"
--decode-servers-urls "http://localhost:$DECODE_PORT"
)
fi
python disagg_epd_proxy.py \
--host "0.0.0.0" \
--port "$PROXY_PORT" \
--encode-servers-urls "http://localhost:$ENCODE_PORT" \
--prefill-servers-urls "http://localhost:$PREFILL_PORT" \
--decode-servers-urls "http://localhost:$DECODE_PORT" \
"${PROXY_ARGS[@]}" \
>"${PROXY_LOG}" 2>&1 &
PIDS+=($!)
wait_for_server "$PROXY_PORT"
if [[ "${DYNAMIC_REGISTRATION:-0}" == "1" ]]; then
for endpoint in "encode:$ENCODE_PORT" "prefill:$PREFILL_PORT" "decode:$DECODE_PORT"; do
curl --fail-with-body "http://127.0.0.1:$PROXY_PORT/instances" \
-H "X-API-Key: $ADMIN_API_KEY" -H 'Content-Type: application/json' \
-d "{\"role\":\"${endpoint%%:*}\",\"url\":\"http://127.0.0.1:${endpoint#*:}\"}"
done
fi
echo "All services are up!"
###############################################################################
@@ -238,4 +255,4 @@ curl http://127.0.0.1:"${PROXY_PORT}"/v1/chat/completions \
# cleanup
echo "cleanup..."
cleanup
cleanup
@@ -152,17 +152,33 @@ wait_for_server "$PREFILL_DECODE_PORT"
###############################################################################
# Proxy
###############################################################################
PROXY_ARGS=()
if [[ "${DYNAMIC_REGISTRATION:-0}" == "1" ]]; then
: "${ADMIN_API_KEY:?Set ADMIN_API_KEY for dynamic registration}"
export ADMIN_API_KEY
PROXY_ARGS=(--dynamic-registration)
else
PROXY_ARGS=(
--encode-servers-urls "http://localhost:$ENCODE_PORT"
--decode-servers-urls "http://localhost:$PREFILL_DECODE_PORT"
)
fi
python disagg_epd_proxy.py \
--host "0.0.0.0" \
--port "$PROXY_PORT" \
--encode-servers-urls "http://localhost:$ENCODE_PORT" \
--prefill-servers-urls "disable" \
--decode-servers-urls "http://localhost:$PREFILL_DECODE_PORT" \
"${PROXY_ARGS[@]}" \
>"${PROXY_LOG}" 2>&1 &
PIDS+=($!)
wait_for_server "$PROXY_PORT"
if [[ "${DYNAMIC_REGISTRATION:-0}" == "1" ]]; then
for endpoint in "encode:$ENCODE_PORT" "prefill_decode:$PREFILL_DECODE_PORT"; do
curl --fail-with-body "http://127.0.0.1:$PROXY_PORT/instances" \
-H "X-API-Key: $ADMIN_API_KEY" -H 'Content-Type: application/json' \
-d "{\"role\":\"${endpoint%%:*}\",\"url\":\"http://127.0.0.1:${endpoint#*:}\"}"
done
fi
echo "All services are up!"
###############################################################################
@@ -201,4 +217,4 @@ curl http://127.0.0.1:"${PROXY_PORT}"/v1/chat/completions \
# cleanup
echo "cleanup..."
cleanup
cleanup
@@ -1,12 +1,13 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""disagg_encoder_proxy.py
"""OpenAI-compatible proxy for E+PD and E+P+D disaggregation.
Proxy that routes OpenAI-compatible “/v1/chat/completions” requests to two
clusters:
• encode (multimodal feature extraction)
• decode (language-model inference)
Use static server URLs, or --dynamic-registration for launcher-managed HTTP
registration of E, P/PD and D instances. Both modes route OpenAI-compatible
``/v1/chat/completions`` requests through encoder and inference clusters and
share encoder fan-out, metadata-only rewriting, prefill, and retry/streaming
forwarding.
For MM input we:
1. Extract *every* image/audio/video item.
@@ -20,22 +21,27 @@ from __future__ import annotations
import argparse
import asyncio
import enum
import hashlib
import itertools
import json
import logging
import os
import random
import secrets
import time
import uuid
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager, suppress
from dataclasses import dataclass, field
from typing import Any
import aiohttp
import msgspec
import uvicorn
from fastapi import FastAPI, HTTPException, Request
from fastapi import Depends, FastAPI, Header, HTTPException, Request
from fastapi.responses import JSONResponse, Response, StreamingResponse
from pydantic import AnyHttpUrl, BaseModel, Field, model_validator
###############################################################################
# FastAPI app & global state
@@ -44,6 +50,340 @@ from fastapi.responses import JSONResponse, Response, StreamingResponse
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s: %(message)s")
logger = logging.getLogger("proxy")
DEFAULT_PROBE_INTERVAL = 5.0
DEFAULT_PROBE_TIMEOUT = 2.0
DEFAULT_FAIL_THRESHOLD = 3
# Stop probing an instance that has been down this long. 0 probes forever,
# which is what a cluster that restarts instances in place wants.
DEFAULT_EVICTED_TTL = 900.0
class InstanceRole(str, enum.Enum):
ENCODE = "encode"
PREFILL = "prefill"
DECODE = "decode"
PREFILL_DECODE = "prefill_decode"
@dataclass
class InstanceRecord:
"""One registered instance.
Attributes:
role: Which stage this instance serves.
url: Base OpenAI-compatible URL, e.g. ``http://host:8000``.
ec_zmq_addrs: Mooncake TP-rank-0 control addresses, one per DP replica,
supplied by the launcher from the consumer's fixed port config.
dp_size: Data-parallel replicas behind `url`, so the proxy can pick
a replica and name the same one to both halves of a request.
"""
role: InstanceRole
url: str
ec_zmq_addrs: list[str] = field(default_factory=list)
dp_size: int = 1
class InstanceRegistry:
def __init__(
self,
probe_interval: float = DEFAULT_PROBE_INTERVAL,
probe_timeout: float = DEFAULT_PROBE_TIMEOUT,
fail_threshold: int = DEFAULT_FAIL_THRESHOLD,
evicted_ttl: float = DEFAULT_EVICTED_TTL,
):
self._probe_interval = probe_interval
self._probe_timeout = probe_timeout
self._fail_threshold = fail_threshold
self._evicted_ttl = evicted_ttl
self._live: dict[str, InstanceRecord] = {}
self._evicted: dict[str, InstanceRecord] = {}
self._evicted_since: dict[str, float] = {}
self._fail_counts: dict[str, int] = {}
# One cursor per role, only ever incremented. Rebuilding it whenever
# the roster changes -- what an `itertools.cycle` over a mutable list
# forces -- restarts every fan-out at the first instance and hot-spots
# it after each registration.
self._cursors: dict[InstanceRole, int] = {role: 0 for role in InstanceRole}
self._replica_cursors: dict[str, int] = {}
self._probe_task: asyncio.Task | None = None
def register(self, record: InstanceRecord) -> bool:
"""Add or refresh an instance without overriding its health status."""
roles = {
other.role
for other in itertools.chain(self._live.values(), self._evicted.values())
}
split_roles = {InstanceRole.PREFILL, InstanceRole.DECODE}
if (record.role is InstanceRole.PREFILL_DECODE and roles & split_roles) or (
record.role in split_roles and InstanceRole.PREFILL_DECODE in roles
):
raise ValueError("Cannot mix prefill_decode with standalone prefill/decode")
key = record.url
previous = self._live.get(key) or self._evicted.get(key)
if previous is not None:
if previous == record:
return False
if previous.role is not record.role:
raise ValueError("Unregister the instance before changing its role")
target = self._live if key in self._live else self._evicted
target[key] = record
return False
self._live[key] = record
logger.info("Registered instance %s: %s", record.role.value, record.url)
return True
def unregister(self, url: str) -> bool:
"""Drop an instance for good, so a probe cannot bring it back."""
key = url
if key not in self._live and key not in self._evicted:
return False
self._live.pop(key, None)
self._evicted.pop(key, None)
self._evicted_since.pop(key, None)
self._fail_counts.pop(key, None)
self._replica_cursors.pop(key, None)
logger.info("Unregistered instance: %s", url)
return True
def instances(self, role: InstanceRole) -> list[InstanceRecord]:
return [record for record in self._live.values() if record.role is role]
def urls(self, role: InstanceRole) -> list[str]:
return [record.url for record in self.instances(role)]
def pick(self, role: InstanceRole) -> InstanceRecord | None:
"""Take the next instance of `role` in round-robin order."""
picked = self.pick_many(role, 1)
return picked[0] if picked else None
def pick_many(self, role: InstanceRole, count: int) -> list[InstanceRecord]:
"""Take `count` instances, continuing the rotation across calls.
A multimodal request fans out one encoder request per item, so the
assignment has to be contiguous with the previous request's rather
than restart at the first instance every time.
"""
alive = self.instances(role)
if not alive or count <= 0:
return []
start = self._cursors[role]
self._cursors[role] = start + count
return [alive[(start + offset) % len(alive)] for offset in range(count)]
def next_replica(self, record: InstanceRecord) -> int:
"""Take the next data-parallel replica of `record`, round-robin.
The encoder pushes to one replica's receive channel, so the request
has to run on that same replica; the caller names the rank to both
halves.
"""
if record.dp_size <= 1:
return 0
cursor = self._replica_cursors.get(record.url, 0)
self._replica_cursors[record.url] = cursor + 1
return cursor % record.dp_size
def status(self) -> dict[str, Any]:
return {
role.value: {
"live": [record.url for record in self.instances(role)],
"evicted": [
record.url
for record in self._evicted.values()
if record.role is role
],
}
for role in InstanceRole
}
def start_probing(self) -> None:
if self._probe_task is None and self._probe_interval > 0:
self._probe_task = asyncio.create_task(self._probe_loop())
async def stop_probing(self) -> None:
if self._probe_task is None:
return
self._probe_task.cancel()
with suppress(asyncio.CancelledError):
await self._probe_task
self._probe_task = None
async def _probe_loop(self) -> None:
timeout = aiohttp.ClientTimeout(total=self._probe_timeout)
async with aiohttp.ClientSession(timeout=timeout) as session:
while True:
await asyncio.sleep(self._probe_interval)
try:
await self._probe_once(session)
except Exception:
logger.exception("EPD registry probe round failed")
async def _probe_once(self, session: aiohttp.ClientSession) -> None:
targets = list(self._live.values()) + list(self._evicted.values())
if not targets:
return
results = await asyncio.gather(
*(self._probe(session, record.url) for record in targets),
return_exceptions=True,
)
now = time.monotonic()
for record, healthy in zip(targets, results):
current = self._live.get(record.url) or self._evicted.get(record.url)
if current is not record:
continue
if healthy is True:
self._on_probe_success(record)
else:
self._on_probe_failure(record, now)
self._drop_expired(now)
async def _probe(self, session: aiohttp.ClientSession, url: str) -> bool:
async with session.get(f"{url}/health") as resp:
return resp.status == 200
def _on_probe_success(self, record: InstanceRecord) -> None:
key = record.url
self._fail_counts.pop(key, None)
if key in self._evicted:
self._evicted.pop(key, None)
self._evicted_since.pop(key, None)
self._live[key] = record
logger.info(
"Instance %s (%s) is healthy again; routing resumed",
record.url,
record.role.value,
)
def _on_probe_failure(self, record: InstanceRecord, now: float) -> None:
key = record.url
if key not in self._live:
# Either already evicted, or unregistered while this probe was in
# flight. A round snapshots its targets and then awaits, so a
# removal inside that window would otherwise be undone by a result
# describing a registry that no longer exists.
return
failures = self._fail_counts.get(key, 0) + 1
self._fail_counts[key] = failures
if failures < self._fail_threshold:
return
self._live.pop(key, None)
self._evicted[key] = record
self._evicted_since[key] = now
logger.warning(
"Instance %s (%s) failed %d consecutive probes; stopped routing "
"to it. It rejoins on its own once it responds again.",
record.url,
record.role.value,
failures,
)
def _drop_expired(self, now: float) -> None:
if self._evicted_ttl <= 0:
return
for key, since in list(self._evicted_since.items()):
if now - since < self._evicted_ttl:
continue
record = self._evicted.pop(key, None)
self._evicted_since.pop(key, None)
self._fail_counts.pop(key, None)
self._replica_cursors.pop(key, None)
if record is not None:
logger.warning("Instance %s stayed down; forgetting it", record.url)
class InstanceRegistration(BaseModel):
role: InstanceRole
url: AnyHttpUrl
ec_zmq_addrs: list[str] = Field(default_factory=list)
dp_size: int = Field(default=1, ge=1)
@model_validator(mode="after")
def validate_consumer_addresses(self):
if self.ec_zmq_addrs and len(self.ec_zmq_addrs) != self.dp_size:
raise ValueError("Provide one Mooncake control address per DP replica")
return self
def require_admin_key(x_api_key: str = Header(default="")) -> None:
expected = os.getenv("ADMIN_API_KEY", "")
if not expected or not secrets.compare_digest(x_api_key, expected):
raise HTTPException(403, "Invalid admin API key")
@dataclass
class EPDProxyConfig:
probe_interval: float = 5.0
probe_timeout: float = 2.0
fail_threshold: int = 3
evicted_ttl: float = 900.0
@dataclass
class _Route:
"""The instances one request was assigned to.
Attributes:
consumer: The stage that receives the embedding, if any stage does.
consumer_zmq: The receive address named to the encoders.
dp_rank: Which replica of `consumer` was named, so the request can be
pinned to it.
"""
encoder_urls: list[str]
prefill: InstanceRecord | None
decode: InstanceRecord
consumer: InstanceRecord | None = None
consumer_zmq: str | None = None
dp_rank: int | None = None
class EPDProxy:
def __init__(self, registry: InstanceRegistry):
self.registry = registry
# ---------------------------------------------------------------- #
# Routing #
# ---------------------------------------------------------------- #
def route(self, num_items: int) -> _Route:
decode = self.registry.pick(InstanceRole.PREFILL_DECODE) or self.registry.pick(
InstanceRole.DECODE
)
if decode is None:
raise HTTPException(
status_code=503, detail="No decode instance is registered"
)
prefill = (
self.registry.pick(InstanceRole.PREFILL)
if decode.role is InstanceRole.DECODE
else None
)
if decode.role is InstanceRole.DECODE and prefill is None:
raise HTTPException(
status_code=503, detail="No prefill instance is registered"
)
encoders = self.registry.urls(InstanceRole.ENCODE)
if num_items and not encoders:
raise HTTPException(
status_code=503, detail="No encode instance is registered"
)
route = _Route(encoder_urls=encoders, prefill=prefill, decode=decode)
self._name_consumer(route)
return route
def _name_consumer(self, route: _Route) -> None:
"""Pin the EC consumer replica and, for push connectors, its endpoint."""
candidate = route.prefill or route.decode
route.consumer = candidate
rank = self.registry.next_replica(candidate)
route.dp_rank = rank if candidate.dp_size > 1 else None
if candidate.ec_zmq_addrs:
route.consumer_zmq = candidate.ec_zmq_addrs[rank]
app = FastAPI()
encode_session: aiohttp.ClientSession | None = None
prefill_session: aiohttp.ClientSession | None = None
@@ -448,6 +788,7 @@ async def maybe_prefill(
req_data: dict,
p_url: str,
req_id: str,
dp_rank: int | None = None,
) -> dict:
"""- Do prefill-only task if p_url exist;
- Return a new body carrying kv transfer params (for nixl connector)
@@ -459,7 +800,7 @@ async def maybe_prefill(
if p_url:
logger.info("[%s] Processing through prefill: %s", req_id, p_url)
prefill_response = await process_prefill_stage(req_data, p_url, req_id)
prefill_response = await process_prefill_stage(req_data, p_url, req_id, dp_rank)
# for nixl connector to facilitate kv transfer...
prefill_response_json = msgspec.json.decode(await prefill_response.read())
kv_transfer_params = prefill_response_json.get("kv_transfer_params", {})
@@ -473,6 +814,7 @@ async def process_prefill_stage(
req_data: dict,
p_url: str,
req_id: str,
dp_rank: int | None = None,
) -> dict:
"""Process request through Prefill stage and return kv_transfer_params"""
logger.info("[%s] Sending prefill request to: %s", req_id, p_url)
@@ -494,6 +836,8 @@ async def process_prefill_stage(
del prefill_request["stream_options"]
headers = {"x-request-id": req_id, "Content-Type": "application/json"}
if dp_rank is not None:
headers["X-data-parallel-rank"] = str(dp_rank)
try:
prefill_response = await prefill_session.post(
f"{p_url}/v1/chat/completions",
@@ -594,9 +938,7 @@ async def on_startup() -> None:
),
)
encode_session = aiohttp.ClientSession(timeout=timeout, connector=connector)
if app.state.p_urls:
# only setup if prefill instance(s) exist
prefill_session = aiohttp.ClientSession(timeout=timeout, connector=connector)
prefill_session = aiohttp.ClientSession(timeout=timeout, connector=connector)
decode_session = aiohttp.ClientSession(timeout=timeout, connector=connector)
@@ -622,6 +964,7 @@ async def prepare_for_decode(
e_urls: list[str],
p_url: str,
consumer_zmq: str | None,
prefill_dp_rank: int | None = None,
) -> tuple[dict, float, float]:
"""Encode, rewrite and prefill, returning the body to send to decode.
@@ -642,7 +985,7 @@ async def prepare_for_decode(
handles.update(ec_params)
prepared = {**prepared, "ec_transfer_params": handles}
_t2 = time.perf_counter()
prepared = await maybe_prefill(prepared, p_url, req_id)
prepared = await maybe_prefill(prepared, p_url, req_id, prefill_dp_rank)
return prepared, _t1 - _t0, _t2 - _t1
@@ -654,12 +997,13 @@ async def forward_non_stream(
d_url: str,
consumer_zmq: str | None,
dp_rank: int | None = None,
prefill_dp_rank: int | None = None,
) -> Response:
try:
for attempt in range(DECODE_RETRIES + 1):
_t0 = time.perf_counter()
prepared, encode_s, rewrite_s = await prepare_for_decode(
req_data, req_id, e_urls, p_url, consumer_zmq
req_data, req_id, e_urls, p_url, consumer_zmq, prefill_dp_rank
)
_t2 = time.perf_counter()
@@ -733,12 +1077,13 @@ async def forward_stream(
d_url: str,
consumer_zmq: str | None,
dp_rank: int | None = None,
prefill_dp_rank: int | None = None,
) -> AsyncIterator[bytes]:
try:
for attempt in range(DECODE_RETRIES + 1):
_t0 = time.perf_counter()
prepared, encode_s, rewrite_s = await prepare_for_decode(
req_data, req_id, e_urls, p_url, consumer_zmq
req_data, req_id, e_urls, p_url, consumer_zmq, prefill_dp_rank
)
_t2 = time.perf_counter()
@@ -976,10 +1321,147 @@ async def stop_profile(request: Request):
return await _profile_cmd("stop", body, e_url, p_url, d_url)
def build_app(config: EPDProxyConfig | None = None) -> FastAPI:
config = config or EPDProxyConfig()
registry = InstanceRegistry(
probe_interval=config.probe_interval,
probe_timeout=config.probe_timeout,
fail_threshold=config.fail_threshold,
evicted_ttl=config.evicted_ttl,
)
proxy = EPDProxy(registry)
@asynccontextmanager
async def lifespan(app: FastAPI):
await on_startup()
try:
registry.start_probing()
yield
finally:
await registry.stop_probing()
await on_shutdown()
app = FastAPI(lifespan=lifespan)
app.state.proxy = proxy
app.state.registry = registry
@app.get("/instances")
async def list_instances():
return registry.status()
@app.post("/instances", dependencies=[Depends(require_admin_key)])
async def register_instance(body: InstanceRegistration):
if body.ec_zmq_addrs and body.role not in (
InstanceRole.PREFILL,
InstanceRole.PREFILL_DECODE,
):
raise HTTPException(400, "Only EC consumers accept Mooncake addresses")
try:
created = registry.register(
InstanceRecord(
body.role,
str(body.url).rstrip("/"),
body.ec_zmq_addrs,
body.dp_size,
)
)
except ValueError as error:
raise HTTPException(409, str(error)) from error
return {"registered": created}
@app.delete("/instances", dependencies=[Depends(require_admin_key)])
async def unregister_instance(url: AnyHttpUrl):
return {"removed": registry.unregister(str(url).rstrip("/"))}
@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
req_data = msgspec.json.decode(await request.body())
req_id = request.headers.get("x-request-id", str(uuid.uuid4()))
route = proxy.route(len(extract_mm_items(req_data)))
args = (
req_data,
req_id,
route.encoder_urls,
route.prefill.url if route.prefill else None,
route.decode.url,
route.consumer_zmq,
)
ranks = {
"dp_rank": route.dp_rank if route.prefill is None else None,
"prefill_dp_rank": route.dp_rank if route.prefill is not None else None,
}
if req_data.get("stream", False):
return StreamingResponse(
forward_stream(*args, **ranks), media_type="text/event-stream"
)
return await forward_non_stream(*args, **ranks)
@app.get("/v1/models")
async def list_models():
decode = registry.pick(InstanceRole.PREFILL_DECODE) or registry.pick(
InstanceRole.DECODE
)
if decode is None:
raise HTTPException(
status_code=503, detail="No decode instance is registered"
)
async with decode_session.get(f"{decode.url}/v1/models") as resp:
resp.raise_for_status()
return await resp.json()
@app.get("/health")
async def health():
status = registry.status()
# An empty roster is not unhealthy: the proxy is meant to come up
# before anything registers with it.
return JSONResponse({"proxy": "healthy", "instances": status})
@app.post("/start_profile")
async def start_profile(request: Request):
return await _profile(registry, "start", await request.json())
@app.post("/stop_profile")
async def stop_profile(request: Request):
return await _profile(registry, "stop", await request.json())
return app
async def _profile(registry: InstanceRegistry, cmd: str, payload: dict) -> dict:
headers = {"Authorization": f"Bearer {os.getenv('OPENAI_API_KEY', '')}"}
targets = {role.value: registry.pick(role) for role in InstanceRole}
results = await asyncio.gather(
*(
_post_if_available(
decode_session, f"{record.url}/{cmd}_profile", payload, headers
)
for record in targets.values()
if record is not None
)
)
reachable = [result for result in results if result is not None]
if not reachable:
raise HTTPException(
status_code=503,
detail="Profiling endpoints are disabled on every instance",
)
live = [name for name, record in targets.items() if record is not None]
return dict(zip(live, results))
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--host", default="0.0.0.0")
parser.add_argument("--port", type=int, default=8000)
parser.add_argument(
"--dynamic-registration",
action="store_true",
help="Enable launcher-managed HTTP registration for E+PD or E+P+D.",
)
parser.add_argument("--probe-interval", type=float, default=DEFAULT_PROBE_INTERVAL)
parser.add_argument("--probe-timeout", type=float, default=DEFAULT_PROBE_TIMEOUT)
parser.add_argument("--fail-threshold", type=int, default=DEFAULT_FAIL_THRESHOLD)
parser.add_argument("--evicted-ttl", type=float, default=DEFAULT_EVICTED_TTL)
parser.add_argument(
"--log-requests",
action="store_true",
@@ -995,12 +1477,12 @@ if __name__ == "__main__":
)
parser.add_argument(
"--encode-servers-urls",
required=True,
default="",
help='Comma-separated encode URLs ("http://e1:8001,http://e2:8001")',
)
parser.add_argument(
"--prefill-servers-urls",
required=True,
default="none",
help=(
'Comma-separated prefill URLs ("http://p1:8003,http://p2:8004") '
'to enable E->P->D, set "disable" or "none" to enable E->PD'
@@ -1008,7 +1490,7 @@ if __name__ == "__main__":
)
parser.add_argument(
"--decode-servers-urls",
required=True,
default="",
help='Comma-separated decode URLs ("http://d1:8005,http://d2:8006")',
)
parser.add_argument(
@@ -1043,58 +1525,87 @@ if __name__ == "__main__":
)
args = parser.parse_args()
if args.fail_threshold < 1:
parser.error("--fail-threshold must be at least 1")
if args.log_requests:
logging.getLogger().setLevel(logging.DEBUG)
app.middleware("http")(log_requests)
NO_REWRITE = args.no_rewrite
DECODE_RETRIES = max(0, args.decode_retries)
app.state.e_urls = [
u.strip() for u in args.encode_servers_urls.split(",") if u.strip()
]
app.state.d_urls = [
u.strip() for u in args.decode_servers_urls.split(",") if u.strip()
]
app.state.d_ec_urls = [
u.strip() for u in args.ec_consumer_zmq_addrs.split(",") if u.strip()
]
if args.ec_consumer_dp_size < 1:
parser.error("--ec-consumer-dp-size must be at least 1")
app.state.ec_consumer_dp_size = args.ec_consumer_dp_size
app.state.replica_counter = itertools.count()
expected = len(app.state.d_urls) * args.ec_consumer_dp_size
if app.state.d_ec_urls and len(app.state.d_ec_urls) != expected:
parser.error(
"--ec-consumer-zmq-addrs must contain one address per consumer "
f"replica: expected {expected} "
f"({len(app.state.d_urls)} servers x {args.ec_consumer_dp_size} replicas), "
f"got {len(app.state.d_ec_urls)}"
)
# handle prefill instances
if args.prefill_servers_urls.lower() in ("disable", "none", ""):
app.state.p_urls = []
logger.info(
"Disaggregated prefill phase explicitly disabled by user. Running E + PD..."
if args.dynamic_registration:
if not os.getenv("ADMIN_API_KEY"):
parser.error("--dynamic-registration requires ADMIN_API_KEY")
if (
args.encode_servers_urls
or args.prefill_servers_urls.lower() not in ("disable", "none", "")
or args.decode_servers_urls
or args.ec_consumer_zmq_addrs
):
parser.error(
"With --dynamic-registration, instances register through /instances"
)
app = build_app(
EPDProxyConfig(
probe_interval=args.probe_interval,
probe_timeout=args.probe_timeout,
fail_threshold=args.fail_threshold,
evicted_ttl=args.evicted_ttl,
)
)
if args.log_requests:
app.middleware("http")(log_requests)
else:
app.state.p_urls = [
u.strip() for u in args.prefill_servers_urls.split(",") if u.strip()
if not args.encode_servers_urls or not args.decode_servers_urls:
parser.error(
"Static routing requires --encode-servers-urls "
"and --decode-servers-urls"
)
app.state.e_urls = [
u.strip() for u in args.encode_servers_urls.split(",") if u.strip()
]
logger.info("Disaggregated prefill phase is enabled. Running E + P + D...")
try:
validate_ec_consumer_routing(app.state.p_urls, app.state.d_ec_urls)
except ValueError as exc:
parser.error(str(exc))
app.state.d_urls = [
u.strip() for u in args.decode_servers_urls.split(",") if u.strip()
]
app.state.d_ec_urls = [
u.strip() for u in args.ec_consumer_zmq_addrs.split(",") if u.strip()
]
if args.ec_consumer_dp_size < 1:
parser.error("--ec-consumer-dp-size must be at least 1")
app.state.ec_consumer_dp_size = args.ec_consumer_dp_size
app.state.replica_counter = itertools.count()
expected = len(app.state.d_urls) * args.ec_consumer_dp_size
if app.state.d_ec_urls and len(app.state.d_ec_urls) != expected:
parser.error(
"--ec-consumer-zmq-addrs must contain one address per consumer "
f"replica: expected {expected} "
f"({len(app.state.d_urls)} servers x "
f"{args.ec_consumer_dp_size} replicas), "
f"got {len(app.state.d_ec_urls)}"
)
# handle prefill instances
if args.prefill_servers_urls.lower() in ("disable", "none", ""):
app.state.p_urls = []
logger.info("Disaggregated prefill disabled. Running E + PD...")
else:
app.state.p_urls = [
u.strip() for u in args.prefill_servers_urls.split(",") if u.strip()
]
logger.info("Disaggregated prefill phase is enabled. Running E + P + D...")
try:
validate_ec_consumer_routing(app.state.p_urls, app.state.d_ec_urls)
except ValueError as exc:
parser.error(str(exc))
logger.info("Proxy listening on %s:%s", args.host, args.port)
logger.info("Encode servers: %s", app.state.e_urls)
logger.info("Prefill instances %s", app.state.p_urls)
logger.info("Decode servers: %s", app.state.d_urls)
if app.state.ec_consumer_dp_size > 1:
logger.info(
"EC consumer replicas per server: %d (control addresses: %s)",
app.state.ec_consumer_dp_size,
app.state.d_ec_urls,
)
logger.info("Proxy listening on %s:%s", args.host, args.port)
logger.info("Encode servers: %s", app.state.e_urls)
logger.info("Prefill instances %s", app.state.p_urls)
logger.info("Decode servers: %s", app.state.d_urls)
if app.state.ec_consumer_dp_size > 1:
logger.info(
"EC consumer replicas per server: %d (control addresses: %s)",
app.state.ec_consumer_dp_size,
app.state.d_ec_urls,
)
uvicorn.run(
app,