[Rust Frontend] Expose gRPC port in Python vllm serve (#59659)
Co-authored-by: Bugen Zhao <i@bugenzhao.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Alec Flowers <aflowers@nvidia.com> Signed-off-by: Bugen Zhao <i@bugenzhao.com>
This commit is contained in:
co-authored by
Bugen Zhao
Claude Opus 5.5
parent
b558f160a2
commit
9d2a6f52b2
@@ -43,6 +43,27 @@ For example:
|
||||
VLLM_USE_RUST_FRONTEND=1 vllm serve Qwen/Qwen3-0.6B
|
||||
```
|
||||
|
||||
To also expose the Rust frontend's gRPC inference and control services, add
|
||||
`--grpc-port`. The HTTP listener remains on `--port`:
|
||||
|
||||
```bash
|
||||
VLLM_USE_RUST_FRONTEND=1 vllm serve Qwen/Qwen3-0.6B \
|
||||
--host 127.0.0.1 --port 8000 --grpc-port 50051
|
||||
```
|
||||
|
||||
The gRPC listener binds on `--host` like HTTP, or on `127.0.0.1` when HTTP uses
|
||||
`--uds`. When HTTP uses TCP, its port and the gRPC port must differ unless
|
||||
requesting an automatically assigned port with `0`.
|
||||
|
||||
This also works with Python-supervised hybrid data parallelism. It requires a
|
||||
frontend process and cannot be combined with `--headless` or
|
||||
`--data-parallel-multi-port-external-lb`.
|
||||
|
||||
The separate `vllm serve MODEL --grpc --port 50051` command launches the Python
|
||||
SMG `VllmEngine` gRPC server in place of HTTP. The Rust listener exposes the
|
||||
`vllm.Inference` and `vllm.Control` services alongside HTTP. These are different
|
||||
gRPC APIs; `--grpc` and `--grpc-port` are mutually exclusive.
|
||||
|
||||
### RL weight synchronization
|
||||
|
||||
With `VLLM_SERVER_DEV_MODE=1`, the Rust frontend supports the HTTP weight-transfer
|
||||
|
||||
Reference in New Issue
Block a user