[Frontend] Gate scale-out endpoints behind opt-in flag (#54579)
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
This commit is contained in:
@@ -121,6 +121,13 @@ For further details on profiling vLLM, please refer to [this page](../../contrib
|
||||
|
||||
## Scale-Out APIs
|
||||
|
||||
Scale-out APIs are disabled by default on `vllm serve`. The environment
|
||||
variable accepts only `0` or `1`; set `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1` to
|
||||
register the endpoints below. The
|
||||
dedicated `vllm launch render` and `vllm serve --tokens-only` modes are explicit
|
||||
opt-ins and enable their required endpoints when the variable is unset; an
|
||||
explicit value of `0` is rejected for those modes.
|
||||
|
||||
### Tokens IN <> Tokens OUT APIs
|
||||
|
||||
- `/inference/v1/generate` - Generate completions
|
||||
|
||||
@@ -56,6 +56,14 @@ Oversized payloads are rejected with a `400` before any `tokenizer.decode()` or
|
||||
|
||||
The example below drives the full `render → generate → derender` round trip for a chat request against a GPU less render server (`/render`, `/derender`) and a token-in / token-out engine (`/inference/v1/generate`).
|
||||
|
||||
Launch the two servers first:
|
||||
|
||||
```bash
|
||||
vllm launch render meta-llama/Llama-3.2-1B-Instruct --port 8100
|
||||
VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 vllm serve \
|
||||
meta-llama/Llama-3.2-1B-Instruct --tokens-only --port 8200
|
||||
```
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
|
||||
@@ -6,6 +6,19 @@ Our renderer API is designed to disaggregate the render phase(preprocessing) and
|
||||
- Disaggregated tokenization: Support use cases such as llm-d, Dynamo, and custom frontends that need to leverage vLLM's preprocessing logic without running the full inference engine.
|
||||
- Tokens-in / tokens-out engine: Make the engine a pure token-in / token-out service, decoupled from request preprocessing.
|
||||
|
||||
The dedicated `vllm launch render` server always exposes the `/render` and
|
||||
`/derender` endpoints when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS` is unset or set to
|
||||
`1`. An explicit value of `0` conflicts with the renderer command and is
|
||||
rejected at startup.
|
||||
|
||||
Scale-out endpoints, including `/render`, `/derender`, and
|
||||
`/inference/v1/generate`, are disabled by default on a standard inference
|
||||
server. To expose them with `vllm serve`, opt in explicitly:
|
||||
|
||||
```bash
|
||||
VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 vllm serve <model>
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
- [Completions Render API](renderer.md) (`/v1/completions/render`)
|
||||
|
||||
Reference in New Issue
Block a user