[Frontend] Gate scale-out endpoints behind opt-in flag (#54579)

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
This commit is contained in:
Francisco Javier Arceo
2026-09-02 14:31:14 +08:00
committed by GitHub
parent 300f688321
commit c00091e026
20 changed files with 386 additions and 39 deletions
+7
View File
@@ -121,6 +121,13 @@ For further details on profiling vLLM, please refer to [this page](../../contrib
## Scale-Out APIs
Scale-out APIs are disabled by default on `vllm serve`. The environment
variable accepts only `0` or `1`; set `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1` to
register the endpoints below. The
dedicated `vllm launch render` and `vllm serve --tokens-only` modes are explicit
opt-ins and enable their required endpoints when the variable is unset; an
explicit value of `0` is rejected for those modes.
### Tokens IN <> Tokens OUT APIs
- `/inference/v1/generate` - Generate completions
@@ -56,6 +56,14 @@ Oversized payloads are rejected with a `400` before any `tokenizer.decode()` or
The example below drives the full `render → generate → derender` round trip for a chat request against a GPU less render server (`/render`, `/derender`) and a token-in / token-out engine (`/inference/v1/generate`).
Launch the two servers first:
```bash
vllm launch render meta-llama/Llama-3.2-1B-Instruct --port 8100
VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 vllm serve \
meta-llama/Llama-3.2-1B-Instruct --tokens-only --port 8200
```
```python
import httpx
+13
View File
@@ -6,6 +6,19 @@ Our renderer API is designed to disaggregate the render phase(preprocessing) and
- Disaggregated tokenization: Support use cases such as llm-d, Dynamo, and custom frontends that need to leverage vLLM's preprocessing logic without running the full inference engine.
- Tokens-in / tokens-out engine: Make the engine a pure token-in / token-out service, decoupled from request preprocessing.
The dedicated `vllm launch render` server always exposes the `/render` and
`/derender` endpoints when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS` is unset or set to
`1`. An explicit value of `0` conflicts with the renderer command and is
rejected at startup.
Scale-out endpoints, including `/render`, `/derender`, and
`/inference/v1/generate`, are disabled by default on a standard inference
server. To expose them with `vllm serve`, opt in explicitly:
```bash
VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 vllm serve <model>
```
## API Reference
- [Completions Render API](renderer.md) (`/v1/completions/render`)
+9 -7
View File
@@ -158,15 +158,16 @@ When `--api-key` is configured, the following `/v1` endpoints require Bearer tok
- `/v1/models` - List available models
- `/v1/chat/completions` - Chat completions
- `/v1/chat/completions/batch` - Batch chat completions
- `/v1/chat/completions/render` - Render chat completion requests
- `/v1/chat/completions/derender` - Derender chat completion requests
- `/v1/chat/completions/render` - Render chat completion requests (available on `vllm serve` only when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or on `vllm launch render` unless explicitly disabled)
- `/v1/chat/completions/derender` - Derender chat completion requests (available on `vllm serve` only when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or on `vllm launch render` unless explicitly disabled)
- `/v1/completions` - Text completions
- `/v1/completions/render` - Render completion requests
- `/v1/completions/derender` - Derender completion requests
- `/v1/completions/render` - Render completion requests (available on `vllm serve` only when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or on `vllm launch render` unless explicitly disabled)
- `/v1/completions/derender` - Derender completion requests (available on `vllm serve` only when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or on `vllm launch render` unless explicitly disabled)
- `/v1/embeddings` - Generate embeddings
- `/v1/audio/transcriptions` - Audio transcription
- `/v1/audio/translations` - Audio translation
- `/v1/messages` - Anthropic-compatible messages API
- `/v1/messages/render` - Render Anthropic-compatible messages (available on `vllm serve` only when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or on `vllm launch render` unless explicitly disabled)
- `/v1/messages/count_tokens` - Count tokens for Anthropic messages
- `/v1/responses` - Create a response
- `/v1/responses/{response_id}` - Retrieve a response
@@ -175,7 +176,8 @@ When `--api-key` is configured, the following `/v1` endpoints require Bearer tok
- `/v1/rerank` - Reranking API
- `/v1/load_lora_adapter` - Load a LoRA adapter (can alter model behavior; only available when `--enable-lora` is set and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`)
- `/v1/unload_lora_adapter` - Unload a LoRA adapter (can alter model behavior; only available when `--enable-lora` is set and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`)
- `/inference/v1/generate` - Generate completions
- `/inference/v1/generate` - Generate completions (available when `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`, or with `--tokens-only` unless explicitly disabled)
- `/cohere/v2/chat/render` - Render Cohere Chat v2 requests (requires both `VLLM_ENABLE_COHERE_API=1` and `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1`)
- `/v2/embed` - Cohere Embed API
- `/v2/rerank` - Cohere Rerank API
@@ -203,11 +205,11 @@ The following endpoints **do not require authentication** even when `--api-key`
- `/init_weight_transfer_engine` - Initialize weight transfer engine for RLHF
- `/update_weights` - Update model weights (can alter model behavior)
- `/get_world_size` - Get distributed world size
- `/abort_requests` - Abort in-flight requests (only when `--tokens-only` is also set)
- `/abort_requests` - Abort in-flight requests (available with `--tokens-only` unless `VLLM_ENABLE_SCALE_OUT_ENDPOINTS=0`)
**Utility endpoints:**
- `/tokenize` - Tokenize text
- `/tokenize` - Tokenize text (not disabled by `VLLM_ENABLE_SCALE_OUT_ENDPOINTS`)
- `/detokenize` - Detokenize tokens
- `/health` - Health check
- `/ping` - SageMaker health check