[Frontend] Gate scale-out endpoints behind opt-in flag (#54579)

Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
This commit is contained in:
Francisco Javier Arceo
2026-09-02 14:31:14 +08:00
committed by GitHub
parent 300f688321
commit c00091e026
20 changed files with 386 additions and 39 deletions
+2 -1
View File
@@ -13,7 +13,8 @@ zero client-side transformation.
Launch the server first:
vllm serve Qwen/Qwen3-VL-2B-Instruct \
VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 \
vllm serve Qwen/Qwen3-VL-2B-Instruct \
--dtype bfloat16 --max-model-len 4096 --enforce-eager
Then run this script:
@@ -1,5 +1,10 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Call a token-in/token-out server started with:
VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 vllm serve Qwen/Qwen3-0.6B
"""
import httpx
from transformers import AutoTokenizer