feat: Accept any model name in a request
A request that named another model failed with 404 "The model `X` does not exist.", and the model name changes often here. vLLM already has the switch: _is_model_supported() accepts every name when VLLM_SKIP_MODEL_NAME_VALIDATION is set, and _check_model is the only gate, on the chat, completion, responses, tokenize, pooling and realtime paths alike. Turn it on in the serve profile, under the upstream name, so 0 still restores the strict check. The served name does not change: /v1/models and the model field of every reply still report MODEL_NAME. No patch is needed, and Compose bind-mounts start.sh, so a restart applies this. Co-Authored-By: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
@@ -209,6 +209,10 @@ export HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1
|
||||
export MERLIN_FULL_K=1 MERLIN_QSA_SORT=1 MERLIN_TIE_DET=1 MERLIN_TIE_RECENT=1 MERLIN_TOPK_SORTED_EMIT=1
|
||||
export VLLM_E2_GUARD_MODE="${GUARD:-warn}"
|
||||
export VLLM_E2_SIDE_CACHE_LAYOUT=1 VLLM_USE_V2_MODEL_RUNNER=1 VLLM_E2_MAMBA_RETAIN_CHECKPOINT=1
|
||||
# Accept any name in the model field of a request, like llama.cpp. Upstream's own switch, on by
|
||||
# default here: /v1/models and the model field of every reply still report the served name.
|
||||
# Set VLLM_SKIP_MODEL_NAME_VALIDATION=0 to reject a name that is not served.
|
||||
export VLLM_SKIP_MODEL_NAME_VALIDATION="${VLLM_SKIP_MODEL_NAME_VALIDATION:-1}"
|
||||
if [ "${COUNTERS:-0}" = 1 ]; then export MERLIN_TOPK_EXACT_COUNTERS_LOG=30 VLLM_QSA_KV_CLIP_COUNT=300; fi
|
||||
[ -f "$VLLM_QSA_KV_SCALES" ] || { echo "scales sidecar missing: $VLLM_QSA_KV_SCALES (FP8 KV would run at scale 1.0). Set SCALES=<path> to the sidecar of this model."; exit 1; }
|
||||
|
||||
|
||||
Reference in New Issue
Block a user