feat: Set concurrency and the CUDA graph list from .env

MAX_SEQS and the prefill chunk were fixed in the start profile, so any
change to concurrency needed an edit to the script. Read both from .env,
and validate them before the engine starts. The chunk must be at least
the sequence count, the rule vLLM applies at scheduler.py:309. A
MAX_SEQS whose full context exceeds the 924,993-token pool prints a
note, because that shape preempts and recomputes instead of failing.

The captured graph list was a fixed [1,2,4,8,12,16], which covers a
decode batch of 4 requests and nothing wider. One decode step is K+1 = 4
tokens per request, so the list now follows MAX_SEQS: 1, 2, 4, then
every 4 tokens up to MAX_SEQS x 4. At the shipped 4 it produces the old
list exactly. Above 8 graphs a note prints, since each size costs
capture time and graph VRAM.

Drop --mm-processor-cache-gb. This workload sends each image once, so
the cache never had a hit to keep, and vLLM's own 4 GiB default is the
better base.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
2026-10-18 02:04:50 +07:00
co-authored by Hinata
parent fab0eb9fc7
commit d9ca3bc4f2
5 changed files with 53 additions and 3 deletions
+2
View File
@@ -41,6 +41,8 @@ services:
GUARD: ${GUARD:-warn}
COUNTERS: ${COUNTERS:-0}
VLLM_LOG_STATS_INTERVAL: ${VLLM_LOG_STATS_INTERVAL:-1}
MAX_SEQS: ${MAX_SEQS:-4}
MAX_BATCHED_TOKENS: ${MAX_BATCHED_TOKENS:-2048}
QWEN38_ABLITERATION: ${QWEN38_ABLITERATION:-Qwen3.8-Flash-Next-Uncensored}
volumes:
- "${MODEL_DIR:?Set MODEL_DIR in .env}:${MODEL_MOUNT_PATH:-/models/Qwen3.8-Flash-Next}:ro"