[Benchmark] Warn on warm prefix cache for random serve runs (#53920)

Signed-off-by: Zupeng Wang <71580390+zupengwang@users.noreply.github.com>
Co-authored-by: OpenAI <support@openai.com>
This commit is contained in:
Zupeng Wang
2026-08-27 05:51:17 +00:00
committed by GitHub
co-authored by OpenAI
parent b3af042abd
commit 833fa6233c
+9
View File
@@ -111,6 +111,15 @@ P99 ITL (ms): 8.39
==================================================
```
!!! warning
Repeating `vllm bench serve` against the same server can reuse prompts left
in the prefix cache and inflate throughput. This can affect any reproducible
dataset; the synthetic random dataset is reproducible for a fixed `--seed`,
which defaults to `0`. Prefix cache hits can also come from shared prefixes
within the run, so interpret cache metrics in the context of the workload.
If cache reuse is not intended, vary `--seed`, reset or restart the server,
or use `vllm bench sweep serve`, which resets server caches between runs.
#### Understanding the Latency Metrics
`vllm bench serve` measures latency at the benchmark client: