From 833fa6233c61b34ea37397a778c00e545c4732f4 Mon Sep 17 00:00:00 2001 From: Zupeng Wang Date: Thu, 27 Aug 2026 13:51:17 +0800 Subject: [PATCH] [Benchmark] Warn on warm prefix cache for random serve runs (#53920) Signed-off-by: Zupeng Wang <71580390+zupengwang@users.noreply.github.com> Co-authored-by: OpenAI --- docs/benchmarking/cli.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/benchmarking/cli.md b/docs/benchmarking/cli.md index 2437392d9b..5ac9a6126d 100644 --- a/docs/benchmarking/cli.md +++ b/docs/benchmarking/cli.md @@ -111,6 +111,15 @@ P99 ITL (ms): 8.39 ================================================== ``` +!!! warning + Repeating `vllm bench serve` against the same server can reuse prompts left + in the prefix cache and inflate throughput. This can affect any reproducible + dataset; the synthetic random dataset is reproducible for a fixed `--seed`, + which defaults to `0`. Prefix cache hits can also come from shared prefixes + within the run, so interpret cache metrics in the context of the workload. + If cache reuse is not intended, vary `--seed`, reset or restart the server, + or use `vllm bench sweep serve`, which resets server caches between runs. + #### Understanding the Latency Metrics `vllm bench serve` measures latency at the benchmark client: