[Benchmark] Add vllm bench forward: exact-shape forward timing (#61339)

Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Woosuk Kwon
2026-10-14 10:52:26 -07:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 2c74fd1873
commit 9d3dfa2886
14 changed files with 1320 additions and 1 deletions
+14 -1
View File
@@ -131,9 +131,22 @@ To use benchmark commands, please install with extra dependencies using `pip ins
Available Commands:
```bash
vllm bench {latency, serve, throughput}
vllm bench {forward, latency, serve, throughput}
```
### forward
Benchmark the model forward of exact batches: each case lists its requests as `[N*]Q[@K][c][:D]` (query length Q, KV length K including the query, D draft tokens). A prefill's prefix is a prefix-cache hit unless `c` marks a chunked prefill; they differ only on models with SWA bounded replay, which recompute the last window of a hit. Each request's prefix length varies by ±50% by default (`--kv-len-jitter`), identically on every DP rank. Each forward is captured as a one-shot CUDA graph; sampling and CPU overhead aren't measured. Use `torchrun`, one process per GPU, for DP/TP/EP. CUDA only for now.
```bash
vllm bench forward \
--model meta-llama/Llama-3.2-1B-Instruct \
--case decode=32*1@4096 \
--case mix=2048,32*1@4096
```
See [vllm bench forward](./bench/forward.md) for the full reference of all available arguments.
### latency
Benchmark the latency of a single batch of requests.