[Benchmark] Add vllm bench forward: exact-shape forward timing (#61339)

Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Woosuk Kwon
2026-10-14 10:52:26 -07:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 2c74fd1873
commit 9d3dfa2886
14 changed files with 1320 additions and 1 deletions
+14 -1
View File
@@ -131,9 +131,22 @@ To use benchmark commands, please install with extra dependencies using `pip ins
Available Commands:
```bash
vllm bench {latency, serve, throughput}
vllm bench {forward, latency, serve, throughput}
```
### forward
Benchmark the model forward of exact batches: each case lists its requests as `[N*]Q[@K][c][:D]` (query length Q, KV length K including the query, D draft tokens). A prefill's prefix is a prefix-cache hit unless `c` marks a chunked prefill; they differ only on models with SWA bounded replay, which recompute the last window of a hit. Each request's prefix length varies by ±50% by default (`--kv-len-jitter`), identically on every DP rank. Each forward is captured as a one-shot CUDA graph; sampling and CPU overhead aren't measured. Use `torchrun`, one process per GPU, for DP/TP/EP. CUDA only for now.
```bash
vllm bench forward \
--model meta-llama/Llama-3.2-1B-Instruct \
--case decode=32*1@4096 \
--case mix=2048,32*1@4096
```
See [vllm bench forward](./bench/forward.md) for the full reference of all available arguments.
### latency
Benchmark the latency of a single batch of requests.
@@ -145,6 +145,7 @@ def auto_mock(module_name: str, attr: str, max_mocks: int = 100):
)
bench_forward = auto_mock("vllm.benchmarks.forward", "main")
bench_latency = auto_mock("vllm.benchmarks", "latency")
bench_serve = auto_mock("vllm.benchmarks", "serve")
bench_startup = auto_mock("vllm.benchmarks", "startup")
@@ -340,6 +341,7 @@ pages = {
False,
),
"cli/launch/render.md": (create_parser(RenderSubcommand.add_cli_args), True),
"cli/bench/forward.md": (create_parser(bench_forward.add_cli_args), True),
"cli/bench/latency.md": (create_parser(bench_latency.add_cli_args), True),
# URL kept as `mm_processor` for back-compat; command name is `mm-processor`
"cli/bench/mm_processor.md": (