[Benchmark] Add vllm bench forward: exact-shape forward timing (#61339)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
2c74fd1873
commit
9d3dfa2886
+14
-1
@@ -131,9 +131,22 @@ To use benchmark commands, please install with extra dependencies using `pip ins
|
||||
Available Commands:
|
||||
|
||||
```bash
|
||||
vllm bench {latency, serve, throughput}
|
||||
vllm bench {forward, latency, serve, throughput}
|
||||
```
|
||||
|
||||
### forward
|
||||
|
||||
Benchmark the model forward of exact batches: each case lists its requests as `[N*]Q[@K][c][:D]` (query length Q, KV length K including the query, D draft tokens). A prefill's prefix is a prefix-cache hit unless `c` marks a chunked prefill; they differ only on models with SWA bounded replay, which recompute the last window of a hit. Each request's prefix length varies by ±50% by default (`--kv-len-jitter`), identically on every DP rank. Each forward is captured as a one-shot CUDA graph; sampling and CPU overhead aren't measured. Use `torchrun`, one process per GPU, for DP/TP/EP. CUDA only for now.
|
||||
|
||||
```bash
|
||||
vllm bench forward \
|
||||
--model meta-llama/Llama-3.2-1B-Instruct \
|
||||
--case decode=32*1@4096 \
|
||||
--case mix=2048,32*1@4096
|
||||
```
|
||||
|
||||
See [vllm bench forward](./bench/forward.md) for the full reference of all available arguments.
|
||||
|
||||
### latency
|
||||
|
||||
Benchmark the latency of a single batch of requests.
|
||||
|
||||
@@ -145,6 +145,7 @@ def auto_mock(module_name: str, attr: str, max_mocks: int = 100):
|
||||
)
|
||||
|
||||
|
||||
bench_forward = auto_mock("vllm.benchmarks.forward", "main")
|
||||
bench_latency = auto_mock("vllm.benchmarks", "latency")
|
||||
bench_serve = auto_mock("vllm.benchmarks", "serve")
|
||||
bench_startup = auto_mock("vllm.benchmarks", "startup")
|
||||
@@ -340,6 +341,7 @@ pages = {
|
||||
False,
|
||||
),
|
||||
"cli/launch/render.md": (create_parser(RenderSubcommand.add_cli_args), True),
|
||||
"cli/bench/forward.md": (create_parser(bench_forward.add_cli_args), True),
|
||||
"cli/bench/latency.md": (create_parser(bench_latency.add_cli_args), True),
|
||||
# URL kept as `mm_processor` for back-compat; command name is `mm-processor`
|
||||
"cli/bench/mm_processor.md": (
|
||||
|
||||
Reference in New Issue
Block a user