Add Understanding the Latency Metrics docs (#50600)

Signed-off-by: mgoin <mgoin64@gmail.com>
This commit is contained in:
Michael Goin
2026-07-31 23:22:40 -07:00
committed by GitHub
parent 9c110fa522
commit 4ee9702bee
3 changed files with 193 additions and 0 deletions
+39
View File
@@ -111,6 +111,45 @@ P99 ITL (ms): 8.39
==================================================
```
#### Understanding the Latency Metrics
`vllm bench serve` measures latency at the benchmark client:
!!! note
Metric terminology is not standardized across benchmarking tools. When
comparing results, use the measurement points and formulas rather than the
metric names alone. This section explains how we refer to them in vLLM.
- **Time to first token (TTFT)** is the time from sending a request to receiving
its first streamed output.
- **Inter-token latency (ITL)** records the time between consecutive streamed
outputs. The reported ITL statistics aggregate these individual gaps across
all successful requests.
- **Time per output token (TPOT)** is calculated once per request, excluding the
first token, and then aggregated across requests:
$$
\text{TPOT} =
\frac{\text{end-to-end latency} - \text{TTFT}}
{\text{number of output tokens} - 1}
$$
With standard decoding, each streamed output usually contains one token, so ITL
and TPOT are typically similar.
With speculative decoding, one streamed output can contain multiple tokens,
such as several accepted draft tokens within a single engine tstep. ITL records
only the gaps between streamed outputs; it does not add zero-duration gaps for
tokens in the same output. TPOT instead amortizes the request's decoding time
over every output token.
![Latency metrics with bundled tokens (light theme)](../assets/benchmarking/latency-metrics-speculative-decoding-light.svg#only-light)
![Latency metrics with bundled tokens (dark theme)](../assets/benchmarking/latency-metrics-speculative-decoding-dark.svg#only-dark)
In this example, the benchmark observes two 40 ms ITL samples. The three tokens
in the second streamed output do not create additional ITL samples, so mean ITL
is 40 ms. TPOT is `(180 ms - 100 ms) / (5 - 1) = 20 ms/token`.
#### Results Visualization
The `--plot-timeline` and `--plot-dataset-stats` can be used to generate respectively the requests completion timeline and dataset prompt and output tokens statistics, which can be useful for debugging purpose or for deeper analysis.