Add Understanding the Latency Metrics docs (#50600)
Signed-off-by: mgoin <mgoin64@gmail.com>
This commit is contained in:
@@ -111,6 +111,45 @@ P99 ITL (ms): 8.39
|
||||
==================================================
|
||||
```
|
||||
|
||||
#### Understanding the Latency Metrics
|
||||
|
||||
`vllm bench serve` measures latency at the benchmark client:
|
||||
|
||||
!!! note
|
||||
Metric terminology is not standardized across benchmarking tools. When
|
||||
comparing results, use the measurement points and formulas rather than the
|
||||
metric names alone. This section explains how we refer to them in vLLM.
|
||||
|
||||
- **Time to first token (TTFT)** is the time from sending a request to receiving
|
||||
its first streamed output.
|
||||
- **Inter-token latency (ITL)** records the time between consecutive streamed
|
||||
outputs. The reported ITL statistics aggregate these individual gaps across
|
||||
all successful requests.
|
||||
- **Time per output token (TPOT)** is calculated once per request, excluding the
|
||||
first token, and then aggregated across requests:
|
||||
|
||||
$$
|
||||
\text{TPOT} =
|
||||
\frac{\text{end-to-end latency} - \text{TTFT}}
|
||||
{\text{number of output tokens} - 1}
|
||||
$$
|
||||
|
||||
With standard decoding, each streamed output usually contains one token, so ITL
|
||||
and TPOT are typically similar.
|
||||
|
||||
With speculative decoding, one streamed output can contain multiple tokens,
|
||||
such as several accepted draft tokens within a single engine tstep. ITL records
|
||||
only the gaps between streamed outputs; it does not add zero-duration gaps for
|
||||
tokens in the same output. TPOT instead amortizes the request's decoding time
|
||||
over every output token.
|
||||
|
||||

|
||||

|
||||
|
||||
In this example, the benchmark observes two 40 ms ITL samples. The three tokens
|
||||
in the second streamed output do not create additional ITL samples, so mean ITL
|
||||
is 40 ms. TPOT is `(180 ms - 100 ms) / (5 - 1) = 20 ms/token`.
|
||||
|
||||
#### Results Visualization
|
||||
|
||||
The `--plot-timeline` and `--plot-dataset-stats` can be used to generate respectively the requests completion timeline and dataset prompt and output tokens statistics, which can be useful for debugging purpose or for deeper analysis.
|
||||
|
||||
Reference in New Issue
Block a user