From 4ee9702bee668a447e9983a6aefc16ebbc3ad32e Mon Sep 17 00:00:00 2001 From: Michael Goin Date: Sat, 1 Aug 2026 02:22:40 -0400 Subject: [PATCH] Add Understanding the Latency Metrics docs (#50600) Signed-off-by: mgoin --- ...ency-metrics-speculative-decoding-dark.svg | 77 +++++++++++++++++++ ...ncy-metrics-speculative-decoding-light.svg | 77 +++++++++++++++++++ docs/benchmarking/cli.md | 39 ++++++++++ 3 files changed, 193 insertions(+) create mode 100644 docs/assets/benchmarking/latency-metrics-speculative-decoding-dark.svg create mode 100644 docs/assets/benchmarking/latency-metrics-speculative-decoding-light.svg diff --git a/docs/assets/benchmarking/latency-metrics-speculative-decoding-dark.svg b/docs/assets/benchmarking/latency-metrics-speculative-decoding-dark.svg new file mode 100644 index 0000000000..5ff1742ebf --- /dev/null +++ b/docs/assets/benchmarking/latency-metrics-speculative-decoding-dark.svg @@ -0,0 +1,77 @@ + + Latency metrics with token bundling + The client receives token 1 at 100 milliseconds, tokens 2 through 4 together at 140 milliseconds, and token 5 at 180 milliseconds. The two observed ITL gaps average 40 milliseconds. TPOT divides the 80 millisecond generation span by the four tokens after the first, giving 20 milliseconds per token. + + + + + + + + + + + Latency with token bundling + 5 tokens · 3 stream outputs + + + + Request + + + 0 ms + + Output 1 + + + T1 + + + 100 ms + + Output 2 · bundled + + + + + T2 + T3 + T4 + + + 140 ms + + Output 3 + + + T5 + + + 180 ms + + + + TTFT 100 ms + + + ITL 40 ms + + ITL 40 ms + + + 80 ms generation span + + + + Mean ITL + (40 ms + 40 ms) ÷ 2 + 40 ms + 2 observed gaps between stream outputs + + + + TPOT + (180 − 100) ms ÷ (5 − 1) + 20 ms/token + 4 output tokens after the first + diff --git a/docs/assets/benchmarking/latency-metrics-speculative-decoding-light.svg b/docs/assets/benchmarking/latency-metrics-speculative-decoding-light.svg new file mode 100644 index 0000000000..06cb45a03f --- /dev/null +++ b/docs/assets/benchmarking/latency-metrics-speculative-decoding-light.svg @@ -0,0 +1,77 @@ + + Latency metrics with token bundling + The client receives token 1 at 100 milliseconds, tokens 2 through 4 together at 140 milliseconds, and token 5 at 180 milliseconds. The two observed ITL gaps average 40 milliseconds. TPOT divides the 80 millisecond generation span by the four tokens after the first, giving 20 milliseconds per token. + + + + + + + + + + + Latency with token bundling + 5 tokens · 3 stream outputs + + + + Request + + + 0 ms + + Output 1 + + + T1 + + + 100 ms + + Output 2 · bundled + + + + + T2 + T3 + T4 + + + 140 ms + + Output 3 + + + T5 + + + 180 ms + + + + TTFT 100 ms + + + ITL 40 ms + + ITL 40 ms + + + 80 ms generation span + + + + Mean ITL + (40 ms + 40 ms) ÷ 2 + 40 ms + 2 observed gaps between stream outputs + + + + TPOT + (180 − 100) ms ÷ (5 − 1) + 20 ms/token + 4 output tokens after the first + diff --git a/docs/benchmarking/cli.md b/docs/benchmarking/cli.md index 692cb4918b..23d37eb726 100644 --- a/docs/benchmarking/cli.md +++ b/docs/benchmarking/cli.md @@ -111,6 +111,45 @@ P99 ITL (ms): 8.39 ================================================== ``` +#### Understanding the Latency Metrics + +`vllm bench serve` measures latency at the benchmark client: + +!!! note + Metric terminology is not standardized across benchmarking tools. When + comparing results, use the measurement points and formulas rather than the + metric names alone. This section explains how we refer to them in vLLM. + +- **Time to first token (TTFT)** is the time from sending a request to receiving + its first streamed output. +- **Inter-token latency (ITL)** records the time between consecutive streamed + outputs. The reported ITL statistics aggregate these individual gaps across + all successful requests. +- **Time per output token (TPOT)** is calculated once per request, excluding the + first token, and then aggregated across requests: + + $$ + \text{TPOT} = + \frac{\text{end-to-end latency} - \text{TTFT}} + {\text{number of output tokens} - 1} + $$ + +With standard decoding, each streamed output usually contains one token, so ITL +and TPOT are typically similar. + +With speculative decoding, one streamed output can contain multiple tokens, +such as several accepted draft tokens within a single engine tstep. ITL records +only the gaps between streamed outputs; it does not add zero-duration gaps for +tokens in the same output. TPOT instead amortizes the request's decoding time +over every output token. + +![Latency metrics with bundled tokens (light theme)](../assets/benchmarking/latency-metrics-speculative-decoding-light.svg#only-light) +![Latency metrics with bundled tokens (dark theme)](../assets/benchmarking/latency-metrics-speculative-decoding-dark.svg#only-dark) + +In this example, the benchmark observes two 40 ms ITL samples. The three tokens +in the second streamed output do not create additional ITL samples, so mean ITL +is 40 ms. TPOT is `(180 ms - 100 ms) / (5 - 1) = 20 ms/token`. + #### Results Visualization The `--plot-timeline` and `--plot-dataset-stats` can be used to generate respectively the requests completion timeline and dataset prompt and output tokens statistics, which can be useful for debugging purpose or for deeper analysis.