[Frontend][Core][Spec Decode] Per-request acceptance stats in OpenAI API responses (#48915)

Signed-off-by: Matthew Kotila <7692737+matthewkotila@users.noreply.github.com>
Co-authored-by: Reed Meyerson <31574681+reed-meyerson@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matthew Kotila
2026-08-20 15:42:49 -07:00
committed by GitHub
co-authored by Reed Meyerson Claude Opus 4.8
parent 2f41c894e3
commit 7cfb97e337
25 changed files with 798 additions and 76 deletions
+9
View File
@@ -125,3 +125,12 @@ a single prompt's generation.
The `metrics` response field provides per-request values for a single request.
The `/metrics` Prometheus endpoint exposes server-level histograms (e.g.
`vllm:time_to_first_token_seconds`) that aggregate across all requests.
## Speculative Decoding Acceptance
When speculative decoding is enabled, per-request acceptance metrics
(mean acceptance length and the accepted-draft-length distribution) can be
returned via `--per-request-spec-decode-metrics`. They share this `metrics`
object as `metrics.speculative_decoding`, and — like the timing fields — are
reported only for single-sequence (`n == 1`) requests. See
[Per-Request Acceptance Metrics](speculative_decoding/acceptance_metrics.md).