[Frontend][Core][Spec Decode] Per-request acceptance stats in OpenAI API responses (#48915)
Signed-off-by: Matthew Kotila <7692737+matthewkotila@users.noreply.github.com> Co-authored-by: Reed Meyerson <31574681+reed-meyerson@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Reed Meyerson
Claude Opus 4.8
parent
2f41c894e3
commit
7cfb97e337
@@ -125,3 +125,12 @@ a single prompt's generation.
|
||||
The `metrics` response field provides per-request values for a single request.
|
||||
The `/metrics` Prometheus endpoint exposes server-level histograms (e.g.
|
||||
`vllm:time_to_first_token_seconds`) that aggregate across all requests.
|
||||
|
||||
## Speculative Decoding Acceptance
|
||||
|
||||
When speculative decoding is enabled, per-request acceptance metrics
|
||||
(mean acceptance length and the accepted-draft-length distribution) can be
|
||||
returned via `--per-request-spec-decode-metrics`. They share this `metrics`
|
||||
object as `metrics.speculative_decoding`, and — like the timing fields — are
|
||||
reported only for single-sequence (`n == 1`) requests. See
|
||||
[Per-Request Acceptance Metrics](speculative_decoding/acceptance_metrics.md).
|
||||
|
||||
Reference in New Issue
Block a user