[Benchmark] Add Responses API backend to vllm bench serve (#54628)

Signed-off-by: QHarshil <harshil_c@hotmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
This commit is contained in:
Harshil
2026-09-28 05:04:11 +00:00
committed by GitHub
co-authored by Claude Chauncey
parent 0da126676e
commit af7f9488c2
5 changed files with 682 additions and 2 deletions
+40
View File
@@ -596,6 +596,46 @@ vllm bench serve \
--max-concurrency 512
```
#### Responses API Benchmark
The `openai-responses` backend benchmarks vLLM's
[Responses API](../serving/online_serving/openai_compatible_server.md#responses-api)
directly, instead of routing the same workload through `/v1/chat/completions`.
```bash
# Server
vllm serve openai/gpt-oss-20b
# Client
vllm bench serve \
--backend openai-responses \
--endpoint /v1/responses \
--model openai/gpt-oss-20b \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 200
```
Reasoning deltas (`response.reasoning_text.delta`) count towards TTFT and ITL,
because the server is already decoding tokens when it emits them. Only output
text (`response.output_text.delta`) is collected as the generated text, which
matches how the `openai-chat` backend treats `DeltaMessage.reasoning`. End-to-end
latency stops at the last token event, so `latency - ttft` equals `sum(itl)` and
TPOT is comparable with the other endpoints. The input and output token counts
come from the usage block on `response.completed`, which is still required: a
stream that ends without a terminal event is reported as a failed request.
The backend measures one streamed text-generation request per prompt. Built-in
tools, MCP, and multi-turn state via `previous_response_id` are out of scope.
All sampling parameter flags are supported except `--min-p`, which the
Responses API does not accept.
!!! warning
Do not pass `--extra-body '{"include_reasoning": false}'` when benchmarking
a reasoning model. The server still generates reasoning tokens but emits no
events for them, so the whole reasoning phase is absorbed into TTFT.
#### Running With Sampling Parameters
When using OpenAI-compatible backends such as `vllm`, optional sampling