[Bench] Add BFCL dataset for vllm bench serve tool-calling workloads (#42457)

Signed-off-by: Li Zhang <lzhanga@amazon.com>
Co-authored-by: Li Zhang <lzhanga@amazon.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
This commit is contained in:
Li
2026-06-09 23:59:18 -07:00
committed by GitHub
co-authored by Li Zhang Claude Chauncey
parent 7fdfa6441d
commit 89c6a41001
7 changed files with 666 additions and 4 deletions
+41
View File
@@ -405,6 +405,47 @@ vllm bench serve \
Available categories include `[high_entropy, mixed, low_entropy]`, where high entropy data contains unstructued data such as creative writing while low entropy data contains more structured data such as coding, more details are in the dataset card.
#### BFCL (Tool-Calling) Benchmark
The Berkeley Function Calling Leaderboard (BFCL) dataset measures serving
latency and throughput on realistic tool-calling traffic. Each request
carries a per-sample `tools` schema and chat history, so the server must
expose `/v1/chat/completions` with an auto-tool-choice parser enabled.
The benchmark client always uses the `openai-chat` backend.
Start a tool-parser-enabled server, then run the bench. For example, with
`gpt-oss-20b`:
```bash
# Server
vllm serve openai/gpt-oss-20b \
--enable-auto-tool-choice \
--tool-call-parser openai \
--reasoning-parser openai_gptoss
# Client
vllm bench serve \
--backend openai-chat \
--endpoint /v1/chat/completions \
--model openai/gpt-oss-20b \
--dataset-name hf \
--dataset-path gorilla-llm/Berkeley-Function-Calling-Leaderboard \
--bfcl-categories simple,live_simple,multiple \
--num-prompts 200
```
`--bfcl-categories` is a comma-separated list of BFCL v3 category names
(without the `BFCL_v3_` prefix or `.json` suffix). Defaults to
`simple,live_simple,multiple`. Other supported non-multi-turn categories
include `parallel`, `live_parallel`, `parallel_multiple`,
`live_parallel_multiple`, `irrelevance`, `live_irrelevance`,
`live_relevance`, `java`, `javascript`, and `rest`. Multi-turn categories
are not yet supported.
The dataset class normalizes BFCL's loose schema dialect (`dict` →
`object`, `float` → `number`, `tuple` → `array`, `any` → `string`) so
modern grammar backends accept the translated tool definitions.
#### Other HuggingFaceDataset Examples
```bash
+7
View File
@@ -504,6 +504,13 @@ Flags: `--tool-call-parser pythonic --chat-template {see_above}`
!!! warning
Llama's smaller models frequently fail to emit tool calls in the correct format. Results may vary depending on the model.
## Benchmarking Tool-Calling Performance
To measure serving latency and throughput on realistic tool-calling traffic,
use the BFCL (Berkeley Function Calling Leaderboard) dataset with
`vllm bench serve`. See the [BFCL benchmark example](../benchmarking/cli.md#bfcl-tool-calling-benchmark)
for the full server + client commands.
## How to Write a Tool Parser Plugin
A tool parser plugin is a Python file containing one or more ToolParser implementations. You can write a ToolParser similar to the `Hermes2ProToolParser` in [vllm/tool_parsers/hermes_tool_parser.py](../../vllm/tool_parsers/hermes_tool_parser.py).