[Bench] Add BFCL dataset for vllm bench serve tool-calling workloads (#42457)
Signed-off-by: Li Zhang <lzhanga@amazon.com> Co-authored-by: Li Zhang <lzhanga@amazon.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
This commit is contained in:
co-authored by
Li Zhang
Claude
Chauncey
parent
7fdfa6441d
commit
89c6a41001
@@ -405,6 +405,47 @@ vllm bench serve \
|
||||
|
||||
Available categories include `[high_entropy, mixed, low_entropy]`, where high entropy data contains unstructued data such as creative writing while low entropy data contains more structured data such as coding, more details are in the dataset card.
|
||||
|
||||
#### BFCL (Tool-Calling) Benchmark
|
||||
|
||||
The Berkeley Function Calling Leaderboard (BFCL) dataset measures serving
|
||||
latency and throughput on realistic tool-calling traffic. Each request
|
||||
carries a per-sample `tools` schema and chat history, so the server must
|
||||
expose `/v1/chat/completions` with an auto-tool-choice parser enabled.
|
||||
The benchmark client always uses the `openai-chat` backend.
|
||||
|
||||
Start a tool-parser-enabled server, then run the bench. For example, with
|
||||
`gpt-oss-20b`:
|
||||
|
||||
```bash
|
||||
# Server
|
||||
vllm serve openai/gpt-oss-20b \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser openai \
|
||||
--reasoning-parser openai_gptoss
|
||||
|
||||
# Client
|
||||
vllm bench serve \
|
||||
--backend openai-chat \
|
||||
--endpoint /v1/chat/completions \
|
||||
--model openai/gpt-oss-20b \
|
||||
--dataset-name hf \
|
||||
--dataset-path gorilla-llm/Berkeley-Function-Calling-Leaderboard \
|
||||
--bfcl-categories simple,live_simple,multiple \
|
||||
--num-prompts 200
|
||||
```
|
||||
|
||||
`--bfcl-categories` is a comma-separated list of BFCL v3 category names
|
||||
(without the `BFCL_v3_` prefix or `.json` suffix). Defaults to
|
||||
`simple,live_simple,multiple`. Other supported non-multi-turn categories
|
||||
include `parallel`, `live_parallel`, `parallel_multiple`,
|
||||
`live_parallel_multiple`, `irrelevance`, `live_irrelevance`,
|
||||
`live_relevance`, `java`, `javascript`, and `rest`. Multi-turn categories
|
||||
are not yet supported.
|
||||
|
||||
The dataset class normalizes BFCL's loose schema dialect (`dict` →
|
||||
`object`, `float` → `number`, `tuple` → `array`, `any` → `string`) so
|
||||
modern grammar backends accept the translated tool definitions.
|
||||
|
||||
#### Other HuggingFaceDataset Examples
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user