Add HumanEval and GSM8K benchmarks to datasets (#42648)

Signed-off-by: southfreebird <yvorott@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
This commit is contained in:
Sergei Skvortsov
2026-05-15 13:01:21 -07:00
committed by GitHub
co-authored by Michael Goin
parent b2c58ee942
commit 9a7a273dfe
2 changed files with 155 additions and 0 deletions
+22
View File
@@ -34,6 +34,8 @@ th {
| HuggingFace-AIMO | ✅ | ✅ | `AI-MO/aimo-validation-aime`, `AI-MO/NuminaMath-1.5`, `AI-MO/NuminaMath-CoT` |
| HuggingFace-Other | ✅ | ✅ | `lmms-lab/LLaVA-OneVision-Data`, `Aeala/ShareGPT_Vicuna_unfiltered` |
| HuggingFace-MTBench | ✅ | ✅ | `philschmid/mt-bench` |
| HuggingFace-HumanEval | ✅ | ✅ | `openai/openai_humaneval` |
| HuggingFace-GSM8K | ✅ | ✅ | `openai/gsm8k` |
| HuggingFace-Blazedit | ✅ | ✅ | `vdaita/edit_5k_char`, `vdaita/edit_10k_char` |
| HuggingFace-ASR | ✅ | ✅ | `openslr/librispeech_asr`, `facebook/voxpopuli`, `LIUM/tedlium`, `edinburghcstr/ami`, `speechcolab/gigaspeech`, `kensho/spgispeech` |
| Spec Bench | ✅ | ✅ | `wget https://raw.githubusercontent.com/hemingkx/Spec-Bench/refs/heads/main/data/spec_bench/question.jsonl` |
@@ -443,6 +445,26 @@ vllm bench serve \
--num-prompts 80
```
`openai/openai_humaneval`:
``` bash
vllm bench serve \
--model NousResearch/Hermes-3-Llama-3.1-8B \
--dataset-name hf \
--dataset-path openai/openai_humaneval \
--num-prompts 80
```
`openai/gsm8k`:
``` bash
vllm bench serve \
--model NousResearch/Hermes-3-Llama-3.1-8B \
--dataset-name hf \
--dataset-path openai/gsm8k \
--num-prompts 80
```
`vdaita/edit_5k_char` or `vdaita/edit_10k_char`:
``` bash