Implement custom dataset class for ASR benchmarking (#41576)
Signed-off-by: Yasmin Moslem <48152713+ymoslem@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
parent
630492da30
commit
28ee78af54
@@ -39,7 +39,8 @@ th {
|
||||
| Spec Bench | ✅ | ✅ | `wget https://raw.githubusercontent.com/hemingkx/Spec-Bench/refs/heads/main/data/spec_bench/question.jsonl` |
|
||||
| SPEED-Bench | ✅ | ✅ | `curl -LsSf https://raw.githubusercontent.com/NVIDIA-NeMo/Skills/refs/heads/main/nemo_skills/dataset/speed-bench/prepare.py \| python3 -` |
|
||||
| Custom | ✅ | ✅ | Local file: `data.jsonl` |
|
||||
| Custom MM | ✅ | ✅ | Local file: `mm_data.jsonl` |
|
||||
| Custom Audio | ✅ | ✅ | Local file: `audio_data.jsonl` |
|
||||
| Custom Image | ✅ | ✅ | Local file: `image_data.jsonl` |
|
||||
|
||||
Legend:
|
||||
|
||||
@@ -142,7 +143,7 @@ Example output:  which do not require a "prompt" field; and multimodal audio-text chat models (e.g. Qwen2-Audio). Since these model types require different arguments at inference, we are giving two examples.
|
||||
|
||||
- **Example 1: Whisper**
|
||||
|
||||
Whisper is a dedicated ASR encoder-decoder model, so it uses `--backend openai-audio` and `--endpoint /v1/audio/transcriptions`.
|
||||
|
||||
```bash
|
||||
# start server
|
||||
vllm serve openai/whisper-tiny
|
||||
```
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--model openai/whisper-tiny \
|
||||
--backend openai-audio \
|
||||
--endpoint /v1/audio/transcriptions \
|
||||
--dataset-name custom_audio \
|
||||
--dataset-path audio_data.jsonl \
|
||||
--no-oversample \
|
||||
--custom-output-len 256 \
|
||||
--save-result \
|
||||
--save-detailed \
|
||||
--result-filename whisper_bench.json
|
||||
```
|
||||
|
||||
- **Example 2: Qwen2-Audio**
|
||||
|
||||
Qwen2-Audio is a multimodal chat model that can do ASR and speech analysis, so it uses `--backend openai-chat`, and `--endpoint /v1/chat/completions`. It also requires `--enable-multimodal-chat` to enable multimodal chat transformation.
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--model Qwen/Qwen2-Audio-7B-Instruct \
|
||||
--backend openai-chat \
|
||||
--endpoint /v1/chat/completions \
|
||||
--dataset-name custom_audio \
|
||||
--dataset-path audio_data.jsonl \
|
||||
--no-oversample \
|
||||
--custom-output-len 256 \
|
||||
--enable-multimodal-chat \
|
||||
--save-result \
|
||||
--save-detailed \
|
||||
--result-filename qwen_bench.json
|
||||
```
|
||||
|
||||
#### Custom Image Dataset
|
||||
|
||||
If the image dataset you want to benchmark is not supported yet in vLLM, then you can benchmark on it using `CustomImageDataset`. At inference time, use the option `--dataset-name custom_image`. Your data needs to be in the `.jsonl` format and needs to have "prompt" and "image_files" fields per entry, e.g., `image_data.jsonl`:
|
||||
|
||||
```json
|
||||
{"prompt": "How many animals are present in the given image?", "image_files": ["/path/to/image/folder/horsepony.jpg"]}
|
||||
@@ -193,8 +247,8 @@ vllm bench serve--save-result --save-detailed \
|
||||
--backend openai-chat \
|
||||
--model Qwen/Qwen2-VL-7B-Instruct \
|
||||
--endpoint /v1/chat/completions \
|
||||
--dataset-name custom_mm \
|
||||
--dataset-path <path-to-your-mm-data-jsonl> \
|
||||
--dataset-name custom_image \
|
||||
--dataset-path <path-to-your-image-data-jsonl> \
|
||||
--allowed-local-media-path /path/to/image/folder
|
||||
```
|
||||
|
||||
|
||||
Reference in New Issue
Block a user