[feat] add torchcodec as audio loader and implement selective audio backend (#51826)

Signed-off-by: JaredforReal <w13431838023@gmail.com>
Co-authored-by: saltman155 <saltman155@outlook.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
This commit is contained in:
Jared Wen
2026-09-04 16:16:10 +00:00
committed by GitHub
co-authored by saltman155 Isotr0py
parent 131e0285f6
commit 6a039f465e
4 changed files with 444 additions and 65 deletions
+32
View File
@@ -1204,6 +1204,38 @@ Full example: [examples/generate/multimodal/openai_chat_completion_client_for_mu
export VLLM_AUDIO_FETCH_TIMEOUT=<timeout>
```
#### Audio Decoding Backend
vLLM decodes audio bytes into waveforms using a selectable decoding backend:
| Backend | Description |
| --- | --- |
| `auto` (default) | torchcodec, falling back to soundfile, then PyAV |
| `soundfile` | libsndfile only, no fallback |
| `pyav` | PyAV (FFmpeg) only, no fallback |
| `torchcodec` | TorchCodec (PyTorch-native) only, no fallback |
Select the backend per server via `--media-io-kwargs`:
```bash
vllm serve mistralai/Voxtral-Mini-3B-2507 \
--media-io-kwargs '{"audio": {"audio_backend": "soundfile"}}'
```
!!! tip
`pyav` drives FFmpeg through a per-frame Python generator, so under
concurrency the Python/C crossings contend on the GIL. `torchcodec`
decodes each stream in a single call that releases the GIL for its whole
duration, which is why `auto` prefers it when many requests decode audio
concurrently, such as when audio tracks are extracted from video.
!!! note
`torchcodec` ships as a requirement on CUDA, CPU and XPU builds. On other
platforms (e.g. ROCm, TPU) `auto` falls back to the soundfile → PyAV
chain unless you install it manually. torchcodec also links against a
system FFmpeg installation; if FFmpeg is missing, `auto` falls back the
same way.
### Embedding Inputs
To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model,