[feat] add torchcodec as audio loader and implement selective audio backend (#51826)
Signed-off-by: JaredforReal <w13431838023@gmail.com> Co-authored-by: saltman155 <saltman155@outlook.com> Co-authored-by: Isotr0py <Isotr0py@outlook.com>
This commit is contained in:
co-authored by
saltman155
Isotr0py
parent
131e0285f6
commit
6a039f465e
@@ -1204,6 +1204,38 @@ Full example: [examples/generate/multimodal/openai_chat_completion_client_for_mu
|
||||
export VLLM_AUDIO_FETCH_TIMEOUT=<timeout>
|
||||
```
|
||||
|
||||
#### Audio Decoding Backend
|
||||
|
||||
vLLM decodes audio bytes into waveforms using a selectable decoding backend:
|
||||
|
||||
| Backend | Description |
|
||||
| --- | --- |
|
||||
| `auto` (default) | torchcodec, falling back to soundfile, then PyAV |
|
||||
| `soundfile` | libsndfile only, no fallback |
|
||||
| `pyav` | PyAV (FFmpeg) only, no fallback |
|
||||
| `torchcodec` | TorchCodec (PyTorch-native) only, no fallback |
|
||||
|
||||
Select the backend per server via `--media-io-kwargs`:
|
||||
|
||||
```bash
|
||||
vllm serve mistralai/Voxtral-Mini-3B-2507 \
|
||||
--media-io-kwargs '{"audio": {"audio_backend": "soundfile"}}'
|
||||
```
|
||||
|
||||
!!! tip
|
||||
`pyav` drives FFmpeg through a per-frame Python generator, so under
|
||||
concurrency the Python/C crossings contend on the GIL. `torchcodec`
|
||||
decodes each stream in a single call that releases the GIL for its whole
|
||||
duration, which is why `auto` prefers it when many requests decode audio
|
||||
concurrently, such as when audio tracks are extracted from video.
|
||||
|
||||
!!! note
|
||||
`torchcodec` ships as a requirement on CUDA, CPU and XPU builds. On other
|
||||
platforms (e.g. ROCm, TPU) `auto` falls back to the soundfile → PyAV
|
||||
chain unless you install it manually. torchcodec also links against a
|
||||
system FFmpeg installation; if FFmpeg is missing, `auto` falls back the
|
||||
same way.
|
||||
|
||||
### Embedding Inputs
|
||||
|
||||
To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model,
|
||||
|
||||
Reference in New Issue
Block a user