Add TorchCodec as a video decoding backend (#46609)

Signed-off-by: Nicolas Hug <contact@nicolas-hug.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
This commit is contained in:
Nicolas Hug
2026-07-06 19:58:51 -07:00
committed by GitHub
co-authored by Claude Isotr0py
parent a4f019fa25
commit 700e882eab
9 changed files with 300 additions and 12 deletions
+38
View File
@@ -816,6 +816,44 @@ Full example: [examples/generate/multimodal/openai_chat_completion_client_for_mu
export VLLM_VIDEO_FETCH_TIMEOUT=<timeout>
```
#### Video Decoding Backend
vLLM decodes video bytes into frames using a selectable decoding backend. Three
backends are supported:
- `opencv` (default): OpenCV-based decoder.
- `pyav`: PyAV decoder.
- `torchcodec`: TorchCodec (PyTorch-native) decoder.
All three backends are ultimately backed by FFmpeg. `torchcodec` lets
you choose which FFmpeg version is used while `opencv` and `pyav` rely on
whichever FFmpeg build they were linked against.
Select the backend by passing the `backend` parameter via `--media-io-kwargs`:
```bash
vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
--media-io-kwargs '{"video": {"backend": "torchcodec"}}'
```
**TorchCodec-specific parameters:**
The following parameters only apply to the `torchcodec` backend:
- `num_ffmpeg_threads`: Number of FFmpeg decoding threads. `0` (default) relies
on the FFmpeg default, which is `min(cpu_count + 1, 16)`. This allows you to
control thread over-subscription.
- `seek_mode`: Seek mode for the decoder. `"exact"` (default) guarantees
frame-accurate sampling by scanning the file when the decoder is created.
`"approximate"` skips that scan for faster decoder creation, at the cost of
relying on the file's metadata (which may yield less accurate seeking).
```bash
# Example: TorchCodec with approximate seek mode and 4 FFmpeg threads
vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
--media-io-kwargs '{"video": {"backend": "torchcodec", "seek_mode": "approximate", "num_ffmpeg_threads": 4}}'
```
#### Video Frame Recovery
For improved robustness when processing potentially corrupted or truncated video files, vLLM supports optional frame recovery using a dynamic window forward-scan approach. When enabled, if a target frame fails to load during sequential reading, the next successfully grabbed frame (before the next target frame) will be used in its place.