[docs] Add documentation for pynvvideocodec video decoding backend (#49066)
Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
This commit is contained in:
co-authored by
Cyrus Leung
parent
f98061ce6c
commit
5c7a7f9462
@@ -846,15 +846,17 @@ Full example: [examples/generate/multimodal/openai_chat_completion_client_for_mu
|
||||
vLLM decodes video bytes into frames using a selectable decoding backend. Five
|
||||
backends are supported:
|
||||
|
||||
- `opencv` (default): OpenCV-based decoder.
|
||||
- `pyav`: PyAV decoder.
|
||||
- `torchcodec`: TorchCodec (PyTorch-native) decoder.
|
||||
- `pynvvideocodec`: NVIDIA NVDEC-based decoder.
|
||||
- `deepstream`: NVIDIA DeepStream NVDEC-based decoder.
|
||||
| Backend | Device | Description |
|
||||
| --- | --- | --- |
|
||||
| `opencv` (default) | CPU | OpenCV-based decoder |
|
||||
| `pyav` | CPU | PyAV decoder |
|
||||
| `torchcodec` | CPU | TorchCodec (PyTorch-native) decoder |
|
||||
| `pynvvideocodec` | GPU | NVIDIA PyNvVideoCodec decoder |
|
||||
| `deepstream` | GPU | NVIDIA DeepStream decoder |
|
||||
|
||||
The CPU backends are backed by FFmpeg. `torchcodec` lets you choose which FFmpeg
|
||||
version is used while `opencv` and `pyav` rely on whichever FFmpeg build they
|
||||
were linked against.
|
||||
The three CPU backends are ultimately backed by FFmpeg. `torchcodec` lets you
|
||||
choose which FFmpeg version is used while `opencv` and `pyav` rely on whichever
|
||||
FFmpeg build they were linked against.
|
||||
|
||||
Select the backend by passing the `backend` parameter via `--media-io-kwargs`:
|
||||
|
||||
@@ -921,11 +923,55 @@ vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
|
||||
|
||||
Works with common video formats like MP4 when using OpenCV backends.
|
||||
|
||||
#### GPU Video Decoding with PyNvVideoCodec (NVDEC)
|
||||
|
||||
The `pynvvideocodec` backend uses NVIDIA NVDEC to decode the sampled video
|
||||
frames on the GPU before copying them into host memory for multimodal
|
||||
preprocessing. For workloads with large videos and relatively light inference,
|
||||
such as video tagging, this can alleviate bottlenecks in CPU-based video
|
||||
decoders.
|
||||
|
||||
!!! warning
|
||||
[CUDA Multi-Process Service (MPS)](https://docs.nvidia.com/deploy/mps/quick-start.html)
|
||||
is required when using this backend. Video decoding runs in the API server
|
||||
process while model serving runs in the engine process, so multiple CUDA
|
||||
processes share the same GPU. Configure and start MPS before starting vLLM.
|
||||
|
||||
You must also set a positive `--mm-ipc-gpu-memory-gb` value to reserve VRAM for
|
||||
video decoding. vLLM carves this budget out of the memory available to the KV
|
||||
cache and uses it to bound concurrent frontend decode allocations. If the
|
||||
budget is exhausted, decode work waits instead of consuming the engine's VRAM
|
||||
headroom and potentially causing an out-of-memory error while serving requests.
|
||||
|
||||
Select the backend with an environment variable and specify a workload-appropriate
|
||||
VRAM budget. For example, to reserve 1 GiB:
|
||||
|
||||
```bash
|
||||
export VLLM_VIDEO_LOADER_BACKEND=pynvvideocodec
|
||||
vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
|
||||
--mm-ipc-gpu-memory-gb 1
|
||||
```
|
||||
|
||||
Alternatively, select it with `--media-io-kwargs`:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
|
||||
--media-io-kwargs '{"video": {"backend": "pynvvideocodec"}}' \
|
||||
--mm-ipc-gpu-memory-gb 1
|
||||
```
|
||||
|
||||
Choose a budget large enough for the largest sampled video that a single API
|
||||
server process must decode. When using multiple API server processes, vLLM
|
||||
divides the configured budget evenly among them.
|
||||
|
||||
For streaming video sources, use the DeepStream backend instead.
|
||||
|
||||
#### GPU Video Decoding with DeepStream (NVDEC)
|
||||
|
||||
By default vLLM decodes video on the CPU. On NVIDIA GPUs you can instead decode
|
||||
directly on the hardware video engine (NVDEC) with the DeepStream backend, which
|
||||
keeps decoding off the CPU and can significantly increase video throughput.
|
||||
keeps decoding off the CPU and can significantly increase video throughput. It
|
||||
is the recommended GPU backend for streaming video sources.
|
||||
|
||||
Install the backend (Linux x86-64 only):
|
||||
|
||||
|
||||
Reference in New Issue
Block a user