[docs] Add documentation for pynvvideocodec video decoding backend (#49066)
Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
This commit is contained in:
co-authored by
Cyrus Leung
parent
f98061ce6c
commit
5c7a7f9462
@@ -846,15 +846,17 @@ Full example: [examples/generate/multimodal/openai_chat_completion_client_for_mu
|
|||||||
vLLM decodes video bytes into frames using a selectable decoding backend. Five
|
vLLM decodes video bytes into frames using a selectable decoding backend. Five
|
||||||
backends are supported:
|
backends are supported:
|
||||||
|
|
||||||
- `opencv` (default): OpenCV-based decoder.
|
| Backend | Device | Description |
|
||||||
- `pyav`: PyAV decoder.
|
| --- | --- | --- |
|
||||||
- `torchcodec`: TorchCodec (PyTorch-native) decoder.
|
| `opencv` (default) | CPU | OpenCV-based decoder |
|
||||||
- `pynvvideocodec`: NVIDIA NVDEC-based decoder.
|
| `pyav` | CPU | PyAV decoder |
|
||||||
- `deepstream`: NVIDIA DeepStream NVDEC-based decoder.
|
| `torchcodec` | CPU | TorchCodec (PyTorch-native) decoder |
|
||||||
|
| `pynvvideocodec` | GPU | NVIDIA PyNvVideoCodec decoder |
|
||||||
|
| `deepstream` | GPU | NVIDIA DeepStream decoder |
|
||||||
|
|
||||||
The CPU backends are backed by FFmpeg. `torchcodec` lets you choose which FFmpeg
|
The three CPU backends are ultimately backed by FFmpeg. `torchcodec` lets you
|
||||||
version is used while `opencv` and `pyav` rely on whichever FFmpeg build they
|
choose which FFmpeg version is used while `opencv` and `pyav` rely on whichever
|
||||||
were linked against.
|
FFmpeg build they were linked against.
|
||||||
|
|
||||||
Select the backend by passing the `backend` parameter via `--media-io-kwargs`:
|
Select the backend by passing the `backend` parameter via `--media-io-kwargs`:
|
||||||
|
|
||||||
@@ -921,11 +923,55 @@ vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
|
|||||||
|
|
||||||
Works with common video formats like MP4 when using OpenCV backends.
|
Works with common video formats like MP4 when using OpenCV backends.
|
||||||
|
|
||||||
|
#### GPU Video Decoding with PyNvVideoCodec (NVDEC)
|
||||||
|
|
||||||
|
The `pynvvideocodec` backend uses NVIDIA NVDEC to decode the sampled video
|
||||||
|
frames on the GPU before copying them into host memory for multimodal
|
||||||
|
preprocessing. For workloads with large videos and relatively light inference,
|
||||||
|
such as video tagging, this can alleviate bottlenecks in CPU-based video
|
||||||
|
decoders.
|
||||||
|
|
||||||
|
!!! warning
|
||||||
|
[CUDA Multi-Process Service (MPS)](https://docs.nvidia.com/deploy/mps/quick-start.html)
|
||||||
|
is required when using this backend. Video decoding runs in the API server
|
||||||
|
process while model serving runs in the engine process, so multiple CUDA
|
||||||
|
processes share the same GPU. Configure and start MPS before starting vLLM.
|
||||||
|
|
||||||
|
You must also set a positive `--mm-ipc-gpu-memory-gb` value to reserve VRAM for
|
||||||
|
video decoding. vLLM carves this budget out of the memory available to the KV
|
||||||
|
cache and uses it to bound concurrent frontend decode allocations. If the
|
||||||
|
budget is exhausted, decode work waits instead of consuming the engine's VRAM
|
||||||
|
headroom and potentially causing an out-of-memory error while serving requests.
|
||||||
|
|
||||||
|
Select the backend with an environment variable and specify a workload-appropriate
|
||||||
|
VRAM budget. For example, to reserve 1 GiB:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export VLLM_VIDEO_LOADER_BACKEND=pynvvideocodec
|
||||||
|
vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
|
||||||
|
--mm-ipc-gpu-memory-gb 1
|
||||||
|
```
|
||||||
|
|
||||||
|
Alternatively, select it with `--media-io-kwargs`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
|
||||||
|
--media-io-kwargs '{"video": {"backend": "pynvvideocodec"}}' \
|
||||||
|
--mm-ipc-gpu-memory-gb 1
|
||||||
|
```
|
||||||
|
|
||||||
|
Choose a budget large enough for the largest sampled video that a single API
|
||||||
|
server process must decode. When using multiple API server processes, vLLM
|
||||||
|
divides the configured budget evenly among them.
|
||||||
|
|
||||||
|
For streaming video sources, use the DeepStream backend instead.
|
||||||
|
|
||||||
#### GPU Video Decoding with DeepStream (NVDEC)
|
#### GPU Video Decoding with DeepStream (NVDEC)
|
||||||
|
|
||||||
By default vLLM decodes video on the CPU. On NVIDIA GPUs you can instead decode
|
By default vLLM decodes video on the CPU. On NVIDIA GPUs you can instead decode
|
||||||
directly on the hardware video engine (NVDEC) with the DeepStream backend, which
|
directly on the hardware video engine (NVDEC) with the DeepStream backend, which
|
||||||
keeps decoding off the CPU and can significantly increase video throughput.
|
keeps decoding off the CPU and can significantly increase video throughput. It
|
||||||
|
is the recommended GPU backend for streaming video sources.
|
||||||
|
|
||||||
Install the backend (Linux x86-64 only):
|
Install the backend (Linux x86-64 only):
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user