[Feature] Add VidCom2 video token pruning (#47750)

Signed-off-by: Benedikt Falk <bfalk@nvidia.com>
This commit is contained in:
nvbfalk
2026-07-28 03:20:17 +00:00
committed by GitHub
parent 7aea73d83d
commit 60915c972c
17 changed files with 421 additions and 48 deletions
+1 -1
View File
@@ -101,7 +101,7 @@ When `mm_encoder_tp_mode="data"`, the manager distributes images across TP ranks
Following <https://github.com/vllm-project/vllm/pull/35963> (ViT full CUDA graph support for image inference), <https://github.com/vllm-project/vllm/pull/38061> extends the encoder CUDA graph framework to support video inference for Qwen3-VL. Previously, the CUDA graph capture/replay path only handled image inputs (`pixel_values` + `image_grid_thw`). Video inputs use different keys (`pixel_values_videos` + `video_grid_thw`) and require larger `cu_seqlens` buffers because each video item contributes multiple frames (`T` attention sequences). This PR generalizes the protocol and manager to handle both modalities through a single shared graph manager.
!!! note
Video CUDA graphs are automatically disabled when EVS (Efficient Video Sampling) pruning is enabled, since EVS makes the token count data-dependent and incompatible with CUDA graph capture.
Video CUDA graphs are automatically disabled when video token pruning (EVS or VidCom2) is enabled, since pruning makes the token count data-dependent and incompatible with CUDA graph capture.
Mixed inputs (image+video) per prompt are also supported now.
+25
View File
@@ -350,6 +350,31 @@ Instead of NumPy arrays, you can also pass `'torch.Tensor'` instances, as shown
Full example: [examples/generate/multimodal/vision_language_offline.py](../../examples/generate/multimodal/vision_language_offline.py)
#### Video Token Pruning
For supported models, vLLM can prune video tokens after the vision encoder to
reduce prefill time and KV cache usage, at some cost in accuracy. Set
`--video-pruning-rate <q>` to prune the fraction `q` of video tokens from each
video, and `--video-pruning-method` to choose the training-free algorithm:
- **`evs`** (Efficient Video Sampling, default): drops the tokens with the
lowest temporal dissimilarity to the previous frame. The first frame is
always fully retained.
- **`vidcom2`** (Video Compression Commander): scores tokens by similarity to
video-level and frame-level feature centers and gives distinctive frames a
larger share of the budget. At least one token per frame is retained.
```bash
vllm serve Qwen/Qwen3-VL-8B-Instruct \
--video-pruning-rate 0.75 --video-pruning-method vidcom2
```
!!! note
`evs` is supported by all models implementing multimodal pruning;
`vidcom2` is currently supported by Qwen3-VL only. Unsupported combinations
are rejected at startup. Enabling video pruning also disables encoder CUDA
graphs, since the retained token count becomes data-dependent.
### Audio Inputs
You can pass a tuple `(array, sampling_rate)` to the `'audio'` field of the multi-modal dictionary.