[Feature] Add VidCom2 video token pruning (#47750)
Signed-off-by: Benedikt Falk <bfalk@nvidia.com>
This commit is contained in:
@@ -101,7 +101,7 @@ When `mm_encoder_tp_mode="data"`, the manager distributes images across TP ranks
|
||||
Following <https://github.com/vllm-project/vllm/pull/35963> (ViT full CUDA graph support for image inference), <https://github.com/vllm-project/vllm/pull/38061> extends the encoder CUDA graph framework to support video inference for Qwen3-VL. Previously, the CUDA graph capture/replay path only handled image inputs (`pixel_values` + `image_grid_thw`). Video inputs use different keys (`pixel_values_videos` + `video_grid_thw`) and require larger `cu_seqlens` buffers because each video item contributes multiple frames (`T` attention sequences). This PR generalizes the protocol and manager to handle both modalities through a single shared graph manager.
|
||||
|
||||
!!! note
|
||||
Video CUDA graphs are automatically disabled when EVS (Efficient Video Sampling) pruning is enabled, since EVS makes the token count data-dependent and incompatible with CUDA graph capture.
|
||||
Video CUDA graphs are automatically disabled when video token pruning (EVS or VidCom2) is enabled, since pruning makes the token count data-dependent and incompatible with CUDA graph capture.
|
||||
|
||||
Mixed inputs (image+video) per prompt are also supported now.
|
||||
|
||||
|
||||
@@ -350,6 +350,31 @@ Instead of NumPy arrays, you can also pass `'torch.Tensor'` instances, as shown
|
||||
|
||||
Full example: [examples/generate/multimodal/vision_language_offline.py](../../examples/generate/multimodal/vision_language_offline.py)
|
||||
|
||||
#### Video Token Pruning
|
||||
|
||||
For supported models, vLLM can prune video tokens after the vision encoder to
|
||||
reduce prefill time and KV cache usage, at some cost in accuracy. Set
|
||||
`--video-pruning-rate <q>` to prune the fraction `q` of video tokens from each
|
||||
video, and `--video-pruning-method` to choose the training-free algorithm:
|
||||
|
||||
- **`evs`** (Efficient Video Sampling, default): drops the tokens with the
|
||||
lowest temporal dissimilarity to the previous frame. The first frame is
|
||||
always fully retained.
|
||||
- **`vidcom2`** (Video Compression Commander): scores tokens by similarity to
|
||||
video-level and frame-level feature centers and gives distinctive frames a
|
||||
larger share of the budget. At least one token per frame is retained.
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-VL-8B-Instruct \
|
||||
--video-pruning-rate 0.75 --video-pruning-method vidcom2
|
||||
```
|
||||
|
||||
!!! note
|
||||
`evs` is supported by all models implementing multimodal pruning;
|
||||
`vidcom2` is currently supported by Qwen3-VL only. Unsupported combinations
|
||||
are rejected at startup. Enabling video pruning also disables encoder CUDA
|
||||
graphs, since the retained token count becomes data-dependent.
|
||||
|
||||
### Audio Inputs
|
||||
|
||||
You can pass a tuple `(array, sampling_rate)` to the `'audio'` field of the multi-modal dictionary.
|
||||
|
||||
Reference in New Issue
Block a user