[Feature] Add VidCom2 video token pruning (#47750)
Signed-off-by: Benedikt Falk <bfalk@nvidia.com>
This commit is contained in:
@@ -350,6 +350,31 @@ Instead of NumPy arrays, you can also pass `'torch.Tensor'` instances, as shown
|
||||
|
||||
Full example: [examples/generate/multimodal/vision_language_offline.py](../../examples/generate/multimodal/vision_language_offline.py)
|
||||
|
||||
#### Video Token Pruning
|
||||
|
||||
For supported models, vLLM can prune video tokens after the vision encoder to
|
||||
reduce prefill time and KV cache usage, at some cost in accuracy. Set
|
||||
`--video-pruning-rate <q>` to prune the fraction `q` of video tokens from each
|
||||
video, and `--video-pruning-method` to choose the training-free algorithm:
|
||||
|
||||
- **`evs`** (Efficient Video Sampling, default): drops the tokens with the
|
||||
lowest temporal dissimilarity to the previous frame. The first frame is
|
||||
always fully retained.
|
||||
- **`vidcom2`** (Video Compression Commander): scores tokens by similarity to
|
||||
video-level and frame-level feature centers and gives distinctive frames a
|
||||
larger share of the budget. At least one token per frame is retained.
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-VL-8B-Instruct \
|
||||
--video-pruning-rate 0.75 --video-pruning-method vidcom2
|
||||
```
|
||||
|
||||
!!! note
|
||||
`evs` is supported by all models implementing multimodal pruning;
|
||||
`vidcom2` is currently supported by Qwen3-VL only. Unsupported combinations
|
||||
are rejected at startup. Enabling video pruning also disables encoder CUDA
|
||||
graphs, since the retained token count becomes data-dependent.
|
||||
|
||||
### Audio Inputs
|
||||
|
||||
You can pass a tuple `(array, sampling_rate)` to the `'audio'` field of the multi-modal dictionary.
|
||||
|
||||
Reference in New Issue
Block a user