[MM][Perf][CG] Support ViT full cudagraphs for mllama4 (#40660)
Signed-off-by: allgather <all2allops@gmail.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
This commit is contained in:
@@ -82,6 +82,7 @@ Models opt-in to encoder CUDA Graphs by implementing the [SupportsEncoderCudaGra
|
||||
|
||||
| Architecture | Models | CG for Image | CG for Video |
|
||||
| ------------ | ------ | ------------ | ------------ |
|
||||
| `Llama4ForConditionalGeneration` | `Llama 4` | ✅︎ | - |
|
||||
| `InternVLChatModel` | `InternVL3.5`, `InternVL3`, `InternVL2.5`, `InternVL2` | ✅︎ | ✅︎ |
|
||||
| `Qwen2VLForConditionalGeneration` | `Qwen2-VL` | ✅︎ | ✅︎ |
|
||||
| `Qwen2_5_VLForConditionalGeneration` | `Qwen2.5-VL` | ✅︎ | ✅︎ |
|
||||
@@ -114,6 +115,14 @@ vllm serve Qwen/Qwen3-VL-32B \
|
||||
--compilation-config '{"cudagraph_mm_encoder": true}'
|
||||
```
|
||||
|
||||
For `Llama 4` (image only):
|
||||
|
||||
```bash
|
||||
vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct \
|
||||
--limit-mm-per-prompt '{"image": 1}' \
|
||||
--compilation-config '{"cudagraph_mm_encoder": true}'
|
||||
```
|
||||
|
||||
With explicit budgets:
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user