[Attention] Add Triton/FlashInfer composite for multimodal prefix attention (#56305)

Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
This commit is contained in:
Matthew Bonanni
2026-09-13 14:29:41 +00:00
committed by GitHub
co-authored by Codex Lucas Wilkinson Lucas Wilkinson OpenAI Codex
parent 8cf9de9080
commit 82a85dc1d2
15 changed files with 1095 additions and 3 deletions
+49
View File
@@ -56,6 +56,55 @@ llm = LLM(
## Backend Selection Behavior
### Triton/FlashAttention Composite
On Hopper, `TRITON_FLASH_ATTN` is preferred for compatible multimodal-prefix
configurations. It uses Triton when the current queries need bidirectional
image attention and FlashAttention for causal text prefills and decode, with a
shared KV cache. The causal child must resolve to FA4. This happens
automatically for FA4-only shapes such as head size 512 and models whose
version policy promotes all layers to FA4, including Gemma 4. A standalone
head-size-256 configuration whose version policy selects FA3 falls back to
Triton for the whole layer.
No attention override is needed for supported multimodal models:
```bash
vllm serve google/gemma-4-31B-it
```
### Triton/FlashInfer Composite
On Blackwell, `TRITON_FLASHINFER` is preferred for compatible
multimodal-prefix configurations, including Gemma 4 with BF16 or FP8 KV cache. It uses
Triton when a batch's current queries require bidirectional image attention,
and FlashInfer for causal text prefills and decode. Historical image tokens
alone do not select Triton.
This instantiates `create_composite_attention_backend` from
`vllm/v1/attention/backends/composite.py` with Triton,
FlashInfer, and `MMPrefixAttentionRouting`. The reusable factory owns child
implementations, metadata dispatch, compatible cache requirements, and workspace
sharing. The routing policy selects the child and defines graph-capture safety.
Other combinations can reuse the same machinery.
The backend supports head dimensions 256/512, FP16/BF16 and FP8 KV cache,
and 64-token kernel pages with a head-major cache layout. TRTLLM handles causal
attention at both head dimensions; its hdim512 kernels do not support 128-token
pages.
For Gemma 4, full CUDA graphs cover single-token batches; multi-token batches
use the non-full-graph execution path. Models whose image masks extend beyond
the sliding window cannot use full attention graphs with this composite.
Context parallelism, R-SWA, attention sinks,
and adaptive verification are not supported by this composite.
It can also be selected explicitly:
```bash
vllm serve google/gemma-4-31B-it \
--attention-backend TRITON_FLASHINFER
```
### Manual Selection
When you explicitly set a backend via `--attention-backend` or `AttentionConfig`: