[Attention] Add Triton/FlashInfer composite for multimodal prefix attention (#56305)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: OpenAI Codex <codex@openai.com>
This commit is contained in:
co-authored by
Codex
Lucas Wilkinson
Lucas Wilkinson
OpenAI Codex
parent
8cf9de9080
commit
82a85dc1d2
@@ -56,6 +56,55 @@ llm = LLM(
|
||||
|
||||
## Backend Selection Behavior
|
||||
|
||||
### Triton/FlashAttention Composite
|
||||
|
||||
On Hopper, `TRITON_FLASH_ATTN` is preferred for compatible multimodal-prefix
|
||||
configurations. It uses Triton when the current queries need bidirectional
|
||||
image attention and FlashAttention for causal text prefills and decode, with a
|
||||
shared KV cache. The causal child must resolve to FA4. This happens
|
||||
automatically for FA4-only shapes such as head size 512 and models whose
|
||||
version policy promotes all layers to FA4, including Gemma 4. A standalone
|
||||
head-size-256 configuration whose version policy selects FA3 falls back to
|
||||
Triton for the whole layer.
|
||||
|
||||
No attention override is needed for supported multimodal models:
|
||||
|
||||
```bash
|
||||
vllm serve google/gemma-4-31B-it
|
||||
```
|
||||
|
||||
### Triton/FlashInfer Composite
|
||||
|
||||
On Blackwell, `TRITON_FLASHINFER` is preferred for compatible
|
||||
multimodal-prefix configurations, including Gemma 4 with BF16 or FP8 KV cache. It uses
|
||||
Triton when a batch's current queries require bidirectional image attention,
|
||||
and FlashInfer for causal text prefills and decode. Historical image tokens
|
||||
alone do not select Triton.
|
||||
|
||||
This instantiates `create_composite_attention_backend` from
|
||||
`vllm/v1/attention/backends/composite.py` with Triton,
|
||||
FlashInfer, and `MMPrefixAttentionRouting`. The reusable factory owns child
|
||||
implementations, metadata dispatch, compatible cache requirements, and workspace
|
||||
sharing. The routing policy selects the child and defines graph-capture safety.
|
||||
Other combinations can reuse the same machinery.
|
||||
|
||||
The backend supports head dimensions 256/512, FP16/BF16 and FP8 KV cache,
|
||||
and 64-token kernel pages with a head-major cache layout. TRTLLM handles causal
|
||||
attention at both head dimensions; its hdim512 kernels do not support 128-token
|
||||
pages.
|
||||
For Gemma 4, full CUDA graphs cover single-token batches; multi-token batches
|
||||
use the non-full-graph execution path. Models whose image masks extend beyond
|
||||
the sliding window cannot use full attention graphs with this composite.
|
||||
Context parallelism, R-SWA, attention sinks,
|
||||
and adaptive verification are not supported by this composite.
|
||||
|
||||
It can also be selected explicitly:
|
||||
|
||||
```bash
|
||||
vllm serve google/gemma-4-31B-it \
|
||||
--attention-backend TRITON_FLASHINFER
|
||||
```
|
||||
|
||||
### Manual Selection
|
||||
|
||||
When you explicitly set a backend via `--attention-backend` or `AttentionConfig`:
|
||||
|
||||
Reference in New Issue
Block a user