Add Triton Backend for Unlimited-OCR R-SWA (#47102)
Signed-off-by: Dakai An <dakaian108@gmail.com>
This commit is contained in:
@@ -71,8 +71,11 @@ class UnlimitedOCRForCausalLMConfig(VerifyAndUpdateConfig):
|
||||
2. ``--attention-config '{"backend": "FLEX_ATTENTION"}'``
|
||||
→ FlexAttention R-SWA via Triton block mask.
|
||||
|
||||
3. ``--attention-config '{"backend": "auto"}'`` (or omitted)
|
||||
→ Auto-detect: FA4 if available (H20/H100 SM90), else FlexAttention.
|
||||
3. ``--attention-config '{"backend": "TRITON_ATTN"}'``
|
||||
→ Triton unified attention with an R-SWA decode mask.
|
||||
|
||||
4. ``--attention-config '{"backend": "auto"}'`` (or omitted)
|
||||
→ Auto-detect: FA4 if available (H20/H100 SM90), else TritonAttention.
|
||||
|
||||
Regardless of backend, prefix caching is disabled for this model: R-SWA
|
||||
decode-phase KV is not a pure causal function of the prefix (so decode
|
||||
@@ -96,7 +99,7 @@ class UnlimitedOCRForCausalLMConfig(VerifyAndUpdateConfig):
|
||||
attn_config.backend = (
|
||||
AttentionBackendEnum.FLASH_ATTN
|
||||
if fa4_available
|
||||
else AttentionBackendEnum.FLEX_ATTENTION
|
||||
else AttentionBackendEnum.TRITON_ATTN
|
||||
)
|
||||
logger.info(
|
||||
"Unlimited-OCR: auto-selected attention backend=%s (fa4_available=%s).",
|
||||
@@ -110,8 +113,8 @@ class UnlimitedOCRForCausalLMConfig(VerifyAndUpdateConfig):
|
||||
raise RuntimeError(
|
||||
"Unlimited-OCR: --attention-config backend=FLASH_ATTN "
|
||||
"requires FA4 (rswa_mask_mod), but FA4 is not available on "
|
||||
"this device/installation. Use backend=FLEX_ATTENTION or "
|
||||
"upgrade vllm-flash-attn."
|
||||
"this device/installation. Use backend=TRITON_ATTN or "
|
||||
"FLEX_ATTENTION, or upgrade vllm-flash-attn."
|
||||
)
|
||||
# On SM90 (H20), the default FA version is FA3 regardless of FA4
|
||||
# availability (FA4 is only auto-upgraded when head_size > 256).
|
||||
@@ -130,6 +133,11 @@ class UnlimitedOCRForCausalLMConfig(VerifyAndUpdateConfig):
|
||||
attn_config.flash_attn_version,
|
||||
)
|
||||
|
||||
elif attn_config.backend == AttentionBackendEnum.TRITON_ATTN:
|
||||
logger.info(
|
||||
"Unlimited-OCR: TritonAttention — R-SWA via unified attention mask."
|
||||
)
|
||||
|
||||
elif attn_config.backend == AttentionBackendEnum.FLEX_ATTENTION:
|
||||
logger.info(
|
||||
"Unlimited-OCR: FlexAttention — R-SWA via Triton block mask%s.",
|
||||
@@ -144,7 +152,7 @@ class UnlimitedOCRForCausalLMConfig(VerifyAndUpdateConfig):
|
||||
raise ValueError(
|
||||
f"Unlimited-OCR: unsupported attention backend "
|
||||
f"{attn_config.backend!r} for R-SWA. "
|
||||
"Use FLASH_ATTN (FA4) or FLEX_ATTENTION."
|
||||
"Use FLASH_ATTN (FA4), TRITON_ATTN or FLEX_ATTENTION."
|
||||
)
|
||||
|
||||
# R-SWA windows the *generated* tokens, so a decode-token's KV is not a
|
||||
|
||||
@@ -21,15 +21,11 @@ backbone architecture, FlexAttention for R-SWA, vision encoder backend, and
|
||||
Attention backend: the reference applies Reference Sliding Window Attention
|
||||
(R-SWA) -- the prompt/image tokens form a globally-visible prefix while the
|
||||
*generated* tokens additionally attend only a fixed sliding window (128) of
|
||||
recent tokens. We reproduce this (Level 1: full KV cache + custom mask) by
|
||||
forcing the language model onto the FlexAttention backend and installing an
|
||||
R-SWA ``mask_mod``. FlexAttention is the only backend able to express the
|
||||
"global prefix + sliding window" mask; FlashAttention-3 / Triton only support a
|
||||
uniform window (and additionally crash or compute incorrectly on this decoder's
|
||||
10-head,
|
||||
head_dim-128 shape), and FlashInfer's paged decode exposes no custom mask. The
|
||||
window size is published via ``model_config.rswa_window``, which the model
|
||||
runner reads to plumb per-request prefix lengths into the FlexAttention mask.
|
||||
recent tokens. We reproduce this with backend-specific custom masks: FA4 uses a
|
||||
``mask_mod``, FlexAttention uses a Triton block mask, and TritonAttention uses a
|
||||
unified-attention decode mask. FlashInfer's paged decode exposes no custom mask.
|
||||
The window size is published via ``model_config.rswa_window``, which the model
|
||||
runner reads to plumb per-request prefix lengths into the attention metadata.
|
||||
|
||||
The *vision encoder* (DeepEncoder's CLIP stage, head_dim 64) is unaffected and
|
||||
does not use R-SWA: it runs a single full-attention prefill pass. FlashAttention,
|
||||
|
||||
Reference in New Issue
Block a user