[Mini-VLLM] Restore FlashInfer backend for FP8 KV on SM86
FlashInfer is the only backend serving fp8_e4m3 KV cache on compute capability 8.6; the SM86 cut removed it and the Triton fallback refuses FP8 KV, crashing worker startup. Restore flashinfer.py and the triton_flashinfer shim. Co-authored-by: Hinata
This commit is contained in:
Executable
+2794
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,22 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Triton image-mask and FlashInfer causal attention composite."""
|
||||
|
||||
from vllm.v1.attention.backends.composite import (
|
||||
MMPrefixAttentionRouting,
|
||||
create_composite_attention_backend,
|
||||
)
|
||||
from vllm.v1.attention.backends.flashinfer import FlashInferBackend
|
||||
from vllm.v1.attention.backends.triton_attn import TritonAttentionBackend
|
||||
|
||||
TritonFlashInferBackend = create_composite_attention_backend(
|
||||
TritonAttentionBackend,
|
||||
FlashInferBackend,
|
||||
name="TritonFlashInferBackend",
|
||||
backend_name="TRITON_FLASHINFER",
|
||||
module=__name__,
|
||||
routing_policy=MMPrefixAttentionRouting,
|
||||
head_sizes=(256, 512),
|
||||
kernel_block_sizes=(64,),
|
||||
device_major=10,
|
||||
)
|
||||
Reference in New Issue
Block a user