[Mini-VLLM] Restore FlashInfer backend for FP8 KV & only probe quantization overrides that exist

FlashInfer is the only backend serving fp8_e4m3 KV cache on compute
capability 8.6; the SM86 cut removed it and the Triton fallback
refuses FP8 KV, crashing worker startup. Restore flashinfer.py and
the triton_flashinfer shim.

The SM86 cut removed the inc and deepseek_v4_fp8 loaders but
ModelConfig still probed them, crashing compressed-tensors
checkpoints with "Invalid quantization method: inc". Filter the
override probe list to the supported methods.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
2026-10-18 23:31:30 +07:00
co-authored by Hinata
parent e61dcb9143
commit f36850e389
3 changed files with 2819 additions and 1 deletions
+3 -1
View File
@@ -1392,7 +1392,9 @@ class ModelConfig:
# Any custom overrides will be in quantization_methods so we place
# them at the start of the list so custom overrides have preference
# over the built-in ones.
quantization_methods = quantization_methods + overrides
quantization_methods = quantization_methods + [
o for o in overrides if o in supported_quantization
]
# Detect which checkpoint is it
for name in quantization_methods:
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,22 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Triton image-mask and FlashInfer causal attention composite."""
from vllm.v1.attention.backends.composite import (
MMPrefixAttentionRouting,
create_composite_attention_backend,
)
from vllm.v1.attention.backends.flashinfer import FlashInferBackend
from vllm.v1.attention.backends.triton_attn import TritonAttentionBackend
TritonFlashInferBackend = create_composite_attention_backend(
TritonAttentionBackend,
FlashInferBackend,
name="TritonFlashInferBackend",
backend_name="TRITON_FLASHINFER",
module=__name__,
routing_policy=MMPrefixAttentionRouting,
head_sizes=(256, 512),
kernel_block_sizes=(64,),
device_major=10,
)