[Mini-VLLM] Restore FlashInfer backend for FP8 KV & only probe quantization overrides that exist
FlashInfer is the only backend serving fp8_e4m3 KV cache on compute capability 8.6; the SM86 cut removed it and the Triton fallback refuses FP8 KV, crashing worker startup. Restore flashinfer.py and the triton_flashinfer shim. The SM86 cut removed the inc and deepseek_v4_fp8 loaders but ModelConfig still probed them, crashing compressed-tensors checkpoints with "Invalid quantization method: inc". Filter the override probe list to the supported methods. Co-authored-by: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
@@ -1392,7 +1392,9 @@ class ModelConfig:
|
||||
# Any custom overrides will be in quantization_methods so we place
|
||||
# them at the start of the list so custom overrides have preference
|
||||
# over the built-in ones.
|
||||
quantization_methods = quantization_methods + overrides
|
||||
quantization_methods = quantization_methods + [
|
||||
o for o in overrides if o in supported_quantization
|
||||
]
|
||||
|
||||
# Detect which checkpoint is it
|
||||
for name in quantization_methods:
|
||||
|
||||
Executable
+2794
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,22 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Triton image-mask and FlashInfer causal attention composite."""
|
||||
|
||||
from vllm.v1.attention.backends.composite import (
|
||||
MMPrefixAttentionRouting,
|
||||
create_composite_attention_backend,
|
||||
)
|
||||
from vllm.v1.attention.backends.flashinfer import FlashInferBackend
|
||||
from vllm.v1.attention.backends.triton_attn import TritonAttentionBackend
|
||||
|
||||
TritonFlashInferBackend = create_composite_attention_backend(
|
||||
TritonAttentionBackend,
|
||||
FlashInferBackend,
|
||||
name="TritonFlashInferBackend",
|
||||
backend_name="TRITON_FLASHINFER",
|
||||
module=__name__,
|
||||
routing_policy=MMPrefixAttentionRouting,
|
||||
head_sizes=(256, 512),
|
||||
kernel_block_sizes=(64,),
|
||||
device_major=10,
|
||||
)
|
||||
Reference in New Issue
Block a user