[Dependency] Upgrade FlashInfer version to 0.7.0 (#58069)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Yongye Zhu <zyy1102000@gmail.com> Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
This commit is contained in:
co-authored by
Codex
Yongye Zhu
Kevin H. Luu
parent
a1b6763df6
commit
44bf011f44
@@ -79,9 +79,9 @@ def flashinfer_sampler_supported() -> bool:
|
||||
|
||||
Returns False (with appropriate logging) when ``VLLM_USE_FLASHINFER_SAMPLER``
|
||||
is 0, when the platform isn't CUDA, when the GPU's compute capability is
|
||||
unsupported, or when FlashInfer cannot JIT-compile for the current
|
||||
GPU/CUDA toolchain. Raises ``RuntimeError`` if the user explicitly opted in
|
||||
via the env var but FlashInfer is unavailable.
|
||||
unsupported, when the GPU has 16 or fewer SMs, or when FlashInfer cannot
|
||||
JIT-compile for the current GPU/CUDA toolchain. Raises ``RuntimeError`` if
|
||||
the user explicitly opted in via the env var but FlashInfer is unavailable.
|
||||
|
||||
Assumes flashinfer is installed, as guaranteed by ``requirements/cuda.txt``;
|
||||
otherwise importing the FlashInfer backend below raises ``ImportError``.
|
||||
@@ -107,6 +107,16 @@ def flashinfer_sampler_supported() -> bool:
|
||||
unsupported_reason = (
|
||||
f"unsupported compute capability {capability.as_version_str()}"
|
||||
)
|
||||
elif (
|
||||
num_sms := current_platform.num_compute_units(
|
||||
torch.accelerator.current_device_index()
|
||||
)
|
||||
) <= 16:
|
||||
# FlashInfer 0.7+ rejects multi-CTA top-k masking on <=16 SMs because
|
||||
# its cross-CTA software barrier cannot guarantee forward progress.
|
||||
unsupported_reason = (
|
||||
f"top-k masking requires more than 16 SMs; device has {num_sms}"
|
||||
)
|
||||
else:
|
||||
unsupported_reason = _flashinfer_jit_unsupported_reason(capability)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user