Support mnnvl all2allv from Flashinfer (#21003)
Signed-off-by: Shu Wang <shuw@nvidia.com> Signed-off-by: Shu Wang. <shuw@nvidia.com> Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com> Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com> Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
This commit is contained in:
co-authored by
Tyler Michael Smith
Tyler Michael Smith
parent
2dda3e35d0
commit
54e42b72db
+5
-2
@@ -156,7 +156,8 @@ if TYPE_CHECKING:
|
||||
VLLM_ALL2ALL_BACKEND: Literal["naive", "pplx",
|
||||
"deepep_high_throughput",
|
||||
"deepep_low_latency",
|
||||
"allgather_reducescatter"] = \
|
||||
"allgather_reducescatter",
|
||||
"flashinfer_all2allv"] = \
|
||||
"allgather_reducescatter"
|
||||
VLLM_MAX_TOKENS_PER_EXPERT_FP4_MOE: int = 163840
|
||||
VLLM_TOOL_PARSE_REGEX_TIMEOUT_SECONDS: int = 1
|
||||
@@ -1209,12 +1210,14 @@ environment_variables: dict[str, Callable[[], Any]] = {
|
||||
# - "pplx": use pplx kernels
|
||||
# - "deepep_high_throughput", use deepep high-throughput kernels
|
||||
# - "deepep_low_latency", use deepep low-latency kernels
|
||||
# - "flashinfer_all2allv", use flashinfer alltoallv kernels for mnnvl
|
||||
"VLLM_ALL2ALL_BACKEND":
|
||||
env_with_choices("VLLM_ALL2ALL_BACKEND", "allgather_reducescatter",
|
||||
["naive", "pplx",
|
||||
"deepep_high_throughput",
|
||||
"deepep_low_latency",
|
||||
"allgather_reducescatter"]),
|
||||
"allgather_reducescatter",
|
||||
"flashinfer_all2allv"]),
|
||||
|
||||
# Flashinfer MoE backend for vLLM's fused Mixture-of-Experts support.
|
||||
# Both require compute capability 10.0 or above.
|
||||
|
||||
Reference in New Issue
Block a user