Support mnnvl all2allv from Flashinfer (#21003)

Signed-off-by: Shu Wang <shuw@nvidia.com>
Signed-off-by: Shu Wang. <shuw@nvidia.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
This commit is contained in:
Shu Wang
2025-09-24 14:38:16 -04:00
committed by GitHub
co-authored by Tyler Michael Smith Tyler Michael Smith
parent 2dda3e35d0
commit 54e42b72db
10 changed files with 410 additions and 40 deletions
+5 -2
View File
@@ -156,7 +156,8 @@ if TYPE_CHECKING:
VLLM_ALL2ALL_BACKEND: Literal["naive", "pplx",
"deepep_high_throughput",
"deepep_low_latency",
"allgather_reducescatter"] = \
"allgather_reducescatter",
"flashinfer_all2allv"] = \
"allgather_reducescatter"
VLLM_MAX_TOKENS_PER_EXPERT_FP4_MOE: int = 163840
VLLM_TOOL_PARSE_REGEX_TIMEOUT_SECONDS: int = 1
@@ -1209,12 +1210,14 @@ environment_variables: dict[str, Callable[[], Any]] = {
# - "pplx": use pplx kernels
# - "deepep_high_throughput", use deepep high-throughput kernels
# - "deepep_low_latency", use deepep low-latency kernels
# - "flashinfer_all2allv", use flashinfer alltoallv kernels for mnnvl
"VLLM_ALL2ALL_BACKEND":
env_with_choices("VLLM_ALL2ALL_BACKEND", "allgather_reducescatter",
["naive", "pplx",
"deepep_high_throughput",
"deepep_low_latency",
"allgather_reducescatter"]),
"allgather_reducescatter",
"flashinfer_all2allv"]),
# Flashinfer MoE backend for vLLM's fused Mixture-of-Experts support.
# Both require compute capability 10.0 or above.