[Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
Production runs one static model behind the OpenAI-compatible API. Every adapter path and non-OpenAI HTTP surface is unreachable there. - Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora CLI group, and the LoRA tests. - Drop the plugin loader in vllm/plugins. Platform resolution is now CUDA-only and the general/stat-logger plugin hooks are gone. - Delete the speech-to-text and realtime entrypoints, the scale-out render routes, and vllm/config/speech_to_text.py. GenerationTask is generate-only. - Cut DecodeBenchConnector and the disagg-prefill doc and image. The Multi, Offloading and Example connectors stay; production uses the CPU KV offload tier. - Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake arch gate, declarations and op registrations. The CUDA build never defined the guard and no Python caller existed. - Delete the orphan Mistral and Pixtral chat fixtures and examples/. - Fix _check_model: removing the LoRA early return made every request fail with "The model ... does not exist. Valid aliases: ...". Verified on an RTX 3090: import sweep reports 1242 modules with 7 known optional-dependency failures; a live server answers chat, completions, responses, tokenize and detokenize with zero log errors and 404 for the removed surfaces; collection yields 8558 tests with the 3 pre-existing errors; ruff adds no new finding; the incremental _C build succeeds. Co-authored-by: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
@@ -800,6 +800,3 @@ class B12xExperts(mk.FusedMoEExpertsModular):
|
||||
output=output,
|
||||
unit_scale_contract=self._quant_mode == "w4a16",
|
||||
)
|
||||
|
||||
def moe_sum(self, input: torch.Tensor, output: torch.Tensor) -> None:
|
||||
raise NotImplementedError("LoRA is not supported for B12xExperts")
|
||||
|
||||
@@ -1300,7 +1300,6 @@ class FusedMoEConfig:
|
||||
max_num_tokens: int = SchedulerConfig.DEFAULT_MAX_NUM_BATCHED_TOKENS_FOR_BATCHED_DP
|
||||
elastic_ep_max_dp_size: int | None = None
|
||||
has_bias: bool = False
|
||||
is_lora_enabled: bool = False
|
||||
has_hash_routing: bool = False
|
||||
# Routes are adjusted between routing and expert execution.
|
||||
require_decomposed_backend: bool = False
|
||||
|
||||
@@ -307,7 +307,6 @@ class SubstitutedRoutedExperts(RoutedExperts):
|
||||
not isinstance(self.quant_method, UnquantizedFusedMoEMethod),
|
||||
),
|
||||
("monolithic MoE backends", self.quant_method.is_monolithic),
|
||||
("MoE LoRA", moe_config.is_lora_enabled),
|
||||
(
|
||||
"EPLB",
|
||||
parallel_config.enable_eplb
|
||||
@@ -352,7 +351,6 @@ class SubstitutedRoutedExperts(RoutedExperts):
|
||||
ckpt_up_proj_name or self.ckpt_up_proj_name,
|
||||
num_experts=self.expert_substitution.num_logical_experts,
|
||||
routed_experts_prefix="",
|
||||
lora_base_layer_prefix=self.lora_base_layer_prefix,
|
||||
)
|
||||
|
||||
def forward_modular(
|
||||
|
||||
@@ -387,8 +387,3 @@ class FlashInferExperts(mk.FusedMoEExpertsModular):
|
||||
use_mxfp8_act_scaling=use_mxfp8_act_scaling,
|
||||
use_w4_group_scaling=use_w4_group_scaling,
|
||||
)
|
||||
|
||||
def moe_sum(self, input: torch.Tensor, output: torch.Tensor) -> None:
|
||||
# No support for LoRA in flashinfer_cutlass_fused_moe.
|
||||
# See TODOs in flashinfer functions runMoe and runMoeMinLatency.
|
||||
raise NotImplementedError("LoRA is not supported for flashinfer_cutlass_moe")
|
||||
|
||||
@@ -1,72 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
import torch
|
||||
|
||||
from vllm.lora.punica_wrapper.punica_base import PunicaWrapperBase
|
||||
|
||||
|
||||
@dataclass
|
||||
class MoELoRAContext:
|
||||
"""Carries all LoRA state for one MoE forward pass.
|
||||
|
||||
Built by FusedMoEWithLoRA.forward() and propagated explicitly through the
|
||||
modular kernel path (FusedMoEKernel -> FusedMoEExpertsModular.apply) so
|
||||
that TritonExperts.apply() can compute the LoRA contribution inline,
|
||||
replacing the decorator-based monkey-patch approach.
|
||||
"""
|
||||
|
||||
# LoRA weight tensors (same shapes as FusedMoEWithLoRA attributes)
|
||||
w13_lora_a_stacked: tuple[torch.Tensor, ...]
|
||||
w13_lora_b_stacked: tuple[torch.Tensor, ...]
|
||||
w2_lora_a_stacked: tuple[torch.Tensor, ...]
|
||||
w2_lora_b_stacked: tuple[torch.Tensor, ...]
|
||||
|
||||
# (max_loras + 1,) int32; slot 0 is the "no-adapter" sentinel
|
||||
adapter_enabled: torch.Tensor
|
||||
|
||||
# Metadata
|
||||
max_loras: int
|
||||
top_k: int
|
||||
w13_num_slices: int # 2 = gated (gate + up), 1 = non-gated or 3D-fused
|
||||
fully_sharded: bool
|
||||
tp_rank: int
|
||||
tp_size: int
|
||||
local_num_experts: int
|
||||
|
||||
punica_wrapper: PunicaWrapperBase
|
||||
|
||||
# Whether VLLM_TUNED_CONFIG_FOLDER is set; selects get_lora_op_configs vs
|
||||
# try_get_optimal_moe_lora_config for Triton kernel tile configs.
|
||||
use_tuned_config: bool
|
||||
|
||||
# Shared MoE LoRA: w13 lora_A and w2 lora_B are shared across all experts
|
||||
# (stored collapsed with expert-dim 1). When True, LoRAExpertsMixin expands
|
||||
# them to local_num_experts via a stride-0 view before the kernel.
|
||||
enable_moe_shared_loras: bool = False
|
||||
|
||||
# Optional dual-stream support for overlapping each (base GEMM, LoRA)
|
||||
# pair. When aux_stream is None, the experts.apply() path runs the
|
||||
# original sequential schedule. When set, base GEMM runs on the default
|
||||
# stream and the LoRA fast-path writes the delta into a fresh buffer on
|
||||
# aux_stream, which the default stream sums in afterwards.
|
||||
# Events are paired one-per-overlap-pair: events[0,1] for w13,
|
||||
# events[2,3] for w2, so the two pairs do not race on the same event.
|
||||
aux_stream: torch.cuda.Stream | None = None
|
||||
events: tuple[torch.cuda.Event, ...] | None = None
|
||||
|
||||
# Per-rank token→LoRA mapping after EP dispatch. Set by
|
||||
# FusedMoEPrepareAndFinalizeModular.prepare() when EP+LoRA is active, read
|
||||
# by LoRAExpertsMixin helpers in place of punica_wrapper's global mapping.
|
||||
# None means no dispatch happened (non-EP path), in which case callers
|
||||
# fall back to punica_wrapper.token_mapping_meta.
|
||||
local_token_lora_mapping: torch.Tensor | None = None
|
||||
|
||||
# Original unquantized hidden states, stashed by the modular kernel
|
||||
# before the prepare step potentially quantizes them. Used by
|
||||
# apply_w13_lora so the LoRA kernel sees correct-magnitude activations
|
||||
# instead of raw quantized values that are missing the activation scale.
|
||||
# Set per forward pass; None until the modular kernel writes it.
|
||||
original_hidden_states: torch.Tensor | None = None
|
||||
@@ -1,159 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
import torch
|
||||
|
||||
from vllm.model_executor.layers.fused_moe.activation import MoEActivation
|
||||
from vllm.model_executor.layers.fused_moe.config import FusedMoEConfig
|
||||
from vllm.model_executor.layers.fused_moe.experts.lora_context import MoELoRAContext
|
||||
|
||||
|
||||
class LoRAExpertsMixin:
|
||||
"""Mixin for FusedMoEExpertsModular subclasses that natively handle
|
||||
MoELoRAContext inside their apply() implementation.
|
||||
|
||||
Mixing this class in:
|
||||
- Flips supports_lora() to True so _can_fused_experts_support lets
|
||||
LoRA through the gate check.
|
||||
- Stashes a MoELoRAContext on the experts instance via
|
||||
set_lora_context(), which apply() consumes from self._lora_context.
|
||||
- Provides apply_w13_lora / apply_w2_lora helpers that dispatch to
|
||||
the PunicaWrapper kernels.
|
||||
|
||||
The helper methods are pure functions of their inputs; all required
|
||||
state is on lora_context or passed as arguments.
|
||||
"""
|
||||
|
||||
moe_config: FusedMoEConfig
|
||||
_lora_context: MoELoRAContext | None = None
|
||||
|
||||
def set_lora_context(self, ctx: MoELoRAContext) -> None:
|
||||
self._lora_context = ctx
|
||||
|
||||
@staticmethod
|
||||
def supports_lora() -> bool:
|
||||
return True
|
||||
|
||||
def apply_w13_lora(
|
||||
self,
|
||||
lora_context: MoELoRAContext,
|
||||
*,
|
||||
y: torch.Tensor,
|
||||
x: torch.Tensor,
|
||||
topk_ids: torch.Tensor,
|
||||
topk_weights: torch.Tensor,
|
||||
expert_map: torch.Tensor | None,
|
||||
w1: torch.Tensor,
|
||||
w2: torch.Tensor,
|
||||
num_tokens: int,
|
||||
top_k_num: int,
|
||||
add_inputs: bool = True,
|
||||
swap_w13_slices: bool = False,
|
||||
) -> tuple[
|
||||
torch.Tensor | None,
|
||||
torch.Tensor | None,
|
||||
torch.Tensor | None,
|
||||
torch.Tensor | None,
|
||||
]:
|
||||
w13_lora_a_stacked = lora_context.w13_lora_a_stacked
|
||||
w13_lora_b_stacked = lora_context.w13_lora_b_stacked
|
||||
if lora_context.enable_moe_shared_loras:
|
||||
# w13 lora_A is shared across experts (collapsed expert-dim 1);
|
||||
# broadcast to local_num_experts via a stride-0 view. The kernel
|
||||
# derives num_experts from shape[1] and indexes via stride(1)==0.
|
||||
w13_lora_a_stacked = tuple(
|
||||
a.expand(-1, lora_context.local_num_experts, -1, -1)
|
||||
for a in w13_lora_a_stacked
|
||||
)
|
||||
if swap_w13_slices:
|
||||
# The expand kernel writes slice j into the j-th half of y's last
|
||||
# dim. Reversing the (gate, up) slice tuples makes it emit
|
||||
# [up, gate] order directly -- used by the FlashInfer trtllm path,
|
||||
# whose SwiGLU expects the up half first, to avoid an out-of-place
|
||||
# concat swap afterwards.
|
||||
w13_lora_a_stacked = w13_lora_a_stacked[::-1]
|
||||
w13_lora_b_stacked = w13_lora_b_stacked[::-1]
|
||||
interleave_w13 = (
|
||||
lora_context.w13_num_slices == 2
|
||||
and self.moe_config.activation == MoEActivation.SWIGLUOAI
|
||||
)
|
||||
lora_output = torch.zeros_like(y) if interleave_w13 else y
|
||||
result = lora_context.punica_wrapper.add_lora_w13(
|
||||
lora_output,
|
||||
x,
|
||||
w13_lora_a_stacked,
|
||||
w13_lora_b_stacked,
|
||||
topk_ids,
|
||||
topk_weights,
|
||||
expert_map,
|
||||
w1,
|
||||
w2,
|
||||
num_tokens,
|
||||
top_k_num,
|
||||
lora_context.max_loras,
|
||||
lora_context.adapter_enabled,
|
||||
lora_context.local_num_experts,
|
||||
lora_context.top_k,
|
||||
lora_context.w13_num_slices,
|
||||
lora_context.fully_sharded,
|
||||
lora_context.use_tuned_config,
|
||||
add_inputs=add_inputs or interleave_w13,
|
||||
token_lora_mapping=lora_context.local_token_lora_mapping,
|
||||
)
|
||||
if interleave_w13:
|
||||
# Map gate/up slices onto even/odd output channels without a copy.
|
||||
gate_up_delta = lora_output.unflatten(-1, (2, -1))
|
||||
gate_up_output = y.unflatten(-1, (-1, 2)).transpose(-1, -2)
|
||||
if add_inputs:
|
||||
gate_up_output.add_(gate_up_delta)
|
||||
else:
|
||||
gate_up_output.copy_(gate_up_delta)
|
||||
return result
|
||||
|
||||
def apply_w2_lora(
|
||||
self,
|
||||
lora_context: MoELoRAContext,
|
||||
*,
|
||||
y: torch.Tensor,
|
||||
x: torch.Tensor,
|
||||
topk_weights: torch.Tensor,
|
||||
sorted_token_ids_lora: torch.Tensor | None,
|
||||
expert_ids_lora: torch.Tensor | None,
|
||||
num_tokens_post_padded_lora: torch.Tensor | None,
|
||||
token_lora_mapping: torch.Tensor | None,
|
||||
num_tokens: int,
|
||||
w1: torch.Tensor,
|
||||
w2: torch.Tensor,
|
||||
top_k_num: int,
|
||||
add_inputs: bool = True,
|
||||
) -> None:
|
||||
w2_lora_b_stacked = lora_context.w2_lora_b_stacked
|
||||
if lora_context.enable_moe_shared_loras:
|
||||
# w2 lora_B is shared across experts (collapsed expert-dim 1);
|
||||
# broadcast to local_num_experts via a stride-0 view.
|
||||
w2_lora_b_stacked = tuple(
|
||||
b.expand(-1, lora_context.local_num_experts, -1, -1)
|
||||
for b in w2_lora_b_stacked
|
||||
)
|
||||
lora_context.punica_wrapper.add_lora_w2(
|
||||
y,
|
||||
x,
|
||||
lora_context.w2_lora_a_stacked,
|
||||
w2_lora_b_stacked,
|
||||
topk_weights,
|
||||
sorted_token_ids_lora,
|
||||
expert_ids_lora,
|
||||
num_tokens_post_padded_lora,
|
||||
token_lora_mapping,
|
||||
num_tokens,
|
||||
w1,
|
||||
w2,
|
||||
top_k_num,
|
||||
lora_context.max_loras,
|
||||
lora_context.adapter_enabled,
|
||||
lora_context.top_k,
|
||||
lora_context.fully_sharded,
|
||||
lora_context.tp_rank,
|
||||
lora_context.use_tuned_config,
|
||||
add_inputs=add_inputs,
|
||||
)
|
||||
@@ -21,9 +21,6 @@ from vllm.model_executor.layers.fused_moe.config import (
|
||||
FusedMoEParallelConfig,
|
||||
FusedMoEQuantConfig,
|
||||
)
|
||||
from vllm.model_executor.layers.fused_moe.experts.lora_experts_mixin import (
|
||||
LoRAExpertsMixin,
|
||||
)
|
||||
from vllm.model_executor.layers.fused_moe.moe_align_block_size import (
|
||||
batched_moe_align_block_size,
|
||||
moe_align_block_size,
|
||||
@@ -673,7 +670,7 @@ class MarlinExpertsBase(mk.FusedMoEExpertsModular):
|
||||
return E, M, N, K, topk
|
||||
|
||||
|
||||
class MarlinExperts(LoRAExpertsMixin, MarlinExpertsBase):
|
||||
class MarlinExperts(MarlinExpertsBase):
|
||||
"""Marlin-based fused MoE expert implementation."""
|
||||
|
||||
def finalize_weight_and_reduce_impl(self) -> mk.TopKWeightAndReduce:
|
||||
@@ -736,121 +733,7 @@ class MarlinExperts(LoRAExpertsMixin, MarlinExpertsBase):
|
||||
assert self.w1_scale is not None
|
||||
assert self.w2_scale is not None
|
||||
|
||||
ctx = self._lora_context
|
||||
if ctx is None:
|
||||
fused_marlin_moe(
|
||||
hidden_states=hidden_states,
|
||||
w1=w1,
|
||||
w2=w2,
|
||||
bias1=self.w1_bias,
|
||||
bias2=self.w2_bias,
|
||||
w1_scale=self.w1_scale,
|
||||
w2_scale=self.w2_scale,
|
||||
topk_weights=topk_weights,
|
||||
topk_ids=topk_ids,
|
||||
global_scale1=self.g1_alphas,
|
||||
global_scale2=self.g2_alphas,
|
||||
input_global_scale1=self.a1_gscale,
|
||||
input_global_scale2=self.a2_gscale,
|
||||
w1_zeros=self.w1_zp,
|
||||
w2_zeros=self.w2_zp,
|
||||
quant_type_id=self.quant_type_id,
|
||||
apply_router_weight_on_input=apply_router_weight_on_input,
|
||||
global_num_experts=global_num_experts,
|
||||
activation=activation,
|
||||
activation_func=self.activation,
|
||||
activation_config=self.activation_config,
|
||||
moe_sum=self.moe_sum,
|
||||
expert_map=expert_map,
|
||||
output=output,
|
||||
# Workspaces are swapped in workspace_shapes() to account for proper
|
||||
# output buffer allocation. Please refer to workspace_shapes().
|
||||
intermediate_cache13=workspace2,
|
||||
intermediate_cache2=workspace13,
|
||||
input_dtype=self.input_dtype,
|
||||
)
|
||||
return
|
||||
|
||||
# LoRA path: wrap activation_func and moe_sum to inject LoRA at the
|
||||
# two natural injection points.
|
||||
#
|
||||
# Marlin uses moe_align_block_size (same as TritonExperts) so
|
||||
# intermediate_cache1 is indexed by flat (token, expert) pair index,
|
||||
# which is compatible with add_lora_fused_moe's scatter mechanism.
|
||||
|
||||
M = hidden_states.size(0)
|
||||
top_k_num = topk_ids.size(1)
|
||||
lora_state: dict = {}
|
||||
|
||||
def activation_with_lora(
|
||||
act_enum: MoEActivation,
|
||||
act_output: torch.Tensor,
|
||||
act_input: torch.Tensor,
|
||||
*,
|
||||
topk_ids: torch.Tensor | None = None,
|
||||
expert_map: torch.Tensor | None = None,
|
||||
) -> None:
|
||||
# act_input = intermediate_cache1 (M*topk, 2N for gated)
|
||||
# act_output = intermediate_cache2 (M*topk, N)
|
||||
|
||||
(
|
||||
sorted_token_ids_lora,
|
||||
expert_ids_lora,
|
||||
num_tokens_post_padded_lora,
|
||||
token_lora_mapping,
|
||||
) = self.apply_w13_lora(
|
||||
ctx,
|
||||
y=act_input,
|
||||
x=hidden_states,
|
||||
topk_ids=topk_ids,
|
||||
topk_weights=topk_weights,
|
||||
expert_map=expert_map,
|
||||
w1=w1,
|
||||
w2=w2,
|
||||
num_tokens=M,
|
||||
top_k_num=top_k_num,
|
||||
)
|
||||
lora_state.update(
|
||||
{
|
||||
"sorted": sorted_token_ids_lora,
|
||||
"eids": expert_ids_lora,
|
||||
"npad": num_tokens_post_padded_lora,
|
||||
"tlm": token_lora_mapping,
|
||||
}
|
||||
)
|
||||
self.activation(
|
||||
act_enum,
|
||||
act_output,
|
||||
act_input,
|
||||
topk_ids=topk_ids,
|
||||
expert_map=expert_map,
|
||||
)
|
||||
lora_state["cache2"] = act_output
|
||||
|
||||
def moe_sum_with_lora(
|
||||
moe_out: torch.Tensor,
|
||||
out: torch.Tensor,
|
||||
topk_ids: torch.Tensor,
|
||||
expert_map: torch.Tensor | None,
|
||||
) -> None:
|
||||
# moe_out shape: (M, topk, K)
|
||||
self.apply_w2_lora(
|
||||
ctx,
|
||||
y=moe_out,
|
||||
x=lora_state["cache2"],
|
||||
topk_weights=topk_weights,
|
||||
sorted_token_ids_lora=lora_state["sorted"],
|
||||
expert_ids_lora=lora_state["eids"],
|
||||
num_tokens_post_padded_lora=lora_state["npad"],
|
||||
token_lora_mapping=lora_state["tlm"],
|
||||
num_tokens=M,
|
||||
w1=w1,
|
||||
w2=w2,
|
||||
top_k_num=top_k_num,
|
||||
)
|
||||
self.moe_sum(moe_out, out, topk_ids, expert_map)
|
||||
|
||||
return fused_marlin_moe(
|
||||
fused_marlin_moe(
|
||||
hidden_states=hidden_states,
|
||||
w1=w1,
|
||||
w2=w2,
|
||||
@@ -870,11 +753,13 @@ class MarlinExperts(LoRAExpertsMixin, MarlinExpertsBase):
|
||||
apply_router_weight_on_input=apply_router_weight_on_input,
|
||||
global_num_experts=global_num_experts,
|
||||
activation=activation,
|
||||
activation_func=activation_with_lora,
|
||||
activation_func=self.activation,
|
||||
activation_config=self.activation_config,
|
||||
moe_sum=moe_sum_with_lora,
|
||||
moe_sum=self.moe_sum,
|
||||
expert_map=expert_map,
|
||||
output=output,
|
||||
# Workspaces are swapped in workspace_shapes() to account for proper
|
||||
# output buffer allocation. Please refer to workspace_shapes().
|
||||
intermediate_cache13=workspace2,
|
||||
intermediate_cache2=workspace13,
|
||||
input_dtype=self.input_dtype,
|
||||
|
||||
@@ -374,10 +374,6 @@ class Nvfp4QuantizationEmulationTritonExperts(TritonExperts):
|
||||
# Used in experts/triton_moe.py and passed to moe_kernel_quantize_input.
|
||||
return self.a1_gscale
|
||||
|
||||
@staticmethod
|
||||
def supports_lora() -> bool:
|
||||
return False
|
||||
|
||||
@staticmethod
|
||||
def is_supported_config(
|
||||
cls: type[mk.FusedMoEExperts],
|
||||
@@ -386,8 +382,6 @@ class Nvfp4QuantizationEmulationTritonExperts(TritonExperts):
|
||||
activation_key: QuantKey | None,
|
||||
activation_format: mk.FusedMoEActivationFormat,
|
||||
) -> tuple[bool, str | None]:
|
||||
if moe_config.is_lora_enabled:
|
||||
return False, "kernel does not support LoRA"
|
||||
if moe_config.has_bias:
|
||||
return False, "kernel does not support bias"
|
||||
|
||||
|
||||
@@ -15,9 +15,6 @@ from vllm.model_executor.layers.fused_moe.config import (
|
||||
FusedMoEParallelConfig,
|
||||
FusedMoEQuantConfig,
|
||||
)
|
||||
from vllm.model_executor.layers.fused_moe.experts.lora_experts_mixin import (
|
||||
LoRAExpertsMixin,
|
||||
)
|
||||
from vllm.model_executor.layers.fused_moe.fused_moe import (
|
||||
_prepare_expert_assignment,
|
||||
invoke_fused_moe_triton_kernel,
|
||||
@@ -59,10 +56,9 @@ from vllm.model_executor.layers.quantization.utils.quant_utils import (
|
||||
)
|
||||
from vllm.platforms import current_platform
|
||||
from vllm.triton_utils import tl
|
||||
from vllm.utils.multi_stream_utils import maybe_execute_in_parallel
|
||||
|
||||
|
||||
class TritonExperts(LoRAExpertsMixin, mk.FusedMoEExpertsModular):
|
||||
class TritonExperts(mk.FusedMoEExpertsModular):
|
||||
"""Triton-based fused MoE expert implementation."""
|
||||
|
||||
@staticmethod
|
||||
@@ -104,16 +100,6 @@ class TritonExperts(LoRAExpertsMixin, mk.FusedMoEExpertsModular):
|
||||
def activation_format() -> mk.FusedMoEActivationFormat:
|
||||
return mk.FusedMoEActivationFormat.Standard
|
||||
|
||||
@property
|
||||
def expects_unquantized_inputs(self) -> bool:
|
||||
# Defer activation quantization to apply() only when LoRA is active AND
|
||||
# tokens are dispatched across ranks (DP+EP all2all).
|
||||
return (
|
||||
self._lora_context is not None
|
||||
and self.quant_dtype is not None
|
||||
and self.moe_config.moe_parallel_config.use_all2all_kernels
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _supports_current_device() -> bool:
|
||||
return current_platform.is_cuda_alike() or current_platform.is_xpu()
|
||||
@@ -279,16 +265,13 @@ class TritonExperts(LoRAExpertsMixin, mk.FusedMoEExpertsModular):
|
||||
torch.int8,
|
||||
]
|
||||
|
||||
# We declared expects_unquantized_inputs (LoRA + DP/EP all2all), so the
|
||||
# prepare step deferred activation quantization to this kernel:
|
||||
# `hidden_states` arrives unquantized. Keep the unquantized tensor for
|
||||
# the LoRA shrink input and quantize a copy here for the base GEMM
|
||||
# (mirrors what the prepare step would have done, but after the
|
||||
# all-gather so the layout matches the gathered topk_ids / token map).
|
||||
lora_unquantized_hidden_states: torch.Tensor | None = None
|
||||
# The prepare step deferred activation quantization to this kernel:
|
||||
# `hidden_states` arrives unquantized. Quantize a copy here for the
|
||||
# base GEMM (mirrors what the prepare step would have done, but after
|
||||
# the all-gather so the layout matches the gathered topk_ids / token
|
||||
# map).
|
||||
if self.expects_unquantized_inputs:
|
||||
assert a1q_scale is None
|
||||
lora_unquantized_hidden_states = hidden_states
|
||||
hidden_states, a1q_scale = moe_kernel_quantize_input(
|
||||
hidden_states,
|
||||
self.a1_scale,
|
||||
@@ -343,7 +326,7 @@ class TritonExperts(LoRAExpertsMixin, mk.FusedMoEExpertsModular):
|
||||
# Their rows in the workspace caches are then never written, so every
|
||||
# consumer of those caches must be row-local: a dynamic per-tensor
|
||||
# scale for the second GEMM's activation would take its amax over the
|
||||
# untouched rows, and LoRA aligns the full `topk_ids` itself.
|
||||
# untouched rows.
|
||||
a2_scale_is_global = (
|
||||
self.quant_dtype is not None
|
||||
and a2_scale is None
|
||||
@@ -352,7 +335,6 @@ class TritonExperts(LoRAExpertsMixin, mk.FusedMoEExpertsModular):
|
||||
)
|
||||
skip_invalid = (
|
||||
expert_map is not None
|
||||
and self._lora_context is None
|
||||
and not a2_scale_is_global
|
||||
)
|
||||
|
||||
@@ -373,34 +355,6 @@ class TritonExperts(LoRAExpertsMixin, mk.FusedMoEExpertsModular):
|
||||
)
|
||||
)
|
||||
|
||||
# LoRA w13: applied to intermediate_cache1 before activation. When
|
||||
# the LoRA layer requested a dual-stream schedule, we run base w13
|
||||
# GEMM on the default stream and the LoRA fast-path on aux_stream;
|
||||
# the LoRA writes its delta into a fresh zero buffer (add_inputs=
|
||||
# False) and we sum it into intermediate_cache1 after both finish.
|
||||
#
|
||||
# The LoRA shrink kernel needs unquantized, gathered-layout
|
||||
# activations. When activation quant was deferred to this kernel
|
||||
# (expects_unquantized_inputs), the input we quantized above is exactly
|
||||
# that, so use it directly. Otherwise fall back to the context stash
|
||||
# (e.g. weight-only quant), guarding on a row-count match so a
|
||||
# DP-gathered layout never indexes a local stash out of bounds.
|
||||
sorted_token_ids_lora = None
|
||||
expert_ids_lora = None
|
||||
num_tokens_post_padded_lora = None
|
||||
token_lora_mapping = None
|
||||
lora_context = self._lora_context
|
||||
if lora_unquantized_hidden_states is not None:
|
||||
lora_x = lora_unquantized_hidden_states
|
||||
elif (
|
||||
lora_context is not None
|
||||
and lora_context.original_hidden_states is not None
|
||||
and lora_context.original_hidden_states.shape[0] == hidden_states.shape[0]
|
||||
):
|
||||
lora_x = lora_context.original_hidden_states
|
||||
else:
|
||||
lora_x = hidden_states
|
||||
|
||||
# TODO: The fallback to self.a1_scale was added for deferred static
|
||||
# activation quantization in https://github.com/vllm-project/vllm/pull/40857.
|
||||
# Activation emulation relies solely on `a1q_scale` output of
|
||||
@@ -412,93 +366,34 @@ class TritonExperts(LoRAExpertsMixin, mk.FusedMoEExpertsModular):
|
||||
else (a1q_scale if a1q_scale is not None else self.a1_scale)
|
||||
)
|
||||
|
||||
def _base_w13_fn():
|
||||
invoke_fused_moe_triton_kernel(
|
||||
hidden_states,
|
||||
w1,
|
||||
intermediate_cache1,
|
||||
input_scale,
|
||||
self.w1_scale,
|
||||
None, # topk_weights
|
||||
sorted_token_ids,
|
||||
expert_ids,
|
||||
num_tokens_post_padded,
|
||||
False, # mul_routed_weights
|
||||
top_k_num,
|
||||
config,
|
||||
compute_type=compute_type,
|
||||
use_fp8_w8a8=self.quant_config.use_fp8_w8a8,
|
||||
use_int8_w8a8=self.quant_config.use_int8_w8a8,
|
||||
use_int8_w8a16=self.quant_config.use_int8_w8a16,
|
||||
use_int4_w4a16=self.quant_config.use_int4_w4a16,
|
||||
per_channel_quant=self.per_act_token_quant,
|
||||
block_shape=self.block_shape,
|
||||
B_bias=self.w1_bias,
|
||||
)
|
||||
|
||||
if lora_context is not None and lora_context.aux_stream is not None:
|
||||
# add_inputs=False: kernel overwrites lora_delta_w13. zeros (not
|
||||
# empty) so untouched rows -- e.g. blocks where every program
|
||||
# early-exits because lora_id<0 -- stay at zero and the trailing
|
||||
# add_() is a no-op there.
|
||||
lora_delta_w13 = torch.zeros_like(intermediate_cache1)
|
||||
|
||||
def _lora_w13_fn():
|
||||
return self.apply_w13_lora(
|
||||
lora_context,
|
||||
y=lora_delta_w13,
|
||||
x=lora_x,
|
||||
topk_ids=topk_ids,
|
||||
topk_weights=topk_weights,
|
||||
expert_map=expert_map,
|
||||
w1=w1,
|
||||
w2=w2,
|
||||
num_tokens=num_tokens,
|
||||
top_k_num=top_k_num,
|
||||
add_inputs=False,
|
||||
)
|
||||
|
||||
assert lora_context.events is not None
|
||||
_, lora_meta = maybe_execute_in_parallel(
|
||||
_base_w13_fn,
|
||||
_lora_w13_fn,
|
||||
lora_context.events[0],
|
||||
lora_context.events[1],
|
||||
lora_context.aux_stream,
|
||||
)
|
||||
(
|
||||
sorted_token_ids_lora,
|
||||
expert_ids_lora,
|
||||
num_tokens_post_padded_lora,
|
||||
token_lora_mapping,
|
||||
) = lora_meta
|
||||
intermediate_cache1.add_(lora_delta_w13)
|
||||
else:
|
||||
_base_w13_fn()
|
||||
if lora_context is not None:
|
||||
(
|
||||
sorted_token_ids_lora,
|
||||
expert_ids_lora,
|
||||
num_tokens_post_padded_lora,
|
||||
token_lora_mapping,
|
||||
) = self.apply_w13_lora(
|
||||
lora_context,
|
||||
y=intermediate_cache1,
|
||||
x=lora_x,
|
||||
topk_ids=topk_ids,
|
||||
topk_weights=topk_weights,
|
||||
expert_map=expert_map,
|
||||
w1=w1,
|
||||
w2=w2,
|
||||
num_tokens=num_tokens,
|
||||
top_k_num=top_k_num,
|
||||
)
|
||||
invoke_fused_moe_triton_kernel(
|
||||
hidden_states,
|
||||
w1,
|
||||
intermediate_cache1,
|
||||
input_scale,
|
||||
self.w1_scale,
|
||||
None, # topk_weights
|
||||
sorted_token_ids,
|
||||
expert_ids,
|
||||
num_tokens_post_padded,
|
||||
False, # mul_routed_weights
|
||||
top_k_num,
|
||||
config,
|
||||
compute_type=compute_type,
|
||||
use_fp8_w8a8=self.quant_config.use_fp8_w8a8,
|
||||
use_int8_w8a8=self.quant_config.use_int8_w8a8,
|
||||
use_int8_w8a16=self.quant_config.use_int8_w8a16,
|
||||
use_int4_w4a16=self.quant_config.use_int4_w4a16,
|
||||
per_channel_quant=self.per_act_token_quant,
|
||||
block_shape=self.block_shape,
|
||||
B_bias=self.w1_bias,
|
||||
)
|
||||
|
||||
a2q_scale: torch.Tensor | None = None
|
||||
|
||||
# Fuse SiLU+Mul + FP8 block quantize into a single kernel
|
||||
# when conditions permit (gated SiLU, fp8 block quant with
|
||||
# group_size=128, no LoRA requiring the BF16 intermediate).
|
||||
# group_size=128).
|
||||
# The fused kernel has no clamp parameter, so a configured
|
||||
# SwiGLU clamp limit falls through to the unfused path.
|
||||
if (
|
||||
@@ -506,7 +401,6 @@ class TritonExperts(LoRAExpertsMixin, mk.FusedMoEExpertsModular):
|
||||
and self.activation_config.clamp_limit is None
|
||||
and self.quant_config.use_fp8_w8a8
|
||||
and self.block_shape == [128, 128]
|
||||
and lora_context is None
|
||||
and not is_deep_gemm_e8m0_used()
|
||||
):
|
||||
qintermediate_cache2, a2q_scale = ops.silu_and_mul_per_block_quant(
|
||||
@@ -528,83 +422,29 @@ class TritonExperts(LoRAExpertsMixin, mk.FusedMoEExpertsModular):
|
||||
quantization_emulation=self.quantization_emulation,
|
||||
)
|
||||
|
||||
# LoRA w2: applied to intermediate_cache3 before moe_sum, using the
|
||||
# unquantized intermediate_cache2 as the lora_a input. Reuses the
|
||||
# sorted_token_ids_lora computed above. Same dual-stream pattern as
|
||||
# the w13 pair: base GEMM on default stream, LoRA delta on aux,
|
||||
# join via .add_() into intermediate_cache3.
|
||||
def _base_w2_fn():
|
||||
invoke_fused_moe_triton_kernel(
|
||||
qintermediate_cache2,
|
||||
w2,
|
||||
intermediate_cache3,
|
||||
a2q_scale,
|
||||
self.w2_scale,
|
||||
topk_weights,
|
||||
sorted_token_ids,
|
||||
expert_ids,
|
||||
num_tokens_post_padded,
|
||||
not apply_router_weight_on_input,
|
||||
1,
|
||||
config,
|
||||
compute_type=compute_type,
|
||||
use_fp8_w8a8=self.quant_config.use_fp8_w8a8,
|
||||
use_int8_w8a8=self.quant_config.use_int8_w8a8,
|
||||
use_int8_w8a16=self.quant_config.use_int8_w8a16,
|
||||
use_int4_w4a16=self.quant_config.use_int4_w4a16,
|
||||
per_channel_quant=self.per_act_token_quant,
|
||||
block_shape=self.block_shape,
|
||||
B_bias=self.w2_bias,
|
||||
)
|
||||
invoke_fused_moe_triton_kernel(
|
||||
qintermediate_cache2,
|
||||
w2,
|
||||
intermediate_cache3,
|
||||
a2q_scale,
|
||||
self.w2_scale,
|
||||
topk_weights,
|
||||
sorted_token_ids,
|
||||
expert_ids,
|
||||
num_tokens_post_padded,
|
||||
not apply_router_weight_on_input,
|
||||
1,
|
||||
config,
|
||||
compute_type=compute_type,
|
||||
use_fp8_w8a8=self.quant_config.use_fp8_w8a8,
|
||||
use_int8_w8a8=self.quant_config.use_int8_w8a8,
|
||||
use_int8_w8a16=self.quant_config.use_int8_w8a16,
|
||||
use_int4_w4a16=self.quant_config.use_int4_w4a16,
|
||||
per_channel_quant=self.per_act_token_quant,
|
||||
block_shape=self.block_shape,
|
||||
B_bias=self.w2_bias,
|
||||
)
|
||||
|
||||
if lora_context is not None and lora_context.aux_stream is not None:
|
||||
lora_delta_w2 = torch.zeros_like(intermediate_cache3)
|
||||
|
||||
def _lora_w2_fn():
|
||||
self.apply_w2_lora(
|
||||
lora_context,
|
||||
y=lora_delta_w2,
|
||||
x=intermediate_cache2,
|
||||
topk_weights=topk_weights,
|
||||
sorted_token_ids_lora=sorted_token_ids_lora,
|
||||
expert_ids_lora=expert_ids_lora,
|
||||
num_tokens_post_padded_lora=num_tokens_post_padded_lora,
|
||||
token_lora_mapping=token_lora_mapping,
|
||||
num_tokens=num_tokens,
|
||||
w1=w1,
|
||||
w2=w2,
|
||||
top_k_num=top_k_num,
|
||||
add_inputs=False,
|
||||
)
|
||||
|
||||
assert lora_context.events is not None
|
||||
maybe_execute_in_parallel(
|
||||
_base_w2_fn,
|
||||
_lora_w2_fn,
|
||||
lora_context.events[2],
|
||||
lora_context.events[3],
|
||||
lora_context.aux_stream,
|
||||
)
|
||||
intermediate_cache3.add_(lora_delta_w2)
|
||||
else:
|
||||
_base_w2_fn()
|
||||
if lora_context is not None:
|
||||
self.apply_w2_lora(
|
||||
lora_context,
|
||||
y=intermediate_cache3,
|
||||
x=intermediate_cache2,
|
||||
topk_weights=topk_weights,
|
||||
sorted_token_ids_lora=sorted_token_ids_lora,
|
||||
expert_ids_lora=expert_ids_lora,
|
||||
num_tokens_post_padded_lora=num_tokens_post_padded_lora,
|
||||
token_lora_mapping=token_lora_mapping,
|
||||
num_tokens=num_tokens,
|
||||
w1=w1,
|
||||
w2=w2,
|
||||
top_k_num=top_k_num,
|
||||
)
|
||||
|
||||
# separate function is required for MoE + LoRA
|
||||
self.moe_sum(intermediate_cache3, output, topk_ids, expert_map)
|
||||
|
||||
def moe_sum(
|
||||
@@ -804,5 +644,4 @@ class TritonWNA16Experts(TritonExperts):
|
||||
block_shape=self.block_shape,
|
||||
)
|
||||
|
||||
# separate function is required for MoE + LoRA
|
||||
self.moe_sum(intermediate_cache3, output)
|
||||
|
||||
@@ -1,731 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""LoRA-aware FlashInfer TRT-LLM MoE experts (BF16, MXFP4 x BF16).
|
||||
|
||||
Reuses the routed API + ``gemm1_lora_delta`` path from FlashInfer PR #3153
|
||||
(BF16) and PR #3987 (FP4 block-scale):
|
||||
|
||||
* The W13 (gate_up) LoRA delta is passed directly as ``gemm1_lora_delta`` to
|
||||
the routed kernel, which fuses it into FC1 before SwiGLU (BiasType::Mn).
|
||||
* The W2 (down) LoRA cannot be fused -- we take the FC1 activation output
|
||||
returned by the kernel (``gemm1_activation_output``, permuted) together with
|
||||
``expanded_idx_to_permuted_idx``, unpermute it, compute the W2 delta out of
|
||||
kernel via punica, and add it to the already-finalized output.
|
||||
|
||||
Constraints (matching the PR support matrix; final gating lives in the oracle):
|
||||
* SM100+ (Blackwell), gated SwiGLU, shuffled weights only;
|
||||
* BF16 weights, or MXFP4 weights with BF16 activations;
|
||||
* routing must be computed outside the MoE (the Modular path satisfies this).
|
||||
|
||||
"""
|
||||
|
||||
import functools
|
||||
from abc import abstractmethod
|
||||
|
||||
import torch
|
||||
|
||||
import vllm.model_executor.layers.fused_moe.modular_kernel as mk
|
||||
from vllm.model_executor.layers.fused_moe.activation import MoEActivation
|
||||
from vllm.model_executor.layers.fused_moe.config import (
|
||||
FusedMoEConfig,
|
||||
FusedMoEParallelConfig,
|
||||
FusedMoEQuantConfig,
|
||||
RoutingMethodType,
|
||||
)
|
||||
from vllm.model_executor.layers.fused_moe.experts.lora_context import MoELoRAContext
|
||||
from vllm.model_executor.layers.fused_moe.experts.lora_experts_mixin import (
|
||||
LoRAExpertsMixin,
|
||||
)
|
||||
from vllm.model_executor.layers.fused_moe.experts.trtllm_bf16_moe import (
|
||||
view_as_block_major_k,
|
||||
)
|
||||
from vllm.model_executor.layers.fused_moe.experts.trtllm_mxfp4_moe import (
|
||||
TrtLlmMxfp4ExpertsBase,
|
||||
)
|
||||
from vllm.model_executor.layers.fused_moe.topk_weight_and_reduce import (
|
||||
TopKWeightAndReduceNoOP,
|
||||
)
|
||||
from vllm.model_executor.layers.fused_moe.utils import fi_moe_largest_bucket
|
||||
from vllm.model_executor.layers.quantization.utils.quant_utils import (
|
||||
QuantKey,
|
||||
kMxfp4Static,
|
||||
)
|
||||
from vllm.platforms import current_platform
|
||||
from vllm.triton_utils import tl, triton
|
||||
from vllm.utils.flashinfer import has_flashinfer_trtllm_fused_moe
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _unpermute_activation_kernel(
|
||||
act_ptr, # act_permuted: (num_permuted, num_cols)
|
||||
idx_ptr, # idx_map: (num_rows,), values in [0, num_permuted) or -1
|
||||
out_ptr, # out: (num_rows, num_cols)
|
||||
num_cols,
|
||||
stride_ar,
|
||||
stride_or,
|
||||
BLOCK_I: tl.constexpr,
|
||||
):
|
||||
row = tl.program_id(0)
|
||||
col_offs = tl.program_id(1) * BLOCK_I + tl.arange(0, BLOCK_I)
|
||||
col_mask = col_offs < num_cols
|
||||
|
||||
idx = tl.load(idx_ptr + row)
|
||||
out_ptrs = out_ptr + row * stride_or + col_offs
|
||||
if idx >= 0:
|
||||
vals = tl.load(act_ptr + idx * stride_ar + col_offs, mask=col_mask, other=0.0)
|
||||
tl.store(out_ptrs, vals, mask=col_mask)
|
||||
else:
|
||||
zeros = tl.zeros((BLOCK_I,), dtype=out_ptr.dtype.element_ty)
|
||||
tl.store(out_ptrs, zeros, mask=col_mask)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _finalize_lora_kernel(
|
||||
gemm2_ptr, # (num_permuted, K) base FC2 output, permuted, unweighted
|
||||
weight_ptr, # (num_tokens * top_k,) routing weights (expanded order)
|
||||
idx_ptr, # (num_tokens * top_k,) expanded_idx -> permuted_idx or -1
|
||||
delta_ptr, # (num_tokens, top_k, K) W2 LoRA delta, already routing-weighted
|
||||
out_ptr, # (num_tokens, K)
|
||||
K,
|
||||
stride_g0,
|
||||
stride_d0,
|
||||
stride_d1,
|
||||
stride_o0,
|
||||
scale,
|
||||
TOP_K: tl.constexpr,
|
||||
BLOCK_K: tl.constexpr,
|
||||
):
|
||||
token = tl.program_id(0)
|
||||
col = tl.program_id(1) * BLOCK_K + tl.arange(0, BLOCK_K)
|
||||
mask = col < K
|
||||
|
||||
acc_base = tl.zeros((BLOCK_K,), dtype=tl.float32)
|
||||
acc_delta = tl.zeros((BLOCK_K,), dtype=tl.float32)
|
||||
for k in tl.static_range(TOP_K):
|
||||
eid = token * TOP_K + k
|
||||
pidx = tl.load(idx_ptr + eid)
|
||||
if pidx >= 0:
|
||||
w = tl.load(weight_ptr + eid).to(tl.float32)
|
||||
base = tl.load(gemm2_ptr + pidx * stride_g0 + col, mask=mask, other=0.0).to(
|
||||
tl.float32
|
||||
)
|
||||
acc_base += w * base
|
||||
acc_delta += tl.load(
|
||||
delta_ptr + token * stride_d0 + k * stride_d1 + col, mask=mask, other=0.0
|
||||
).to(tl.float32)
|
||||
|
||||
out = acc_base * scale + acc_delta
|
||||
tl.store(
|
||||
out_ptr + token * stride_o0 + col, out.to(out_ptr.dtype.element_ty), mask=mask
|
||||
)
|
||||
|
||||
|
||||
class _TrtLlmLoRAExpertsBase(LoRAExpertsMixin, mk.FusedMoEExpertsModular):
|
||||
"""LoRA-aware trtllm MoE experts."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
moe_config: FusedMoEConfig,
|
||||
quant_config: FusedMoEQuantConfig,
|
||||
):
|
||||
super().__init__(moe_config, quant_config)
|
||||
self.routing_method_type = moe_config.routing_method
|
||||
self.topk = moe_config.experts_per_token
|
||||
self.intermediate_size_per_partition = (
|
||||
moe_config.intermediate_size_per_partition
|
||||
)
|
||||
self.hidden_dim = moe_config.hidden_dim
|
||||
self.local_num_experts = moe_config.num_local_experts
|
||||
self.ep_rank = moe_config.moe_parallel_config.ep_rank
|
||||
|
||||
@staticmethod
|
||||
def activation_format() -> mk.FusedMoEActivationFormat:
|
||||
return mk.FusedMoEActivationFormat.Standard
|
||||
|
||||
@staticmethod
|
||||
def _supports_current_device() -> bool:
|
||||
p = current_platform
|
||||
return (
|
||||
p.is_cuda()
|
||||
and p.is_device_capability_family(100)
|
||||
and has_flashinfer_trtllm_fused_moe()
|
||||
)
|
||||
|
||||
def workspace_shapes(
|
||||
self,
|
||||
M,
|
||||
N,
|
||||
K,
|
||||
topk,
|
||||
global_num_experts,
|
||||
local_num_experts,
|
||||
expert_tokens_meta,
|
||||
activation,
|
||||
):
|
||||
# flashinfer manages its own workspace; only declare output (M, K)
|
||||
return (0,), (0,), (M, K)
|
||||
|
||||
def moe_problem_size(self, a1, w1, w2, topk_ids):
|
||||
"""Override the base 3D-weight assumption.
|
||||
|
||||
FusedMoEKernel._fused_experts calls moe_problem_size before apply(),
|
||||
but the base impl asserts ``len(w1.shape) == 3``. The flashinfer
|
||||
trtllm path stores shuffled weights in 4D BlockMajorK layout, so we
|
||||
derive the (E, M, N, K, topk) tuple from config + inputs instead.
|
||||
The N/K here only feed workspace sizing, which we zero out in
|
||||
workspace_shapes(); the real shapes are handled inside flashinfer.
|
||||
"""
|
||||
E = self.local_num_experts
|
||||
N = 2 * self.intermediate_size_per_partition
|
||||
K = self.hidden_dim
|
||||
M = a1.size(0) if a1.dim() == 2 else a1.size(1)
|
||||
topk = topk_ids.size(1)
|
||||
return E, M, N, K, topk
|
||||
|
||||
def finalize_weight_and_reduce_impl(self) -> mk.TopKWeightAndReduce:
|
||||
# apply() writes the fully finalized result into `output` (fused base
|
||||
# finalize + W2 LoRA reduction), so this is a No-Op.
|
||||
return TopKWeightAndReduceNoOP()
|
||||
|
||||
@staticmethod
|
||||
def _supports_parallel_config(
|
||||
moe_parallel_config: FusedMoEParallelConfig,
|
||||
) -> bool:
|
||||
return (
|
||||
not moe_parallel_config.use_all2all_kernels
|
||||
or moe_parallel_config.use_ag_rs_all2all_kernels
|
||||
) and not moe_parallel_config.enable_eplb
|
||||
|
||||
@staticmethod
|
||||
def _supports_router_logits_dtype(router_logits_dtype, routing_method) -> bool:
|
||||
return True
|
||||
|
||||
@staticmethod
|
||||
def _supports_no_act_and_mul() -> bool:
|
||||
return False # gated only
|
||||
|
||||
@property
|
||||
def expects_unquantized_inputs(self) -> bool:
|
||||
return True
|
||||
|
||||
@abstractmethod
|
||||
def invoke_routed_moe(
|
||||
self,
|
||||
*,
|
||||
hidden_states: torch.Tensor,
|
||||
w1: torch.Tensor,
|
||||
w2: torch.Tensor,
|
||||
topk_ids_and_weights: tuple[torch.Tensor, torch.Tensor],
|
||||
gemm1_lora_delta: torch.Tensor | None,
|
||||
global_num_experts: int,
|
||||
a1q_scale: torch.Tensor | None,
|
||||
output: torch.Tensor,
|
||||
) -> list[torch.Tensor]:
|
||||
"""Call the dtype-specific trtllm_*_routed_moe and return list[Tensor].
|
||||
|
||||
The LoRA path always sets gemm1_lora_delta and runs with
|
||||
do_finalize=False so the base finalize can be fused with the W2 LoRA
|
||||
reduction (see _finalize_with_w2_lora). Return contract:
|
||||
gemm1_lora_delta is None -> [output] (do_finalize=True)
|
||||
otherwise -> [gemm2_output(permuted, unweighted),
|
||||
expert_weights,
|
||||
expanded_idx_to_permuted_idx,
|
||||
gemm1_activation_output(permuted)]
|
||||
"""
|
||||
raise NotImplementedError
|
||||
|
||||
def prepare_gemm1_lora_delta(
|
||||
self,
|
||||
gemm1_lora_delta: torch.Tensor,
|
||||
lora_context: MoELoRAContext,
|
||||
topk_ids: torch.Tensor,
|
||||
) -> torch.Tensor:
|
||||
"""Hook to rewrite the W13 delta before it is handed to the kernel."""
|
||||
return gemm1_lora_delta
|
||||
|
||||
def apply(
|
||||
self,
|
||||
output: torch.Tensor,
|
||||
hidden_states: torch.Tensor,
|
||||
w1: torch.Tensor,
|
||||
w2: torch.Tensor,
|
||||
topk_weights: torch.Tensor,
|
||||
topk_ids: torch.Tensor,
|
||||
activation: MoEActivation,
|
||||
global_num_experts: int,
|
||||
expert_map: torch.Tensor | None,
|
||||
a1q_scale: torch.Tensor | None,
|
||||
a2_scale: torch.Tensor | None,
|
||||
workspace13: torch.Tensor,
|
||||
workspace2: torch.Tensor,
|
||||
expert_tokens_meta: mk.ExpertTokensMetadata | None,
|
||||
apply_router_weight_on_input: bool,
|
||||
):
|
||||
lora_context = self._lora_context
|
||||
assert lora_context is not None, "LoRA context must be set"
|
||||
num_tokens = hidden_states.size(0)
|
||||
top_k = self.topk
|
||||
intermediate_size = self.intermediate_size_per_partition
|
||||
# LoRA buffers use the padded hidden size; `output` may be narrower.
|
||||
K = self.hidden_dim
|
||||
|
||||
# ---- Base-model fast path ----
|
||||
# When no token in the batch selects a LoRA adapter, skip the LoRA machinery
|
||||
# and run the plain base MoE with do_finalize=True, which writes the finalized
|
||||
# result straight into `output`.
|
||||
if self._batch_has_no_lora(lora_context):
|
||||
self.invoke_routed_moe(
|
||||
hidden_states=hidden_states,
|
||||
w1=w1,
|
||||
w2=w2,
|
||||
topk_ids_and_weights=(topk_ids, topk_weights),
|
||||
gemm1_lora_delta=None, # without LoRA, no delta
|
||||
global_num_experts=global_num_experts,
|
||||
a1q_scale=a1q_scale,
|
||||
output=output,
|
||||
)
|
||||
return
|
||||
|
||||
# The LoRA tile-config heuristic (try_get_optimal_moe_config) unpacks
|
||||
# w1/w2 as standard 3D MoE weights, but flashinfer stores shuffled
|
||||
# 4D BlockMajorK weights. add_lora_w13/add_lora_w2 only read .shape
|
||||
# from w1/w2 (the actual GEMM uses lora_a/b_stacked), so pass
|
||||
# zero-storage meta tensors carrying the logical 3D shapes:
|
||||
# w1: (E, 2I, H) w2: (E, H, I)
|
||||
w1_cfg = torch.empty(
|
||||
(self.local_num_experts, 2 * intermediate_size, K),
|
||||
device="meta",
|
||||
dtype=torch.bfloat16,
|
||||
)
|
||||
w2_cfg = torch.empty(
|
||||
(self.local_num_experts, K, intermediate_size),
|
||||
device="meta",
|
||||
dtype=torch.bfloat16,
|
||||
)
|
||||
|
||||
# ---- 1) W13 LoRA delta -> gemm1_lora_delta (bf16, [T, top_k, 2I]) ----
|
||||
gemm1_lora_delta = None
|
||||
w13_meta = (None, None, None, None)
|
||||
|
||||
# zeros (not empty): under EP the punica expand kernel only writes
|
||||
# slots whose expert is local to this rank; the kernel skips the rest.
|
||||
gemm1_lora_delta = torch.zeros(
|
||||
num_tokens,
|
||||
top_k,
|
||||
2 * intermediate_size,
|
||||
dtype=torch.bfloat16,
|
||||
device=hidden_states.device,
|
||||
)
|
||||
|
||||
lora_x = hidden_states
|
||||
if not self.expects_unquantized_inputs:
|
||||
orig = lora_context.original_hidden_states
|
||||
assert orig is not None and orig.shape[0] == hidden_states.shape[0], (
|
||||
"quantized trtllm LoRA path requires original_hidden_states"
|
||||
)
|
||||
lora_x = orig
|
||||
# add_inputs=False: write the pure delta only (the base is fused in
|
||||
# by the kernel) and do NOT multiply by the routing weight (it is a
|
||||
# pre-SwiGLU bias).
|
||||
# swap_w13_slices=True: apply_w13_lora writes in vLLM's w13 order
|
||||
# (gate=w1 first, up=w3 second), but FlashInfer's gemm1_lora_delta
|
||||
# expects [up, gate]; reversing the slices emits that order directly,
|
||||
# avoiding an out-of-place concat swap.
|
||||
w13_meta = self.apply_w13_lora(
|
||||
lora_context,
|
||||
y=gemm1_lora_delta,
|
||||
x=lora_x,
|
||||
topk_ids=topk_ids,
|
||||
topk_weights=topk_weights,
|
||||
expert_map=expert_map,
|
||||
w1=w1_cfg,
|
||||
w2=w2_cfg,
|
||||
num_tokens=num_tokens,
|
||||
top_k_num=top_k,
|
||||
add_inputs=False,
|
||||
swap_w13_slices=True,
|
||||
)
|
||||
gemm1_lora_delta = self.prepare_gemm1_lora_delta(
|
||||
gemm1_lora_delta, lora_context, topk_ids
|
||||
)
|
||||
|
||||
# ---- 2) Call the routed flashinfer kernel ----
|
||||
ret = self.invoke_routed_moe(
|
||||
hidden_states=hidden_states,
|
||||
w1=w1,
|
||||
w2=w2,
|
||||
topk_ids_and_weights=(topk_ids, topk_weights),
|
||||
gemm1_lora_delta=gemm1_lora_delta,
|
||||
global_num_experts=global_num_experts,
|
||||
a1q_scale=a1q_scale,
|
||||
output=output,
|
||||
)
|
||||
# ---- 3) W2 LoRA (computed out of kernel) ----
|
||||
# do_finalize=False: flashinfer returns the *unfinalized* base output.
|
||||
# ret = [gemm2_output(permuted, unweighted),
|
||||
# expert_weights, expanded_idx_to_permuted_idx,
|
||||
# gemm1_activation_output(permuted)]
|
||||
gemm2_permuted = ret[0]
|
||||
expert_weights = ret[1]
|
||||
expanded_idx_to_permuted_idx = ret[2]
|
||||
gemm1_act_permuted = ret[3] # [max_padded, I], post-act
|
||||
act = self._unpermute_activation(
|
||||
gemm1_act_permuted,
|
||||
expanded_idx_to_permuted_idx,
|
||||
num_tokens,
|
||||
top_k,
|
||||
intermediate_size,
|
||||
) # (T*top_k, I) -- same layout as the triton path's intermediate_cache2
|
||||
|
||||
(
|
||||
sorted_token_ids_lora,
|
||||
expert_ids_lora,
|
||||
num_tokens_post_padded_lora,
|
||||
token_lora_mapping,
|
||||
) = w13_meta
|
||||
|
||||
w2_delta = torch.zeros(
|
||||
num_tokens,
|
||||
top_k,
|
||||
K,
|
||||
dtype=output.dtype,
|
||||
device=output.device,
|
||||
)
|
||||
self.apply_w2_lora(
|
||||
lora_context,
|
||||
y=w2_delta,
|
||||
x=act,
|
||||
topk_weights=topk_weights,
|
||||
sorted_token_ids_lora=sorted_token_ids_lora,
|
||||
expert_ids_lora=expert_ids_lora,
|
||||
num_tokens_post_padded_lora=num_tokens_post_padded_lora,
|
||||
token_lora_mapping=token_lora_mapping,
|
||||
num_tokens=num_tokens,
|
||||
w1=w1_cfg,
|
||||
w2=w2_cfg,
|
||||
top_k_num=top_k,
|
||||
add_inputs=False,
|
||||
)
|
||||
# ---- 4) Fused finalize: reduce the base path over top_k (with routing
|
||||
# weights) and add the already-weighted W2 delta, in a single kernel.
|
||||
# This replaces flashinfer's internal finalize launch plus a separate
|
||||
# w2_delta.sum(dim=1) + add_.
|
||||
self._finalize_with_w2_lora(
|
||||
output,
|
||||
gemm2_permuted,
|
||||
expert_weights,
|
||||
expanded_idx_to_permuted_idx,
|
||||
w2_delta,
|
||||
num_tokens,
|
||||
top_k,
|
||||
scale=1.0,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _batch_has_no_lora(lora_context: MoELoRAContext) -> bool:
|
||||
"""True when no token in the batch selects a LoRA adapter.
|
||||
|
||||
Mirrors the no-lora fast path in
|
||||
``PunicaWrapperGPU.add_lora_fused_moe``: the punica kernel metadata
|
||||
carries a CPU ``no_lora_flag`` computed once per forward from the
|
||||
token->LoRA mapping. Reading it is a host-only check (no device sync),
|
||||
and under CUDA graphs the branch is frozen at capture time against the
|
||||
graph's ``has_lora`` dispatch key, so it stays correct on replay.
|
||||
"""
|
||||
meta = getattr(lora_context.punica_wrapper, "token_mapping_meta", None)
|
||||
if meta is None:
|
||||
return False
|
||||
flag = meta.no_lora_flag_cpu
|
||||
return bool(flag.numel() == 1 and flag.item())
|
||||
|
||||
@staticmethod
|
||||
def _unpermute_activation(
|
||||
act_permuted: torch.Tensor,
|
||||
idx_map: torch.Tensor,
|
||||
num_tokens: int,
|
||||
top_k: int,
|
||||
intermediate_size: int,
|
||||
) -> torch.Tensor:
|
||||
"""Permuted FC1 activation -> (num_tokens*top_k, I).
|
||||
|
||||
expanded_idx = token*top_k + k; idx_map[expanded_idx] = permuted_idx or -1.
|
||||
Fused gather + drop-masking: each output row copies the matching
|
||||
permuted row, or is zeroed when idx_map < 0.
|
||||
"""
|
||||
num_rows = num_tokens * top_k
|
||||
out = torch.empty(
|
||||
(num_rows, intermediate_size),
|
||||
dtype=act_permuted.dtype,
|
||||
device=act_permuted.device,
|
||||
)
|
||||
BLOCK_I = 1024
|
||||
grid = (num_rows, triton.cdiv(intermediate_size, BLOCK_I))
|
||||
_unpermute_activation_kernel[grid](
|
||||
act_permuted,
|
||||
idx_map,
|
||||
out,
|
||||
intermediate_size,
|
||||
act_permuted.stride(0),
|
||||
out.stride(0),
|
||||
BLOCK_I=BLOCK_I,
|
||||
)
|
||||
return out
|
||||
|
||||
@staticmethod
|
||||
def _finalize_with_w2_lora(
|
||||
output: torch.Tensor,
|
||||
gemm2_permuted: torch.Tensor,
|
||||
expert_weights: torch.Tensor,
|
||||
idx_map: torch.Tensor,
|
||||
w2_delta: torch.Tensor,
|
||||
num_tokens: int,
|
||||
top_k: int,
|
||||
scale: float = 1.0,
|
||||
) -> None:
|
||||
"""Fused base finalize + W2 LoRA reduction, written into ``output``.
|
||||
|
||||
For each token: sum the routing-weighted permuted base rows over top_k
|
||||
(``expert_weights`` in expanded order, ``idx_map < 0`` dropped), scale by
|
||||
``scale``, and add the already-weighted ``w2_delta`` reduced over top_k.
|
||||
"""
|
||||
# Inputs may carry hidden padding that `output` does not.
|
||||
K = output.size(1)
|
||||
assert gemm2_permuted.size(1) >= K and w2_delta.size(2) >= K
|
||||
BLOCK_K = 512
|
||||
grid = (num_tokens, triton.cdiv(K, BLOCK_K))
|
||||
_finalize_lora_kernel[grid](
|
||||
gemm2_permuted,
|
||||
expert_weights.reshape(-1),
|
||||
idx_map,
|
||||
w2_delta,
|
||||
output,
|
||||
K,
|
||||
gemm2_permuted.stride(0),
|
||||
w2_delta.stride(0),
|
||||
w2_delta.stride(1),
|
||||
output.stride(0),
|
||||
scale,
|
||||
TOP_K=top_k,
|
||||
BLOCK_K=BLOCK_K,
|
||||
)
|
||||
|
||||
|
||||
# BF16 unquantized trtllm MoE + LoRA
|
||||
class TrtLlmBf16LoRAExperts(_TrtLlmLoRAExpertsBase):
|
||||
"""BF16 unquantized trtllm MoE + LoRA."""
|
||||
|
||||
@staticmethod
|
||||
def _supports_quant_scheme(weight_key, activation_key) -> bool:
|
||||
return weight_key is None and activation_key is None
|
||||
|
||||
@staticmethod
|
||||
def _supports_activation(activation: MoEActivation) -> bool:
|
||||
return activation in [MoEActivation.SILU]
|
||||
|
||||
@staticmethod
|
||||
def _supports_routing_method(routing_method, weight_key, activation_key) -> bool:
|
||||
return routing_method in [
|
||||
RoutingMethodType.DeepSeekV3,
|
||||
RoutingMethodType.Llama4,
|
||||
RoutingMethodType.Renormalize,
|
||||
RoutingMethodType.RenormalizeNaive,
|
||||
]
|
||||
|
||||
def invoke_routed_moe(
|
||||
self,
|
||||
*,
|
||||
hidden_states: torch.Tensor,
|
||||
w1: torch.Tensor,
|
||||
w2: torch.Tensor,
|
||||
topk_ids_and_weights: tuple[torch.Tensor, torch.Tensor],
|
||||
gemm1_lora_delta: torch.Tensor | None,
|
||||
global_num_experts: int,
|
||||
a1q_scale: torch.Tensor | None,
|
||||
output: torch.Tensor,
|
||||
) -> list[torch.Tensor]:
|
||||
import flashinfer
|
||||
|
||||
# With gemm1_lora_delta set (the LoRA path) run do_finalize=False and
|
||||
# return the unfinalized permuted base output so apply() can fuse the
|
||||
# finalize with the W2 LoRA reduction (see _finalize_with_w2_lora).
|
||||
# Without a delta (base path), run do_finalize=True and hand flashinfer
|
||||
# the caller's buffer via output= so it finalizes in place -- no copy.
|
||||
do_finalize = gemm1_lora_delta is None
|
||||
ret = flashinfer.fused_moe.trtllm_bf16_routed_moe(
|
||||
topk_ids=topk_ids_and_weights,
|
||||
hidden_states=hidden_states,
|
||||
gemm1_weights=view_as_block_major_k(w1),
|
||||
gemm2_weights=view_as_block_major_k(w2),
|
||||
gemm1_lora_delta=gemm1_lora_delta,
|
||||
num_experts=global_num_experts,
|
||||
top_k=self.topk,
|
||||
n_group=None,
|
||||
topk_group=None,
|
||||
intermediate_size=self.intermediate_size_per_partition,
|
||||
local_expert_offset=self.ep_rank * self.local_num_experts,
|
||||
local_num_experts=self.local_num_experts,
|
||||
routed_scaling_factor=None,
|
||||
routing_method_type=self.routing_method_type,
|
||||
do_finalize=do_finalize,
|
||||
output=output if do_finalize else None,
|
||||
)
|
||||
if not do_finalize:
|
||||
# [gemm2_output, expert_weights, expanded_idx, gemm1_activation]
|
||||
return list(ret)
|
||||
# do_finalize=True finalized directly into `output`.
|
||||
return [output]
|
||||
|
||||
|
||||
# MXFP4 weights x BF16 activations trtllm MoE + LoRA
|
||||
class TrtLlmMxfp4LoRAExperts(_TrtLlmLoRAExpertsBase, TrtLlmMxfp4ExpertsBase):
|
||||
"""MXFP4-weight, BF16-activation trtllm MoE + LoRA. Not MXFP8 activations:
|
||||
the kernel then returns the FC1 activation quantized, without its scales."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
moe_config: FusedMoEConfig,
|
||||
quant_config: FusedMoEQuantConfig,
|
||||
):
|
||||
_TrtLlmLoRAExpertsBase.__init__(self, moe_config, quant_config)
|
||||
# gemm1_alpha / gemm1_beta / gemm1_clamp_limit, hidden_dim_unpadded.
|
||||
TrtLlmMxfp4ExpertsBase.__init__(self, moe_config, quant_config)
|
||||
|
||||
@staticmethod
|
||||
def _supports_quant_scheme(
|
||||
weight_key: QuantKey | None,
|
||||
activation_key: QuantKey | None,
|
||||
) -> bool:
|
||||
return weight_key == kMxfp4Static and activation_key is None
|
||||
|
||||
@staticmethod
|
||||
def _supports_activation(activation: MoEActivation) -> bool:
|
||||
return activation in (MoEActivation.SWIGLUOAI, MoEActivation.SILU)
|
||||
|
||||
@staticmethod
|
||||
def _supports_routing_method(routing_method, weight_key, activation_key) -> bool:
|
||||
return True # routing is done outside the kernel
|
||||
|
||||
@staticmethod
|
||||
def _supports_parallel_config(
|
||||
moe_parallel_config: FusedMoEParallelConfig,
|
||||
) -> bool:
|
||||
# DP gathers tokens, but the no-LoRA fast path reads the local batch.
|
||||
p = moe_parallel_config
|
||||
return p.dp_size == 1 and not p.use_all2all_kernels and not p.enable_eplb
|
||||
|
||||
def workspace_shapes(
|
||||
self,
|
||||
M,
|
||||
N,
|
||||
K,
|
||||
topk,
|
||||
global_num_experts,
|
||||
local_num_experts,
|
||||
expert_tokens_meta,
|
||||
activation,
|
||||
):
|
||||
# flashinfer manages its own workspace and drops the hidden padding.
|
||||
return (0,), (0,), (M, self.hidden_dim_unpadded)
|
||||
|
||||
def output_hidden_dim(self, hidden_dim: int) -> int:
|
||||
# FusedMoEExpertsModular precedes TrtLlmMxfp4ExpertsBase in the MRO.
|
||||
return self.hidden_dim_unpadded
|
||||
|
||||
@functools.cached_property
|
||||
def _w13_bias_up_gate_index(self) -> torch.Tensor:
|
||||
"""Columns of the kernel-format FC1 bias in [up | gate] order."""
|
||||
from flashinfer.fused_moe.core import get_w2_permute_indices_with_cache
|
||||
|
||||
# Kernel format is the (up_i, gate_i) bias permuted for the tile.
|
||||
assert self.w1_bias is not None
|
||||
bias = self.w1_bias.reshape(self.local_num_experts, -1)[0]
|
||||
permute = get_w2_permute_indices_with_cache({}, bias.reshape(-1, 1), 128)
|
||||
interleaved = torch.argsort(permute.to(bias.device))
|
||||
return torch.cat([interleaved[0::2], interleaved[1::2]])
|
||||
|
||||
def prepare_gemm1_lora_delta(
|
||||
self,
|
||||
gemm1_lora_delta: torch.Tensor,
|
||||
lora_context: MoELoRAContext,
|
||||
topk_ids: torch.Tensor,
|
||||
) -> torch.Tensor:
|
||||
"""Bring the W13 delta to [up | gate] halves and fold in the FC1 bias."""
|
||||
intermediate_size = gemm1_lora_delta.size(-1) // 2
|
||||
# View as (I, 2) pairs of the first and second w13 half. SWIGLUOAI
|
||||
# weights, and the deltas that follow them, are interleaved.
|
||||
if self.moe_config.activation == MoEActivation.SWIGLUOAI:
|
||||
pairs = gemm1_lora_delta.unflatten(-1, (intermediate_size, 2))
|
||||
else:
|
||||
pairs = gemm1_lora_delta.unflatten(-1, (2, intermediate_size)).mT
|
||||
# Per-expert deltas arrive as (up, gate); fused 3D ones follow the
|
||||
# checkpoint (gate, up) order.
|
||||
if lora_context.w13_num_slices != 2:
|
||||
pairs = pairs.flip(-1)
|
||||
delta = pairs.mT.flatten(-2) # [up | gate]
|
||||
|
||||
# The kernel has a single FC1 bias slot, shared by the static bias and
|
||||
# the per-token delta, so the static bias rides along in the delta.
|
||||
if self.w1_bias is not None:
|
||||
bias = self.w1_bias.reshape(self.local_num_experts, -1)
|
||||
bias = bias[:, self._w13_bias_up_gate_index].float()
|
||||
local_ids = topk_ids.long() - self.ep_rank * self.local_num_experts
|
||||
local_ids = local_ids.clamp_(0, self.local_num_experts - 1)
|
||||
delta = (delta.float() + bias[local_ids]).to(torch.bfloat16)
|
||||
return delta.contiguous()
|
||||
|
||||
def invoke_routed_moe(
|
||||
self,
|
||||
*,
|
||||
hidden_states: torch.Tensor,
|
||||
w1: torch.Tensor,
|
||||
w2: torch.Tensor,
|
||||
topk_ids_and_weights: tuple[torch.Tensor, torch.Tensor],
|
||||
gemm1_lora_delta: torch.Tensor | None,
|
||||
global_num_experts: int,
|
||||
a1q_scale: torch.Tensor | None,
|
||||
output: torch.Tensor,
|
||||
) -> list[torch.Tensor]:
|
||||
from flashinfer import trtllm_fp4_block_scale_routed_moe
|
||||
|
||||
assert a1q_scale is None and hidden_states.dtype == torch.bfloat16
|
||||
topk_ids, topk_weights = topk_ids_and_weights
|
||||
# See TrtLlmBf16LoRAExperts.invoke_routed_moe for the return contract.
|
||||
do_finalize = gemm1_lora_delta is None
|
||||
ret = trtllm_fp4_block_scale_routed_moe(
|
||||
topk_ids=(topk_ids.to(torch.int32), topk_weights),
|
||||
routing_bias=None,
|
||||
hidden_states=hidden_states,
|
||||
hidden_states_scale=None,
|
||||
gemm1_weights=w1,
|
||||
gemm1_weights_scale=self.w1_scale,
|
||||
# With a delta the bias is folded into it (prepare_gemm1_lora_delta).
|
||||
gemm1_bias=self.w1_bias if do_finalize else None,
|
||||
gemm1_alpha=self.gemm1_alpha,
|
||||
gemm1_beta=self.gemm1_beta,
|
||||
gemm1_clamp_limit=self.gemm1_clamp_limit,
|
||||
gemm2_weights=w2,
|
||||
gemm2_weights_scale=self.w2_scale,
|
||||
gemm2_bias=self.w2_bias,
|
||||
output1_scale_scalar=None,
|
||||
output1_scale_gate_scalar=None,
|
||||
output2_scale_scalar=None,
|
||||
num_experts=global_num_experts,
|
||||
top_k=self.topk,
|
||||
n_group=None,
|
||||
topk_group=None,
|
||||
intermediate_size=self.intermediate_size_per_partition,
|
||||
local_expert_offset=self.ep_rank * self.local_num_experts,
|
||||
local_num_experts=self.local_num_experts,
|
||||
routed_scaling_factor=None,
|
||||
# Routing is already done; Renormalize is a safe default.
|
||||
routing_method_type=RoutingMethodType.Renormalize,
|
||||
do_finalize=do_finalize,
|
||||
enable_pdl=True,
|
||||
activation_type=self._flashinfer_activation_type(
|
||||
self.moe_config.activation
|
||||
),
|
||||
output=output if do_finalize else None,
|
||||
tune_max_num_tokens=fi_moe_largest_bucket(self.moe_config),
|
||||
gemm1_lora_delta=gemm1_lora_delta,
|
||||
)
|
||||
return [output] if do_finalize else list(ret)
|
||||
@@ -371,7 +371,6 @@ def FusedMoEFactory(
|
||||
max_num_tokens=max_num_batched_tokens,
|
||||
elastic_ep_max_dp_size=vllm_config.parallel_config.elastic_ep_max_dp_size,
|
||||
has_bias=has_bias,
|
||||
is_lora_enabled=vllm_config.lora_config is not None,
|
||||
shared_expert_prefix=shared_expert_prefix,
|
||||
activation=moe_activation,
|
||||
device=vllm_config.device_config.device,
|
||||
|
||||
@@ -606,8 +606,6 @@ class FusedMoEExperts(ABC):
|
||||
return False, _make_reason(f"{activation_format.value} activation format")
|
||||
elif envs.VLLM_BATCH_INVARIANT and not cls._supports_batch_invariance():
|
||||
return False, _make_reason("batch invariance")
|
||||
elif moe_config.is_lora_enabled and not cls.supports_lora():
|
||||
return False, _make_reason("LoRA")
|
||||
return True, None
|
||||
|
||||
@staticmethod
|
||||
@@ -772,15 +770,6 @@ class FusedMoEExperts(ABC):
|
||||
def g2_alphas(self) -> torch.Tensor | None:
|
||||
return self.quant_config.g2_alphas
|
||||
|
||||
@staticmethod
|
||||
def supports_lora() -> bool:
|
||||
"""Return True if this expert impl natively handles LoRA.
|
||||
|
||||
LoRA-aware experts should mix in LoRAExpertsMixin, which flips this
|
||||
to True and provides the per-forward LoRA state plumbing.
|
||||
"""
|
||||
return False
|
||||
|
||||
def supports_packed_ue8m0_act_scales(self) -> bool:
|
||||
"""A flag indicating whether or not this class can process packed ue8m0
|
||||
activation scales.
|
||||
@@ -1531,13 +1520,6 @@ class FusedMoEKernelModularImpl:
|
||||
apply_router_weight_on_input,
|
||||
)
|
||||
|
||||
# Stash the original unquantized hidden states on the LoRA context
|
||||
# so apply_w13_lora sees correct-magnitude activations instead of
|
||||
# the potentially quantized values produced by _prepare().
|
||||
lora_ctx = getattr(self.fused_experts, "_lora_context", None)
|
||||
if lora_ctx is not None:
|
||||
lora_ctx.original_hidden_states = hidden_states
|
||||
|
||||
fused_out = self._fused_experts(
|
||||
in_dtype=hidden_states.dtype,
|
||||
a1q=a1q,
|
||||
@@ -1555,9 +1537,6 @@ class FusedMoEKernelModularImpl:
|
||||
output_alias=output,
|
||||
)
|
||||
|
||||
if lora_ctx is not None:
|
||||
lora_ctx.original_hidden_states = None
|
||||
|
||||
if isinstance(fused_out, UnfinalizedMoEOutput):
|
||||
# Nothing below can run on an unfinalized output: finalize is the
|
||||
# local top-k reduction (and, for DP/EP, the combine) that the
|
||||
@@ -1733,9 +1712,6 @@ class FusedMoEKernel:
|
||||
def moe_config(self) -> FusedMoEConfig:
|
||||
return self.fused_experts.moe_config
|
||||
|
||||
def supports_lora(self) -> bool:
|
||||
return self.fused_experts.supports_lora()
|
||||
|
||||
def _post_init_setup(self):
|
||||
"""Resolve any leftover setup dependencies between self.prepare_finalize
|
||||
and self.fused_experts here.
|
||||
|
||||
@@ -337,13 +337,11 @@ def pad_tp_shard_to_weight_blocks(
|
||||
config.intermediate_size % block_n != 0
|
||||
or config.hidden_dim % block_n != 0
|
||||
or config.ep_size != 1
|
||||
or config.is_lora_enabled
|
||||
or config.has_bias
|
||||
):
|
||||
raise ValueError(
|
||||
f"Block-aligned FP8 TP sharding requires {block_n}-aligned "
|
||||
"global expert dimensions, pure TP, and no LoRA or "
|
||||
"expert bias."
|
||||
"global expert dimensions, pure TP, and no expert bias."
|
||||
)
|
||||
config.tp_shard_with_padding = True
|
||||
config.intermediate_size_per_partition = round_up(
|
||||
|
||||
@@ -522,30 +522,6 @@ def _make_log_unsupported(backend: Mxfp4MoeBackend, reason: str | None) -> str:
|
||||
return f"{base} since {reason}." if reason else f"{base}."
|
||||
|
||||
|
||||
def _requested_kernel_cls(
|
||||
backend: Mxfp4MoeBackend,
|
||||
config: FusedMoEConfig,
|
||||
) -> list[type[mk.FusedMoEExperts]]:
|
||||
"""Kernel classes for an explicitly requested backend. The LoRA-aware
|
||||
TRT-LLM experts are opt-in: "auto" keeps its current LoRA selection."""
|
||||
candidates = backend_to_kernel_cls(backend)
|
||||
if not config.is_lora_enabled or backend not in TRTLLM_BACKENDS:
|
||||
return candidates
|
||||
if backend == Mxfp4MoeBackend.FLASHINFER_TRTLLM_MXFP4_MXFP8:
|
||||
raise ValueError(
|
||||
_make_log_unsupported(
|
||||
backend,
|
||||
"expert LoRA is not supported with MXFP8 activations; use "
|
||||
"moe_backend='flashinfer_trtllm'",
|
||||
)
|
||||
)
|
||||
from vllm.model_executor.layers.fused_moe.experts.trtllm_lora_moe import (
|
||||
TrtLlmMxfp4LoRAExperts,
|
||||
)
|
||||
|
||||
return [TrtLlmMxfp4LoRAExperts, *candidates]
|
||||
|
||||
|
||||
def _return_or_raise(
|
||||
backend: Mxfp4MoeBackend,
|
||||
config: FusedMoEConfig,
|
||||
@@ -555,7 +531,7 @@ def _return_or_raise(
|
||||
scope: Literal["process", "global", "local"] = "local",
|
||||
) -> tuple[Mxfp4MoeBackend, type[mk.FusedMoEExperts]]:
|
||||
reason: str | None = None
|
||||
for k_cls in _requested_kernel_cls(backend, config):
|
||||
for k_cls in backend_to_kernel_cls(backend):
|
||||
supported, reason = k_cls.is_supported_config(
|
||||
k_cls, config, weight_key, activation_key, activation_format
|
||||
)
|
||||
|
||||
@@ -194,31 +194,6 @@ def map_unquantized_backend(runner_backend: MoEBackend) -> UnquantizedMoeBackend
|
||||
)
|
||||
|
||||
|
||||
def _trtllm_bf16_lora_supported(moe_config: FusedMoEConfig) -> bool:
|
||||
"""Gate for routing LoRA-enabled BF16 MoE to the FlashInfer TRT-LLM
|
||||
gemm1_lora_delta path (PR #3153).
|
||||
"""
|
||||
from vllm.model_executor.layers.fused_moe.experts.trtllm_lora_moe import (
|
||||
TrtLlmBf16LoRAExperts,
|
||||
)
|
||||
|
||||
# LoRA path returns before the oracle loop; reuse is_supported_config here.
|
||||
supported, _ = TrtLlmBf16LoRAExperts.is_supported_config(
|
||||
TrtLlmBf16LoRAExperts,
|
||||
moe_config,
|
||||
None,
|
||||
None,
|
||||
mk.FusedMoEActivationFormat.Standard,
|
||||
)
|
||||
if not supported:
|
||||
return False
|
||||
# The flashinfer trtllm fused-MoE kernel requires the per-partition
|
||||
# intermediate size to be a multiple of 128. Plain TP shards the MoE
|
||||
# intermediate dim (e.g. 768 -> 192 at tp=4), which would crash the kernel
|
||||
# at runtime; fall back to Triton in that case.
|
||||
return moe_config.intermediate_size_per_partition % 128 == 0
|
||||
|
||||
|
||||
def select_unquantized_moe_backend(
|
||||
moe_config: FusedMoEConfig,
|
||||
) -> tuple[UnquantizedMoeBackend, type[mk.FusedMoEExperts] | None]:
|
||||
@@ -231,22 +206,6 @@ def select_unquantized_moe_backend(
|
||||
if current_platform.is_out_of_tree():
|
||||
return UnquantizedMoeBackend.OOT, None
|
||||
|
||||
if moe_config.is_lora_enabled:
|
||||
if _trtllm_bf16_lora_supported(moe_config):
|
||||
from vllm.model_executor.layers.fused_moe.experts.trtllm_lora_moe import (
|
||||
TrtLlmBf16LoRAExperts,
|
||||
)
|
||||
|
||||
logger.info_once(
|
||||
"Using TrtLlmBf16LoRAExperts Unquantized MoE LoRA backend "
|
||||
"(TrtLlmBf16LoRAExperts)."
|
||||
)
|
||||
return UnquantizedMoeBackend.FLASHINFER_TRTLLM, TrtLlmBf16LoRAExperts
|
||||
logger.info_once("Using TRITON Unquantized MoE LoRA backend")
|
||||
return UnquantizedMoeBackend.TRITON, backend_to_kernel_cls(
|
||||
UnquantizedMoeBackend.TRITON
|
||||
)[0]
|
||||
|
||||
# NOTE: the kernels are selected in the following order.
|
||||
AVAILABLE_BACKENDS = _get_priority_backends(moe_config)
|
||||
|
||||
|
||||
@@ -93,14 +93,6 @@ class MoEPrepareAndFinalizeNaiveDPEPModular(mk.FusedMoEPrepareAndFinalizeModular
|
||||
super().__init__()
|
||||
self.is_sequence_parallel = is_sequence_parallel
|
||||
self._num_dispatchers = num_dispatchers
|
||||
# Set by FusedMoEWithLoRA.set_mapping() when LoRA is active. When
|
||||
# present, prepare() dispatches the per-token LoRA mapping alongside
|
||||
# hidden_states and writes the gathered result back to the context so
|
||||
# experts can use the per-rank-local mapping.
|
||||
self._lora_context = None
|
||||
|
||||
def set_lora_context(self, ctx) -> None:
|
||||
self._lora_context = ctx
|
||||
|
||||
@property
|
||||
def activation_format(self) -> mk.FusedMoEActivationFormat:
|
||||
@@ -193,27 +185,9 @@ class MoEPrepareAndFinalizeNaiveDPEPModular(mk.FusedMoEPrepareAndFinalizeModular
|
||||
a1, quant_config, defer_input_quant
|
||||
)
|
||||
|
||||
# When LoRA is active, dispatch the per-token LoRA id along with
|
||||
# hidden_states so every rank receives the correct mapping for the
|
||||
# tokens it ends up processing. The punica_wrapper stores indices as
|
||||
# int64 but the moe_lora_align_block_size kernel expects int32, so
|
||||
# pull the pre-cast view from token_mapping_meta.
|
||||
lora_ctx = self._lora_context
|
||||
local_token_lora_mapping = None
|
||||
if lora_ctx is not None:
|
||||
local_token_lora_mapping = (
|
||||
lora_ctx.punica_wrapper.token_mapping_meta.token_lora_mapping[
|
||||
: a1.shape[0]
|
||||
]
|
||||
)
|
||||
|
||||
extra_tensors: list[torch.Tensor] | None = None
|
||||
if scales is not None:
|
||||
extra_tensors = list(scales)
|
||||
if local_token_lora_mapping is not None:
|
||||
if extra_tensors is None:
|
||||
extra_tensors = []
|
||||
extra_tensors.append(local_token_lora_mapping)
|
||||
|
||||
with self._comm_region():
|
||||
res = get_ep_group().dispatch(
|
||||
@@ -232,10 +206,6 @@ class MoEPrepareAndFinalizeNaiveDPEPModular(mk.FusedMoEPrepareAndFinalizeModular
|
||||
assert len(res) == 4
|
||||
a1q, topk_weights, topk_ids, gathered_extras = res
|
||||
gathered_extras = list(gathered_extras)
|
||||
if local_token_lora_mapping is not None:
|
||||
dispatched_lora_mapping = gathered_extras.pop()
|
||||
assert lora_ctx is not None
|
||||
lora_ctx.local_token_lora_mapping = dispatched_lora_mapping
|
||||
if scales is not None:
|
||||
a1q_scale = _unwrap_scale_and_prepare_for_moe(
|
||||
gathered_extras, quant_config
|
||||
|
||||
@@ -212,8 +212,6 @@ class RoutedExperts(PluggableLayer):
|
||||
|
||||
self.quant_method.create_weights(layer=self, **moe_quant_params)
|
||||
|
||||
self.lora_base_layer_prefix = ""
|
||||
|
||||
# TODO(bnell): Temporary hack. Get rid of this.
|
||||
def _replace_quant_method(self, quant_method: FusedMoEMethodBase):
|
||||
self.quant_method = quant_method
|
||||
@@ -1121,7 +1119,6 @@ class RoutedExperts(PluggableLayer):
|
||||
num_experts=moe_config.num_logical_experts + num_fused_shared_experts,
|
||||
num_redundant_experts=num_redundant_experts,
|
||||
routed_experts_prefix="",
|
||||
lora_base_layer_prefix=self.lora_base_layer_prefix,
|
||||
include_fused=include_fused,
|
||||
is_gated=moe_config.is_act_and_mul,
|
||||
)
|
||||
@@ -1136,17 +1133,11 @@ class RoutedExperts(PluggableLayer):
|
||||
num_redundant_experts: int = 0,
|
||||
routed_experts_prefix: str = "routed_experts",
|
||||
) -> list[tuple[str, str, int, str]]:
|
||||
"""Build the expert mapping, detecting the LoRA `base_layer.` prefix by
|
||||
scanning `model`'s parameters.
|
||||
|
||||
Legacy entry point for models that still hand-roll `load_weights`; the
|
||||
`RoutedExperts` weight loader uses `get_expert_mapping` /
|
||||
`build_expert_params_mapping` instead (which take the prefix directly).
|
||||
See `build_expert_params_mapping` for the returned tuple format.
|
||||
"""Legacy entry point for models that still hand-roll `load_weights`;
|
||||
the `RoutedExperts` weight loader uses `get_expert_mapping` /
|
||||
`build_expert_params_mapping` instead. See
|
||||
`build_expert_params_mapping` for the returned tuple format.
|
||||
"""
|
||||
has_base_layer = any(".base_layer." in n for n, _ in model.named_parameters())
|
||||
prefix = "base_layer." if has_base_layer else ""
|
||||
# These loaders index ``params_dict[full_name]``, so both sides get it.
|
||||
return RoutedExperts.build_expert_params_mapping(
|
||||
ckpt_gate_proj_name,
|
||||
ckpt_down_proj_name,
|
||||
@@ -1154,8 +1145,6 @@ class RoutedExperts(PluggableLayer):
|
||||
num_experts,
|
||||
num_redundant_experts,
|
||||
routed_experts_prefix,
|
||||
lora_base_layer_prefix=prefix,
|
||||
lora_base_layer_prefix_on_param_name=prefix,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
@@ -1166,8 +1155,6 @@ class RoutedExperts(PluggableLayer):
|
||||
num_experts: int,
|
||||
num_redundant_experts: int = 0,
|
||||
routed_experts_prefix: str = "routed_experts",
|
||||
lora_base_layer_prefix: str = "",
|
||||
lora_base_layer_prefix_on_param_name: str = "",
|
||||
include_fused: bool = False,
|
||||
is_gated: bool = True,
|
||||
) -> list[tuple[str, str, int, str]]:
|
||||
@@ -1182,14 +1169,6 @@ class RoutedExperts(PluggableLayer):
|
||||
ckpt_up_proj_name: Name of up projection in checkpoint
|
||||
num_experts: Number of logical (non-redundant) experts
|
||||
num_redundant_experts: Number of redundant experts
|
||||
lora_base_layer_prefix: LoRA ``base_layer.`` prefix for the
|
||||
``weight_name`` (checkpoint) side
|
||||
lora_base_layer_prefix_on_param_name: same, for the ``param_name``
|
||||
side. Independent because ``get_expert_mapping`` resolves
|
||||
``param_name`` via ``getattr`` against this layer's bare
|
||||
``w13_weight``/``w2_weight`` (no prefix), while
|
||||
``make_expert_params_mapping`` indexes the model-wide
|
||||
``params_dict`` (prefix included).
|
||||
include_fused: Prepend the fused pre-fused-checkpoint entries
|
||||
is_gated: Whether w13 holds gate and up (gated activation) or a
|
||||
single up projection. Selects the fused checkpoint layout.
|
||||
@@ -1219,10 +1198,8 @@ class RoutedExperts(PluggableLayer):
|
||||
if routed_experts_prefix != "":
|
||||
routed_experts_prefix = f"{routed_experts_prefix}."
|
||||
|
||||
w13 = (
|
||||
f"experts.{lora_base_layer_prefix_on_param_name}{routed_experts_prefix}w13_"
|
||||
)
|
||||
w2 = f"experts.{lora_base_layer_prefix_on_param_name}{routed_experts_prefix}w2_"
|
||||
w13 = f"experts.{routed_experts_prefix}w13_"
|
||||
w2 = f"experts.{routed_experts_prefix}w2_"
|
||||
|
||||
fused_mapping = []
|
||||
if include_fused:
|
||||
@@ -1260,10 +1237,7 @@ class RoutedExperts(PluggableLayer):
|
||||
fused_mapping.extend(
|
||||
(
|
||||
w13,
|
||||
(
|
||||
f"experts.{physical_to_logical_map[expert_id]}."
|
||||
f"{gate_up}.{lora_base_layer_prefix}"
|
||||
),
|
||||
f"experts.{physical_to_logical_map[expert_id]}.{gate_up}",
|
||||
expert_id,
|
||||
shard_id,
|
||||
)
|
||||
@@ -1275,7 +1249,7 @@ class RoutedExperts(PluggableLayer):
|
||||
# (param_name, weight_name, expert_id, shard_id)
|
||||
(
|
||||
w13 if weight_name in [ckpt_gate_proj_name, ckpt_up_proj_name] else w2,
|
||||
f"experts.{physical_to_logical_map[expert_id]}.{weight_name}.{lora_base_layer_prefix}",
|
||||
f"experts.{physical_to_logical_map[expert_id]}.{weight_name}",
|
||||
expert_id,
|
||||
shard_id,
|
||||
)
|
||||
|
||||
@@ -195,8 +195,6 @@ def _moe_forward_shared_fake(
|
||||
return shared_out, fused_out
|
||||
|
||||
|
||||
# NOTE: `moe_forward` and `moe_forward_shared` being opaque custom ops is a
|
||||
# load-bearing assumption for the MoE-LoRA dual-stream path.
|
||||
direct_register_custom_op(
|
||||
op_name="moe_forward",
|
||||
op_func=_moe_forward,
|
||||
|
||||
@@ -83,7 +83,7 @@ class LogitsProcessor(PluggableLayer):
|
||||
self.vocab_size = vocab_size
|
||||
# Whether the input is logits (default is hidden states).
|
||||
self.logits_as_input = logits_as_input
|
||||
# original vocabulary size (without LoRA).
|
||||
# original vocabulary size (without added embeddings).
|
||||
self.org_vocab_size = org_vocab_size or vocab_size
|
||||
# Soft cap the logits. Used in Gemma 2.
|
||||
self.soft_cap = soft_cap
|
||||
|
||||
@@ -607,8 +607,8 @@ class QwenGatedDeltaNetAttention(GatedDeltaNetAttention):
|
||||
# projection of the input hidden states
|
||||
# Qwen3-Next and Qwen3.5 has a different qkv_proj layout,
|
||||
# we need to create qkvz_proj adaptively here.
|
||||
# When create_in_proj_qkvz is False (e.g. LoRA enabled in Qwen3.5),
|
||||
# in_proj_qkv and in_proj_z are created separately instead.
|
||||
# When create_in_proj_qkvz is False, in_proj_qkv and in_proj_z
|
||||
# are created separately instead.
|
||||
self.in_proj_qkvz = self.create_qkvz_proj(
|
||||
hidden_size=self.hidden_size,
|
||||
key_dim=self.key_dim,
|
||||
@@ -1281,7 +1281,9 @@ class QwenGatedDeltaNetAttention(GatedDeltaNetAttention):
|
||||
self,
|
||||
hidden_states: torch.Tensor,
|
||||
) -> torch.Tensor:
|
||||
assert not hasattr(self, "in_proj_qkv"), "lora isn't supported on CPU."
|
||||
assert not hasattr(self, "in_proj_qkv"), (
|
||||
"the split in_proj_qkv/in_proj_z layout is not supported on CPU"
|
||||
)
|
||||
|
||||
mixed_qkvz, _ = self.in_proj_qkvz(hidden_states)
|
||||
ba, _ = self.in_proj_ba(hidden_states)
|
||||
|
||||
-2
@@ -106,7 +106,6 @@ class CompressedTensorsMoEMethod(FusedMoEMethodBase):
|
||||
from vllm.platforms.rocm import on_gfx950
|
||||
|
||||
vllm_config = get_current_vllm_config()
|
||||
is_lora_disabled = vllm_config.lora_config is None
|
||||
moe_backend = vllm_config.kernel_config.moe_backend
|
||||
group_size = weight_quant.group_size or -1
|
||||
if (
|
||||
@@ -114,7 +113,6 @@ class CompressedTensorsMoEMethod(FusedMoEMethodBase):
|
||||
and weight_quant.type == QuantizationType.INT
|
||||
and group_size == 32
|
||||
and weight_quant.num_bits == 4
|
||||
and is_lora_disabled
|
||||
and on_gfx950()
|
||||
and moe_backend == "flydsl"
|
||||
):
|
||||
|
||||
@@ -518,12 +518,6 @@ def get_and_maybe_dequant_weights(
|
||||
Fp8PerTensorOnlineLinearMethod,
|
||||
)
|
||||
|
||||
# LoRA linear wrappers store quantization metadata on `base_layer`.
|
||||
# Unwrap here so callers can pass either a raw linear layer or its LoRA
|
||||
# wrapper without special-casing.
|
||||
while hasattr(layer, "base_layer") and hasattr(layer.base_layer, "quant_method"):
|
||||
layer = layer.base_layer
|
||||
|
||||
weight = get_attribute_fallback(layer, ["weight", "qweight", "weight_packed"])
|
||||
|
||||
# Unquantized layer: just return base weights
|
||||
|
||||
@@ -209,9 +209,9 @@ class VocabParallelEmbedding(PluggableLayer):
|
||||
Adapted from torch.nn.Embedding, note that we pad the vocabulary size to
|
||||
make sure it is divisible by the number of model parallel GPUs.
|
||||
|
||||
In order to support various loading methods, we ensure that LoRA-added
|
||||
In order to support various loading methods, we ensure that added
|
||||
embeddings are always at the end of TP-sharded tensors. In other words,
|
||||
we shard base embeddings and LoRA embeddings separately (both padded),
|
||||
we shard base embeddings and added embeddings separately (both padded),
|
||||
and place them in the same tensor.
|
||||
In this example, we will have the original vocab size = 1010,
|
||||
added vocab size = 16 and padding to 64. Therefore, the total
|
||||
@@ -219,16 +219,16 @@ class VocabParallelEmbedding(PluggableLayer):
|
||||
1024, add 16, and then pad to 1088).
|
||||
Therefore, the tensor format looks like the following:
|
||||
TP1, rank 0 (no sharding):
|
||||
|< --------BASE-------- >|< -BASE PADDING-- >|< -----LORA------ >|< -LORA PADDING-- >|
|
||||
|< --------BASE-------- >|< -BASE PADDING-- >|< -----ADDED----- >|< -ADDED PADDING- >|
|
||||
corresponding token_id: | 0 | 1 | ... | 1009 | -1 | ... | -1 | 1010 | ... | 1025 | -1 | ... | -1 |
|
||||
index: | 0 | 1 | ... | 1009 | 1010 | ... | 1023 | 1024 | ... | 1039 | 1040 | ... | 1087 |
|
||||
|
||||
TP2, rank 0:
|
||||
|< --------------------BASE--------------------- >|< -----LORA------ >|< -LORA PADDING- >|
|
||||
|< --------------------BASE--------------------- >|< -----ADDED----- >|< -ADDED PADDING- >|
|
||||
corresponding token_id: | 0 | 1 | 2 | ... | 497 | 498 | ... | 511 | 1010 | ... | 1025 | -1 | ... | -1 |
|
||||
index: | 0 | 1 | 2 | ... | 497 | 498 | ... | 511 | 512 | ... | 527 | 528 | ... | 543 |
|
||||
TP2, rank 1:
|
||||
|< -----------BASE----------- >|< -BASE PADDING- >|< -----------LORA PADDING----------- >|
|
||||
|< -----------BASE----------- >|< -BASE PADDING- >|< ----------ADDED PADDING---------- >|
|
||||
corresponding token_id: | 512 | 513 | 514 | ... | 1009 | -1 | ... | -1 | -1 | ... | -1 | -1 | ... | -1 |
|
||||
index: | 0 | 1 | 2 | ... | 497 | 498 | ... | 511 | 512 | ... | 527 | 528 | ... | 543 |
|
||||
|
||||
@@ -236,7 +236,7 @@ class VocabParallelEmbedding(PluggableLayer):
|
||||
num_embeddings: vocabulary size.
|
||||
embedding_dim: size of hidden state.
|
||||
params_dtype: type of the parameters.
|
||||
org_num_embeddings: original vocabulary size (without LoRA).
|
||||
org_num_embeddings: original vocabulary size (without added embeddings).
|
||||
padding_size: padding size for the vocabulary.
|
||||
quant_config: quant config for the layer
|
||||
prefix: full name of the layer in the state dict
|
||||
@@ -587,7 +587,7 @@ class ParallelLMHead(VocabParallelEmbedding):
|
||||
embedding_dim: size of hidden state.
|
||||
bias: whether to use bias.
|
||||
params_dtype: type of the parameters.
|
||||
org_num_embeddings: original vocabulary size (without LoRA).
|
||||
org_num_embeddings: original vocabulary size (without added embeddings).
|
||||
padding_size: padding size for the vocabulary.
|
||||
disable_tp: If true, tensor parallelism will be disabled for this layer.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user