[Bugfix] Fall back to auto for unquantized MoE layers when --moe-backend is marlin (#60219)

Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Vadim Gimpelson
2026-10-13 08:48:13 +00:00
committed by GitHub
co-authored by Claude
parent 6f11c220b5
commit 183625d4f9
2 changed files with 18 additions and 3 deletions
@@ -303,9 +303,10 @@ def select_unquantized_moe_backend(
)
runner_backend = moe_config.moe_backend
# 'humming' is quantization-only; an unquantized layer (e.g. excluded via
# modules_to_not_convert) falls through to auto instead of erroring.
if runner_backend not in ["auto", "humming"]:
# 'humming' and 'marlin' are quantization-only; an unquantized layer (e.g.
# excluded via modules_to_not_convert, or an unquantized MTP draft of a
# quantized model) falls through to auto instead of erroring.
if runner_backend not in ["auto", "humming", "marlin"]:
requested_backend = map_unquantized_backend(runner_backend)
if (
activation_format == mk.FusedMoEActivationFormat.BatchedExperts