From 8de50e46d4a0a227db1888410cb6ac93058fb6cf Mon Sep 17 00:00:00 2001 From: Harjoth Khara Date: Sun, 26 Jul 2026 20:53:28 -0700 Subject: [PATCH] [Docs] Document NVFP4 GEMM kernel selection and Marlin weight-only fallback (#49376) Signed-off-by: harjoth Co-authored-by: Claude Opus 5 --- docs/features/quantization/modelopt.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/features/quantization/modelopt.md b/docs/features/quantization/modelopt.md index ad417bcb30..7850dc75cc 100644 --- a/docs/features/quantization/modelopt.md +++ b/docs/features/quantization/modelopt.md @@ -19,6 +19,20 @@ following `quantization.quant_algo` values: - `NVFP4`: ModelOpt NVFP4 checkpoints (use `quantization="modelopt_fp4"`). - `MXFP8`: ModelOpt MXFP8 checkpoints (use `quantization="modelopt_mxfp8"`). +!!! note + For NVFP4 checkpoints, vLLM selects a GEMM kernel automatically at load + time from the backends available on the current platform (CUTLASS, + FlashInfer, Marlin, and others). On GPUs without a supported native FP4 + GEMM kernel, vLLM falls back to weight-only (W4A16) execution via Marlin + and logs a warning; this may reduce throughput for compute-heavy + workloads. Use `--linear-backend` to override the automatic selection + (this replaces the deprecated `VLLM_NVFP4_GEMM_BACKEND` environment + variable). Values relevant to NVFP4 include `cutlass`, + `flashinfer_cutlass`, `flashinfer_trtllm`, `flashinfer_cudnn`, and + `marlin`; the full list is documented under `KernelConfig` on the + [Engine Arguments](../../configuration/engine_args.md) page and shown by + `vllm serve --help=KernelConfig`. + ## Quantizing HuggingFace Models with PTQ You can quantize HuggingFace models using the example scripts provided in the Model Optimizer repository. The primary script for LLM PTQ is typically found within the `examples/llm_ptq` directory.