diff --git a/docs/features/quantization/modelopt.md b/docs/features/quantization/modelopt.md index ad417bcb30..7850dc75cc 100644 --- a/docs/features/quantization/modelopt.md +++ b/docs/features/quantization/modelopt.md @@ -19,6 +19,20 @@ following `quantization.quant_algo` values: - `NVFP4`: ModelOpt NVFP4 checkpoints (use `quantization="modelopt_fp4"`). - `MXFP8`: ModelOpt MXFP8 checkpoints (use `quantization="modelopt_mxfp8"`). +!!! note + For NVFP4 checkpoints, vLLM selects a GEMM kernel automatically at load + time from the backends available on the current platform (CUTLASS, + FlashInfer, Marlin, and others). On GPUs without a supported native FP4 + GEMM kernel, vLLM falls back to weight-only (W4A16) execution via Marlin + and logs a warning; this may reduce throughput for compute-heavy + workloads. Use `--linear-backend` to override the automatic selection + (this replaces the deprecated `VLLM_NVFP4_GEMM_BACKEND` environment + variable). Values relevant to NVFP4 include `cutlass`, + `flashinfer_cutlass`, `flashinfer_trtllm`, `flashinfer_cudnn`, and + `marlin`; the full list is documented under `KernelConfig` on the + [Engine Arguments](../../configuration/engine_args.md) page and shown by + `vllm serve --help=KernelConfig`. + ## Quantizing HuggingFace Models with PTQ You can quantize HuggingFace models using the example scripts provided in the Model Optimizer repository. The primary script for LLM PTQ is typically found within the `examples/llm_ptq` directory.