[Docs] Document NVFP4 GEMM kernel selection and Marlin weight-only fallback (#49376)

Signed-off-by: harjoth <harjoth.khara@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Harjoth Khara
2026-07-26 20:53:28 -07:00
committed by GitHub
co-authored by Claude Opus 5
parent da99ffcc13
commit 8de50e46d4
+14
View File
@@ -19,6 +19,20 @@ following `quantization.quant_algo` values:
- `NVFP4`: ModelOpt NVFP4 checkpoints (use `quantization="modelopt_fp4"`).
- `MXFP8`: ModelOpt MXFP8 checkpoints (use `quantization="modelopt_mxfp8"`).
!!! note
For NVFP4 checkpoints, vLLM selects a GEMM kernel automatically at load
time from the backends available on the current platform (CUTLASS,
FlashInfer, Marlin, and others). On GPUs without a supported native FP4
GEMM kernel, vLLM falls back to weight-only (W4A16) execution via Marlin
and logs a warning; this may reduce throughput for compute-heavy
workloads. Use `--linear-backend` to override the automatic selection
(this replaces the deprecated `VLLM_NVFP4_GEMM_BACKEND` environment
variable). Values relevant to NVFP4 include `cutlass`,
`flashinfer_cutlass`, `flashinfer_trtllm`, `flashinfer_cudnn`, and
`marlin`; the full list is documented under `KernelConfig` on the
[Engine Arguments](../../configuration/engine_args.md) page and shown by
`vllm serve --help=KernelConfig`.
## Quantizing HuggingFace Models with PTQ
You can quantize HuggingFace models using the example scripts provided in the Model Optimizer repository. The primary script for LLM PTQ is typically found within the `examples/llm_ptq` directory.