[Docs] Document NVFP4 GEMM kernel selection and Marlin weight-only fallback (#49376)
Signed-off-by: harjoth <harjoth.khara@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
da99ffcc13
commit
8de50e46d4
@@ -19,6 +19,20 @@ following `quantization.quant_algo` values:
|
||||
- `NVFP4`: ModelOpt NVFP4 checkpoints (use `quantization="modelopt_fp4"`).
|
||||
- `MXFP8`: ModelOpt MXFP8 checkpoints (use `quantization="modelopt_mxfp8"`).
|
||||
|
||||
!!! note
|
||||
For NVFP4 checkpoints, vLLM selects a GEMM kernel automatically at load
|
||||
time from the backends available on the current platform (CUTLASS,
|
||||
FlashInfer, Marlin, and others). On GPUs without a supported native FP4
|
||||
GEMM kernel, vLLM falls back to weight-only (W4A16) execution via Marlin
|
||||
and logs a warning; this may reduce throughput for compute-heavy
|
||||
workloads. Use `--linear-backend` to override the automatic selection
|
||||
(this replaces the deprecated `VLLM_NVFP4_GEMM_BACKEND` environment
|
||||
variable). Values relevant to NVFP4 include `cutlass`,
|
||||
`flashinfer_cutlass`, `flashinfer_trtllm`, `flashinfer_cudnn`, and
|
||||
`marlin`; the full list is documented under `KernelConfig` on the
|
||||
[Engine Arguments](../../configuration/engine_args.md) page and shown by
|
||||
`vllm serve --help=KernelConfig`.
|
||||
|
||||
## Quantizing HuggingFace Models with PTQ
|
||||
|
||||
You can quantize HuggingFace models using the example scripts provided in the Model Optimizer repository. The primary script for LLM PTQ is typically found within the `examples/llm_ptq` directory.
|
||||
|
||||
Reference in New Issue
Block a user