[Kernel] Add FlashInfer TRTLLM MXFP8 linear backend (#52204)

Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
This commit is contained in:
Seonjin
2026-08-20 10:07:22 -04:00
committed by GitHub
co-authored by Misha Goin
parent de216b6e64
commit bd8865a299
5 changed files with 265 additions and 0 deletions
+5
View File
@@ -33,6 +33,11 @@ following `quantization.quant_algo` values:
[Engine Arguments](../../configuration/engine_args.md) page and shown by
`vllm serve --help=KernelConfig`.
!!! note
For models quantized to MXFP8 with BF16 activations on SM100-family GPUs,
use `--linear-backend flashinfer_trtllm` to select FlashInfer's TensorRT-LLM
GEMM backend.
## Quantizing HuggingFace Models with PTQ
You can quantize HuggingFace models using the example scripts provided in the Model Optimizer repository. The primary script for LLM PTQ is typically found within the `examples/llm_ptq` directory.