[Quantization] Layer re-quantization for linear layers through online quantization API (MXFP8 -> FP8 PTPC showcase) (#55684)

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com>
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: Tan Pin Siang <tanpinsiang@gmail.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
This commit is contained in:
fxmarty-amd
2026-10-06 11:38:18 -05:00
committed by GitHub
co-authored by vllmellm Tan Pin Siang Misha Goin
parent 29f955cce0
commit f8d92cae41
12 changed files with 422 additions and 45 deletions
+29
View File
@@ -99,6 +99,35 @@ vllm serve openai/gpt-oss-20b --quantization-config.moe.activation mxfp8
Combine with `--moe-backend` to pin a specific kernel family.
### Re-quantizing layers to a different precision
An online target can re-quantize a checkpoint-quantized (prequantized) layer at load time.
vLLM uses the checkpoint method to load and dequantize the checkpoint weight, and then applies the online quantization method. This requires the checkpoint
method to implement weight dequantization, unsupported source and target
combinations raise an error.
Currently supported combinations are:
| Checkpoint layer | Online target | Requirements |
| ---------------- | ------------- | ------------ |
| Serialized ModelOpt MXFP8 linear | any | Target-specific hardware requirements |
For example, serialized ModelOpt MXFP8 linears can be converted to
per-channel-weight/per-token-activation FP8 at load time:
```bash
vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
--linear-backend auto \
--quantization-config.linear fp8_per_channel
```
!!! info
This is only available for linear layers at the moment.
Checkpoint-exclusion policy is not enforced by online quantization/re-quantization. Only exclusion through `--quantization-config.targets`/`--quantization-config.ignore` is enforced.
### Online quantization on unquantized layers from partially-quantized checkpoints
Online quantization can be used on already quantized checkpoints independently of their original `quant_method` (`modelopt`, `compressed-tensors`, `quark`, etc.), for layers that are left unquantized in the original checkpoint.