[Quantization] Layer re-quantization for linear layers through online quantization API (MXFP8 -> FP8 PTPC showcase) (#55684)
Signed-off-by: Felix Marty <Felix.Marty@amd.com> Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com> Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com> Co-authored-by: Tan Pin Siang <tanpinsiang@gmail.com> Co-authored-by: Misha Goin <mgoin64@gmail.com>
This commit is contained in:
co-authored by
vllmellm
Tan Pin Siang
Misha Goin
parent
29f955cce0
commit
f8d92cae41
@@ -99,6 +99,35 @@ vllm serve openai/gpt-oss-20b --quantization-config.moe.activation mxfp8
|
||||
|
||||
Combine with `--moe-backend` to pin a specific kernel family.
|
||||
|
||||
### Re-quantizing layers to a different precision
|
||||
|
||||
An online target can re-quantize a checkpoint-quantized (prequantized) layer at load time.
|
||||
|
||||
vLLM uses the checkpoint method to load and dequantize the checkpoint weight, and then applies the online quantization method. This requires the checkpoint
|
||||
method to implement weight dequantization, unsupported source and target
|
||||
combinations raise an error.
|
||||
|
||||
Currently supported combinations are:
|
||||
|
||||
| Checkpoint layer | Online target | Requirements |
|
||||
| ---------------- | ------------- | ------------ |
|
||||
| Serialized ModelOpt MXFP8 linear | any | Target-specific hardware requirements |
|
||||
|
||||
For example, serialized ModelOpt MXFP8 linears can be converted to
|
||||
per-channel-weight/per-token-activation FP8 at load time:
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
|
||||
--linear-backend auto \
|
||||
--quantization-config.linear fp8_per_channel
|
||||
```
|
||||
|
||||
!!! info
|
||||
|
||||
This is only available for linear layers at the moment.
|
||||
|
||||
Checkpoint-exclusion policy is not enforced by online quantization/re-quantization. Only exclusion through `--quantization-config.targets`/`--quantization-config.ignore` is enforced.
|
||||
|
||||
### Online quantization on unquantized layers from partially-quantized checkpoints
|
||||
|
||||
Online quantization can be used on already quantized checkpoints independently of their original `quant_method` (`modelopt`, `compressed-tensors`, `quark`, etc.), for layers that are left unquantized in the original checkpoint.
|
||||
|
||||
Reference in New Issue
Block a user