[Online quantization] Extend re-quantization API to MOE (showcase NVFP4 modelopt/CT/quark to any online MOE method) (#60276)

Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com>
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: Tan Pin Siang <tanpinsiang@gmail.com>
This commit is contained in:
fxmarty-amd
2026-10-14 15:49:37 +08:00
committed by GitHub
co-authored by vllmellm Tan Pin Siang
parent e1726d9142
commit c9bfa574ce
16 changed files with 1353 additions and 148 deletions
+14 -3
View File
@@ -111,7 +111,10 @@ Currently supported combinations are:
| Checkpoint layer | Online target | Requirements |
| ---------------- | ------------- | ------------ |
| Serialized ModelOpt MXFP8 linear | any | Target-specific hardware requirements |
| Serialized ModelOpt MXFP8 linear | any online linear method | Target-specific hardware requirements |
| Serialized compressed-tensors NVFP4 MoE | any online MoE method | Target-specific hardware requirements |
| Serialized ModelOpt NVFP4 MoE | any online MoE method | Target-specific hardware requirements |
| Serialized Quark NVFP4 MoE | any online MoE method | Target-specific hardware requirements |
For example, serialized ModelOpt MXFP8 linears can be converted to
per-channel-weight/per-token-activation FP8 at load time:
@@ -122,10 +125,18 @@ vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
--quantization-config.linear fp8_per_channel
```
Similarly, for example, NVFP4 MoE checkpoints can be re-quantized to any online MoE method,
for example compressed-tensors NVFP4 experts to online MXFP4:
```bash
vllm serve RedHatAI/Qwen3-30B-A3B-NVFP4 \
--quantization-config '{"targets":{"*mlp.experts*":"mxfp4"}}'
```
!!! info
This is only available for linear layers at the moment.
Re-quantization is available for both linear and MoE layers.
Checkpoint-exclusion policy is not enforced by online quantization/re-quantization. Only exclusion through `--quantization-config.targets`/`--quantization-config.ignore` is enforced.
### Online quantization on unquantized layers from partially-quantized checkpoints