[Online quantization] Extend re-quantization API to MOE (showcase NVFP4 modelopt/CT/quark to any online MOE method) (#60276)
Signed-off-by: Felix Marty <Felix.Marty@amd.com> Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com> Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com> Co-authored-by: Tan Pin Siang <tanpinsiang@gmail.com>
This commit is contained in:
co-authored by
vllmellm
Tan Pin Siang
parent
e1726d9142
commit
c9bfa574ce
@@ -111,7 +111,10 @@ Currently supported combinations are:
|
||||
|
||||
| Checkpoint layer | Online target | Requirements |
|
||||
| ---------------- | ------------- | ------------ |
|
||||
| Serialized ModelOpt MXFP8 linear | any | Target-specific hardware requirements |
|
||||
| Serialized ModelOpt MXFP8 linear | any online linear method | Target-specific hardware requirements |
|
||||
| Serialized compressed-tensors NVFP4 MoE | any online MoE method | Target-specific hardware requirements |
|
||||
| Serialized ModelOpt NVFP4 MoE | any online MoE method | Target-specific hardware requirements |
|
||||
| Serialized Quark NVFP4 MoE | any online MoE method | Target-specific hardware requirements |
|
||||
|
||||
For example, serialized ModelOpt MXFP8 linears can be converted to
|
||||
per-channel-weight/per-token-activation FP8 at load time:
|
||||
@@ -122,10 +125,18 @@ vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
|
||||
--quantization-config.linear fp8_per_channel
|
||||
```
|
||||
|
||||
Similarly, for example, NVFP4 MoE checkpoints can be re-quantized to any online MoE method,
|
||||
for example compressed-tensors NVFP4 experts to online MXFP4:
|
||||
|
||||
```bash
|
||||
vllm serve RedHatAI/Qwen3-30B-A3B-NVFP4 \
|
||||
--quantization-config '{"targets":{"*mlp.experts*":"mxfp4"}}'
|
||||
```
|
||||
|
||||
!!! info
|
||||
|
||||
This is only available for linear layers at the moment.
|
||||
|
||||
Re-quantization is available for both linear and MoE layers.
|
||||
|
||||
Checkpoint-exclusion policy is not enforced by online quantization/re-quantization. Only exclusion through `--quantization-config.targets`/`--quantization-config.ignore` is enforced.
|
||||
|
||||
### Online quantization on unquantized layers from partially-quantized checkpoints
|
||||
|
||||
Reference in New Issue
Block a user