[Quantization] Rework quantization_config to use QuantKey and allow for activation override (#41566)

This commit is contained in:
Michael Goin
2026-05-13 16:58:32 -04:00
committed by GitHub
parent cca32d55a2
commit 8efd508204
16 changed files with 621 additions and 322 deletions
+45 -3
View File
@@ -42,18 +42,59 @@ vllm serve meta-llama/Llama-3.1-8B --quantization mxfp8
For fine-grained control, use a `quantization_config` dictionary.
### Schema
```yaml
quantization_config:
linear:
weight: <name> # see QUANT_KEY_NAMES in vllm/config/quantization.py
activation: <name>
moe:
weight: <name>
activation: <name>
ignore: [<layer-name-or-regex>, ...]
```
`linear` and `moe` accept a full `{weight, activation}` dict, or a bare
string. A string resolves first against the `--quantization` shorthands
(taking the matching layer-kind slot), then against `QUANT_KEY_NAMES` as a
weight name. Unset fields fall back to the `--quantization` shorthand's
defaults, or for already-quantized checkpoints to whatever the checkpoint
declares.
The CLI accepts the same shape as JSON or as dotted keys:
```bash
vllm serve <model> --quantization-config '{"moe":{"activation":"mxfp8"}}'
vllm serve <model> --quantization-config.moe.activation mxfp8
```
### Activation overrides on already-quantized checkpoints
For checkpoint-quantized models, `quantization_config` lets you pick an
activation format independently of the baked-in weights. The supported
overrides are checkpoint-specific; today this is wired up for MXFP4 MoE
checkpoints (gpt-oss) where you can opt into FP8 activations:
```bash
vllm serve openai/gpt-oss-20b --quantization-config.moe.activation mxfp8
```
Combine with `--moe-backend` to pin a specific kernel family.
### Separate Schemes for Dense and MoE Layers
You can apply different quantization schemes to dense linear layers and MoE expert layers:
You can apply different quantization schemes to dense linear layers and MoE expert layers via the `linear` and `moe` fields. Each accepts either a full spec dict, or a bare string naming an online shorthand (e.g. `"fp8_per_block"`) or weight format (e.g. `"fp8_per_block_static"`); fields not set fall back to the shorthand defaults.
```python
from vllm import LLM
# Linear: per-block FP8; MoE: per-tensor FP8 (inherited from the shorthand)
llm = LLM(
"ibm-granite/granite-3.0-1b-a400m-base",
quantization="fp8_per_tensor",
quantization_config={
"linear_scheme_override": "fp8_per_block",
"linear": "fp8_per_block",
},
)
```
@@ -63,11 +104,12 @@ Or,
```python
from vllm import LLM
# Linear: per-tensor FP8 (inherited); MoE: per-block FP8
llm = LLM(
"ibm-granite/granite-3.0-1b-a400m-base",
quantization="fp8_per_tensor",
quantization_config={
"moe_scheme_override": "fp8_per_block",
"moe": "fp8_per_block",
},
)
```