[Quantization] Add ModelOpt IQ2_XS, IQ2_XXS, and Q8_0 support via b12x (#59604)
Signed-off-by: Andrii Skliar <askliar@nvidia.com> Signed-off-by: Luke Alonso <lalonso@gmail.com> Co-authored-by: Andrii Skliar <askliar@nvidia.com> Co-authored-by: Luke Alonso <lalonso@gmail.com>
This commit is contained in:
co-authored by
Andrii Skliar
Luke Alonso
parent
ac71b17517
commit
6849ca696a
@@ -17,7 +17,7 @@ vllm serve <model> \
|
||||
--moe-backend b12x
|
||||
```
|
||||
|
||||
Only pass `--moe-backend b12x` for a compatible NVFP4 or MXFP4 MoE model. The
|
||||
Only pass `--moe-backend b12x` for a compatible packed-weight MoE model. The
|
||||
linear and MoE backends can be selected independently.
|
||||
|
||||
b12x uses MXFP8 activations by default for MXFP4 MoE and the checkpoint's
|
||||
@@ -29,9 +29,34 @@ force BF16 activations for either FP4 weight format.
|
||||
|
||||
| Backend | Supported configurations |
|
||||
| ------- | ------------------------ |
|
||||
| Linear | Per-tensor FP8, 128x128 block FP8, MXFP8, NVFP4, and MXFP4 |
|
||||
| MoE | Tensor-parallel MXFP4 weights with BF16 or MXFP8 activations; NVFP4 weights with BF16, NVFP4, or MXFP8 activations |
|
||||
| Linear | Per-tensor FP8, 128x128 block FP8, MXFP8, NVFP4 (including W4A16), MXFP4, IQ2_XS, IQ2_XXS, and Q8_0 |
|
||||
| MoE | Tensor-parallel MXFP4 weights with BF16 or MXFP8 activations; NVFP4 weights with BF16, NVFP4, or MXFP8 activations; IQ2_XS, IQ2_XXS, and Q8_0 weights with BF16 activations |
|
||||
| Embedding | Packed Q8_0 lookup with BF16 output |
|
||||
|
||||
Dense W4A16 layers are not handled by b12x and continue to use another
|
||||
compatible backend such as Marlin. The b12x MoE backend does not support expert
|
||||
parallelism, expert maps, EXL3, or NF3.
|
||||
The b12x MoE backend does not support expert parallelism, expert maps, EXL3, or NF3.
|
||||
|
||||
## ModelOpt block codecs
|
||||
|
||||
ModelOpt mixed-precision checkpoints select block codecs through the
|
||||
`quantization_config.quantized_layers` entries in `config.json`. Each entry
|
||||
declares `quant_algo`, `group_size`, `block_payload_bytes`, and `packing="ggml"`:
|
||||
|
||||
| `quant_algo` | `group_size` | `block_payload_bytes` |
|
||||
| --- | --- | --- |
|
||||
| `IQ2_XS` | 256 | 74 |
|
||||
| `IQ2_XXS` | 256 | 66 |
|
||||
| `Q8_0` | 32 | 34 |
|
||||
|
||||
Serialized matrices have uint8 shape `[N, K/group_size, block_payload_bytes]`.
|
||||
Dense preparation rearranges the encoded blocks without expanding the matrix
|
||||
to BF16. Routed experts use bias-free SiLU or non-gated ReLU²; tensor-parallel
|
||||
shards must preserve whole blocks and satisfy the kernel's alignment constraints.
|
||||
Only Q8_0 supports packed embedding lookup.
|
||||
|
||||
Quantized normalization weights and other packed vectors are not supported.
|
||||
Store normalization weights as ordinary floating-point tensors and omit their
|
||||
entries from `quantization_config.quantized_layers`.
|
||||
|
||||
Dense, embedding, and expert matrices remain packed. Their execution uses BF16
|
||||
activations without activation quantization. Prepare the kernels and workspaces
|
||||
before CUDA graph capture.
|
||||
|
||||
Reference in New Issue
Block a user