[Quantization] Add ModelOpt IQ2_XS, IQ2_XXS, and Q8_0 support via b12x (#59604)

Signed-off-by: Andrii Skliar <askliar@nvidia.com>
Signed-off-by: Luke Alonso <lalonso@gmail.com>
Co-authored-by: Andrii Skliar <askliar@nvidia.com>
Co-authored-by: Luke Alonso <lalonso@gmail.com>
This commit is contained in:
Andrii Skliar
2026-10-13 09:06:33 -07:00
committed by GitHub
co-authored by Andrii Skliar Luke Alonso
parent ac71b17517
commit 6849ca696a
15 changed files with 1238 additions and 25 deletions
+31 -6
View File
@@ -17,7 +17,7 @@ vllm serve <model> \
--moe-backend b12x
```
Only pass `--moe-backend b12x` for a compatible NVFP4 or MXFP4 MoE model. The
Only pass `--moe-backend b12x` for a compatible packed-weight MoE model. The
linear and MoE backends can be selected independently.
b12x uses MXFP8 activations by default for MXFP4 MoE and the checkpoint's
@@ -29,9 +29,34 @@ force BF16 activations for either FP4 weight format.
| Backend | Supported configurations |
| ------- | ------------------------ |
| Linear | Per-tensor FP8, 128x128 block FP8, MXFP8, NVFP4, and MXFP4 |
| MoE | Tensor-parallel MXFP4 weights with BF16 or MXFP8 activations; NVFP4 weights with BF16, NVFP4, or MXFP8 activations |
| Linear | Per-tensor FP8, 128x128 block FP8, MXFP8, NVFP4 (including W4A16), MXFP4, IQ2_XS, IQ2_XXS, and Q8_0 |
| MoE | Tensor-parallel MXFP4 weights with BF16 or MXFP8 activations; NVFP4 weights with BF16, NVFP4, or MXFP8 activations; IQ2_XS, IQ2_XXS, and Q8_0 weights with BF16 activations |
| Embedding | Packed Q8_0 lookup with BF16 output |
Dense W4A16 layers are not handled by b12x and continue to use another
compatible backend such as Marlin. The b12x MoE backend does not support expert
parallelism, expert maps, EXL3, or NF3.
The b12x MoE backend does not support expert parallelism, expert maps, EXL3, or NF3.
## ModelOpt block codecs
ModelOpt mixed-precision checkpoints select block codecs through the
`quantization_config.quantized_layers` entries in `config.json`. Each entry
declares `quant_algo`, `group_size`, `block_payload_bytes`, and `packing="ggml"`:
| `quant_algo` | `group_size` | `block_payload_bytes` |
| --- | --- | --- |
| `IQ2_XS` | 256 | 74 |
| `IQ2_XXS` | 256 | 66 |
| `Q8_0` | 32 | 34 |
Serialized matrices have uint8 shape `[N, K/group_size, block_payload_bytes]`.
Dense preparation rearranges the encoded blocks without expanding the matrix
to BF16. Routed experts use bias-free SiLU or non-gated ReLU²; tensor-parallel
shards must preserve whole blocks and satisfy the kernel's alignment constraints.
Only Q8_0 supports packed embedding lookup.
Quantized normalization weights and other packed vectors are not supported.
Store normalization weights as ordinary floating-point tensors and omit their
entries from `quantization_config.quantized_layers`.
Dense, embedding, and expert matrices remain packed. Their execution uses BF16
activations without activation quantization. Prepare the kernels and workspaces
before CUDA graph capture.