[Kernel] Add b12x FP4 MoE backend (#52018)

Signed-off-by: Luke Alonso <lalonso@gmail.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
This commit is contained in:
Luke Alonso
2026-08-21 08:04:36 -07:00
committed by GitHub
co-authored by mgoin
parent 5ee84d3c52
commit 2740c817ff
23 changed files with 2417 additions and 60 deletions
+20 -7
View File
@@ -1,24 +1,37 @@
# B12X Linear Backend
# b12x Linear and MoE Backends
[B12X](https://pypi.org/project/b12x/) provides optional CUDA kernels for
[b12x](https://pypi.org/project/b12x/) provides optional CUDA kernels for
NVIDIA SM120 and SM121 GPUs. Install the dependency with:
```bash
uv pip install "vllm[b12x]"
```
B12X participates in automatic kernel selection after established optimized
backends and before emulation. Select it explicitly with:
b12x linear kernels participate in automatic selection after established
optimized backends and before emulation. Select the linear and MoE backends
explicitly with:
```bash
vllm serve <model> --linear-backend b12x
vllm serve <model> \
--linear-backend b12x \
--moe-backend b12x
```
Only pass `--moe-backend b12x` for a compatible NVFP4 or MXFP4 MoE model. The
linear and MoE backends can be selected independently.
b12x uses MXFP8 activations by default for MXFP4 MoE and the checkpoint's
activation format for NVFP4 MoE. MXFP4 falls back to BF16 when its A8 path does
not support the model configuration. Set `VLLM_B12X_MOE_FP4_FORCE_A16=1` to
force BF16 activations for either FP4 weight format.
## Supported Configurations
| Backend | Supported configurations |
| ------- | ------------------------ |
| Linear | Per-tensor FP8, 128x128 block FP8, MXFP8, NVFP4, and MXFP4 |
| MoE | Tensor-parallel MXFP4 weights with BF16 or MXFP8 activations; NVFP4 weights with BF16, NVFP4, or MXFP8 activations |
Dense W4A16 layers are not handled by B12X and continue to use another
compatible backend such as Marlin.
Dense W4A16 layers are not handled by b12x and continue to use another
compatible backend such as Marlin. The b12x MoE backend does not support expert
parallelism, expert maps, EXL3, or NF3.