[Transformers backend] Find attention with a fuser and attach vLLM's layer to it (#54941)

Signed-off-by: Thomas Ortner <boh@zurich.ibm.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
This commit is contained in:
Thomas Ortner
2026-09-04 16:26:01 +00:00
committed by GitHub
co-authored by Harry Mellor
parent 99a1ab8e4c
commit 8ad2076c4a
14 changed files with 521 additions and 150 deletions
+32 -15
View File
@@ -72,19 +72,26 @@ This means that, with the Transformers modeling backend for vLLM, new models can
This section details the necessary modifications to make to a Transformers compatible custom model that make it compatible with the Transformers modeling backend for vLLM. (We assume that a Transformers compatible custom model has already been created, see [Transformers - Customizing models](https://huggingface.co/docs/transformers/en/custom_models)).
To make your model compatible with the Transformers modeling backend, it needs:
To make your model compatible with the Transformers modeling backend:
1. `kwargs` passed down through all modules from `MyModel` to `MyAttention`.
- If your model is encoder-only:
1. Add `is_causal = False` to `MyAttention`.
- If your model is mixture-of-experts (MoE):
1. Your sparse MoE block must have an attribute called `experts`.
2. The class of `experts` (`MyExperts`) must either:
- Inherit from `nn.ModuleList` (naive).
- Or contain all 3D `nn.Parameters` (packed).
3. `MyExperts.forward` must accept `hidden_states`, `top_k_index`, `top_k_weights`.
2. `MyAttention` must use `ALL_ATTENTION_FUNCTIONS` to call attention.
3. `MyModel` must contain `_supports_attention_backend = True`.
1. `MyAttention` must use `ALL_ATTENTION_FUNCTIONS` to call attention.
- It must make exactly one such call. vLLM attaches one `Attention` layer per `MyAttention` module.
- `MyAttention` must contain a unique `layer_idx`. vLLM keys its KV cache using this index.
- Pass `scaling=` to the interface if your scale is not `head_size**-0.5`. vLLM reads it from the call to the attention interface.
2. If your model is encoder-only:
1. Add `is_causal = False` to `MyAttention`.
3. If your model is mixture-of-experts (MoE):
1. Your sparse MoE block must have an attribute called `experts`.
2. The class of `experts` (`MyExperts`) must either:
- Inherit from `nn.ModuleList` (naive).
- Or contain all 3D `nn.Parameters` (packed).
3. `MyExperts.forward` must accept `hidden_states`, `top_k_index`, `top_k_weights`.
!!! note
`MyModel` no longer needs `_supports_attention_backend = True`, and `kwargs` no
longer need to be passed down through every module from `MyModel` to
`MyAttention`. vLLM reaches its attention layer through `MyAttention` itself, so
all it asks is that `MyAttention` dispatches through the interface.
<details class="code">
<summary>modeling_my_model.py</summary>
@@ -97,14 +104,24 @@ from torch import nn
class MyAttention(nn.Module):
is_causal = False # Only do this for encoder-only models
def __init__(self, config, layer_idx):
...
self.config = config
self.layer_idx = layer_idx
self.scaling = self.head_dim**-0.5
...
def forward(self, hidden_states, **kwargs):
...
attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]
attention_interface = ALL_ATTENTION_FUNCTIONS.get_interface(
self.config._attn_implementation, eager_attention_forward
)
attn_output, attn_weights = attention_interface(
self,
query_states,
key_states,
value_states,
scaling=self.scaling,
**kwargs,
)
...
@@ -127,7 +144,7 @@ class MySparseMoEBlock(nn.Module):
...
class MyModel(PreTrainedModel):
_supports_attention_backend = True
...
```
</details>
@@ -135,7 +152,7 @@ class MyModel(PreTrainedModel):
Here is what happens in the background when this model is loaded:
1. The config is loaded.
2. `MyModel` Python class is loaded from the `auto_map` in config, and we check that the model `is_backend_compatible()`.
2. `MyModel` Python class is loaded from the `auto_map` in config, and we check that the model `_can_set_attn_implementation()`.
3. `MyModel` is loaded into one of the Transformers modeling backend classes in [vllm/model_executor/models/transformers](../../vllm/model_executor/models/transformers) which sets `self.config._attn_implementation = "vllm"` so that vLLM's attention layer is used.
That's it!