[Transformers backend] Find attention with a fuser and attach vLLM's layer to it (#54941)
Signed-off-by: Thomas Ortner <boh@zurich.ibm.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
This commit is contained in:
co-authored by
Harry Mellor
parent
99a1ab8e4c
commit
8ad2076c4a
@@ -1253,7 +1253,10 @@ class _ModelRegistry:
|
||||
"'auto_map' (relevant if the model is custom)."
|
||||
)
|
||||
|
||||
if not model_module.is_backend_compatible():
|
||||
if not (
|
||||
model_module.is_backend_compatible()
|
||||
or model_module._can_set_attn_implementation()
|
||||
):
|
||||
if model_config.model_impl != "transformers":
|
||||
return None
|
||||
|
||||
|
||||
Reference in New Issue
Block a user