[XPU][CI] Fix ExampleConnector KV cache device selection (#51806)
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com> Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
This commit is contained in:
co-authored by
Kunshang Ji
parent
5af7c8dad7
commit
d648236199
@@ -180,7 +180,7 @@ class ExampleConnector(KVConnectorBase_V1):
|
||||
layer_name, request.token_ids, request.mm_hashes
|
||||
)
|
||||
kv_cache_cpu = safetensors.torch.load_file(filename)["kv_cache"]
|
||||
kv_cache = kv_cache_cpu.to("cuda", non_blocking=True)
|
||||
kv_cache = kv_cache_cpu.to(kv_cache_layer.device, non_blocking=True)
|
||||
if isinstance(attn_metadata, dict):
|
||||
inject_kv_into_layer(
|
||||
kv_cache_layer,
|
||||
|
||||
Reference in New Issue
Block a user