[Misc] Use meta tensor for KV cache stride calculation (#47316)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent
9c18e90f6c
commit
04d553f390
@@ -238,7 +238,7 @@ def _reshape_attention_kv_cache(
|
||||
page_stride = kv_cache_spec.page_size_bytes // dtype_size
|
||||
|
||||
num_blocks_dim = inv_order[0]
|
||||
strides = list(torch.empty(permuted_kv_cache_shape).stride())
|
||||
strides = list(torch.empty(permuted_kv_cache_shape, device="meta").stride())
|
||||
strides[num_blocks_dim] = page_stride
|
||||
|
||||
kv_cache = torch.as_strided(
|
||||
|
||||
Reference in New Issue
Block a user