[Bugfix] Include inline per-token-head scales in offloaded page transfer width (#48411)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com> Signed-off-by: Itay Etelis <Itay.etelis@gmail.com> Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com> Co-authored-by: Itay Etelis <itay.etelis@ibm.com> Co-authored-by: Itay Etelis <Itay.etelis@gmail.com>
This commit is contained in:
co-authored by
Itay Etelis
Itay Etelis
parent
7a74a9662b
commit
32aef44388
@@ -106,7 +106,7 @@ class OffloadingConnectorWorker:
|
||||
)
|
||||
page_size_bytes[layer_name] = layer_kv_cache_spec.page_size_bytes
|
||||
unpadded_page_size_bytes[layer_name] = (
|
||||
layer_kv_cache_spec.real_page_size_bytes
|
||||
layer_kv_cache_spec.unpadded_page_size_bytes
|
||||
)
|
||||
|
||||
elif isinstance(layer_kv_cache_spec, MambaSpec):
|
||||
|
||||
Reference in New Issue
Block a user