Use CU_MEMCPY_SRC_ACCESS_ORDER_ANY for batch KV cache swaps (#39306)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com> Signed-off-by: Itay Etelis <etelis2019@gmail.com> Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com> Co-authored-by: Itay Etelis <itay.etelis@ibm.com> Co-authored-by: Or Ozeri <oro@il.ibm.com> Co-authored-by: Itay Etelis <etelis2019@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
Itay Etelis
Or Ozeri
Itay Etelis
mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent
0d382ecde8
commit
00b0618a03
+10
-1
@@ -2805,6 +2805,7 @@ def swap_blocks_batch(
|
||||
src_ptrs: torch.Tensor,
|
||||
dst_ptrs: torch.Tensor,
|
||||
sizes: torch.Tensor,
|
||||
is_src_access_order_any: bool = False,
|
||||
) -> None:
|
||||
"""
|
||||
Batch version of swap_blocks: submit all copies in a single driver call.
|
||||
@@ -2813,8 +2814,16 @@ def swap_blocks_batch(
|
||||
of sizes[i] bytes. All three tensors must be int64 CPU tensors.
|
||||
On CUDA 12.8+ this uses cuMemcpyBatchAsync for minimal submission
|
||||
overhead; on older CUDA it falls back to a loop of cudaMemcpyAsync.
|
||||
|
||||
is_src_access_order_any: if True, pass CU_MEMCPY_SRC_ACCESS_ORDER_ANY to
|
||||
cuMemcpyBatchAsync, letting the DMA engine prefetch source bytes
|
||||
out of stream order. Only safe when no GPU stream is concurrently
|
||||
writing to the source. Defaults to False (STREAM ordering), which
|
||||
is always safe.
|
||||
"""
|
||||
torch.ops._C_cache_ops.swap_blocks_batch(src_ptrs, dst_ptrs, sizes)
|
||||
torch.ops._C_cache_ops.swap_blocks_batch(
|
||||
src_ptrs, dst_ptrs, sizes, is_src_access_order_any
|
||||
)
|
||||
|
||||
|
||||
def convert_fp8(
|
||||
|
||||
Reference in New Issue
Block a user