Use CU_MEMCPY_SRC_ACCESS_ORDER_ANY for batch KV cache swaps (#39306)

Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Signed-off-by: Itay Etelis <etelis2019@gmail.com>
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Itay Etelis <etelis2019@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
Itay Etelis
2026-05-10 05:57:09 +03:00
committed by GitHub
co-authored by Itay Etelis Or Ozeri Itay Etelis mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent 0d382ecde8
commit 00b0618a03
5 changed files with 35 additions and 6 deletions
+10 -1
View File
@@ -2805,6 +2805,7 @@ def swap_blocks_batch(
src_ptrs: torch.Tensor,
dst_ptrs: torch.Tensor,
sizes: torch.Tensor,
is_src_access_order_any: bool = False,
) -> None:
"""
Batch version of swap_blocks: submit all copies in a single driver call.
@@ -2813,8 +2814,16 @@ def swap_blocks_batch(
of sizes[i] bytes. All three tensors must be int64 CPU tensors.
On CUDA 12.8+ this uses cuMemcpyBatchAsync for minimal submission
overhead; on older CUDA it falls back to a loop of cudaMemcpyAsync.
is_src_access_order_any: if True, pass CU_MEMCPY_SRC_ACCESS_ORDER_ANY to
cuMemcpyBatchAsync, letting the DMA engine prefetch source bytes
out of stream order. Only safe when no GPU stream is concurrently
writing to the source. Defaults to False (STREAM ordering), which
is always safe.
"""
torch.ops._C_cache_ops.swap_blocks_batch(src_ptrs, dst_ptrs, sizes)
torch.ops._C_cache_ops.swap_blocks_batch(
src_ptrs, dst_ptrs, sizes, is_src_access_order_any
)
def convert_fp8(