[Kernel][Amd] Add fp8 kv cache support for rocm custom paged attention (#8577)

This commit is contained in:
Charlie Fu
2024-09-19 17:37:57 +00:00
committed by GitHub
parent 76515f303b
commit 9cc373f390
6 changed files with 245 additions and 282 deletions
+2 -1
View File
@@ -26,7 +26,8 @@ TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, rocm_ops) {
" Tensor context_lens, int block_size,"
" int max_context_len,"
" Tensor? alibi_slopes,"
" str kv_cache_dtype) -> ()");
" str kv_cache_dtype,"
" float k_scale, float v_scale) -> ()");
rocm_ops.impl("paged_attention", torch::kCUDA, &paged_attention);
}