[Kernel] Refactor FP8 kv-cache with NVIDIA float8_e4m3 support (#4535)

This commit is contained in:
Cody Yu
2024-05-09 18:04:17 -06:00
committed by GitHub
parent 379da6dcb5
commit c833101740
17 changed files with 843 additions and 558 deletions
+3 -1
View File
@@ -34,5 +34,7 @@ void reshape_and_cache_flash(
// Just for unittest
void convert_fp8(
torch::Tensor& dst_cache,
torch::Tensor& src_cache,
torch::Tensor& dst_cache);
const float scale,
const std::string& kv_cache_dtype);