[Attention Backend] TurboQuant: 2-bit KV cache compression with 4x capacity (#38479)
Signed-off-by: vibhavagarwal5 <vibhavagarwal5@gmail.com> Signed-off-by: Michael Goin <mgoin64@gmail.com> Co-authored-by: Xinyu Chen <xinyu1.chen@intel.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>
This commit is contained in:
co-authored by
Xinyu Chen
Michael Goin
parent
3bfe55a037
commit
f4b42df048
@@ -42,6 +42,10 @@ STR_DTYPE_TO_TORCH_DTYPE = {
|
||||
"fp8_per_token_head": torch.uint8,
|
||||
"fp8_inc": torch.float8_e4m3fn,
|
||||
"fp8_ds_mla": torch.uint8,
|
||||
"turboquant_k8v4": torch.uint8,
|
||||
"turboquant_4bit_nc": torch.uint8,
|
||||
"turboquant_k3v4_nc": torch.uint8,
|
||||
"turboquant_3bit_nc": torch.uint8,
|
||||
}
|
||||
|
||||
TORCH_DTYPE_TO_NUMPY_DTYPE = {
|
||||
|
||||
Reference in New Issue
Block a user