[Feature] Triton INT4 per-token-head KV cache quantization (#40835)
Signed-off-by: JartX <sagformas@epdcenter.es> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent
d20dbf921b
commit
70749fdcca
@@ -39,6 +39,7 @@ STR_DTYPE_TO_TORCH_DTYPE = {
|
||||
"fp8_e4m3": torch.uint8,
|
||||
"fp8_e5m2": torch.uint8,
|
||||
"int8": torch.int8,
|
||||
"int4_per_token_head": torch.uint8,
|
||||
"int8_per_token_head": torch.int8,
|
||||
"fp8_per_token_head": torch.uint8,
|
||||
"fp8_inc": torch.float8_e4m3fn,
|
||||
|
||||
Reference in New Issue
Block a user