[Bugfix][TurboQuant] Add KV quant mode for turboquant (#50533)
Signed-off-by: Soila Kavulya <soila.p.kavulya@intel.com> Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -680,6 +680,7 @@ class Attention(nn.Module, AttentionLayerBase):
|
||||
head_size=self.head_size,
|
||||
head_size_v=self.head_size,
|
||||
dtype=self.kv_cache_torch_dtype,
|
||||
kv_quant_mode=quant_mode,
|
||||
tq_slot_size=tq_config.slot_size_aligned,
|
||||
)
|
||||
else:
|
||||
|
||||
Reference in New Issue
Block a user