[Bugfix][Quantization] Stop sleep(level=2) from zeroing compressed-tensors KV scales (#57163)
Signed-off-by: AlanFokCo <alanfok2868@gmail.com> Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: aoshen02 <aoshen@inferact.ai>
This commit is contained in:
co-authored by
Claude Opus 5.5
aoshen02
parent
8aaeef343a
commit
f4917dadc8
@@ -1208,9 +1208,10 @@ class CompressedTensorsKVCacheMethod(BaseKVCacheMethod):
|
||||
"""Override the default vLLM placeholder scales with the llm-compressor loaded
|
||||
scales. Zero points are not used as only symmetric quantization is supported.
|
||||
"""
|
||||
layer._k_scale = layer.k_scale
|
||||
layer._v_scale = layer.v_scale
|
||||
layer._q_scale = layer.q_scale
|
||||
# Buffers (not Parameters) so sleep snapshots them; ATTN_HEAD changes shape.
|
||||
layer.register_buffer("_k_scale", layer.k_scale.detach())
|
||||
layer.register_buffer("_v_scale", layer.v_scale.detach())
|
||||
layer.register_buffer("_q_scale", layer.q_scale.detach())
|
||||
|
||||
# Set the _float variants that the attention backend uses.
|
||||
def _to_scalar(tensor: torch.Tensor) -> float:
|
||||
|
||||
Reference in New Issue
Block a user