[Bugfix][Quantization] Stop sleep(level=2) from zeroing compressed-tensors KV scales (#57163)

Signed-off-by: AlanFokCo <alanfok2868@gmail.com>
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
This commit is contained in:
AlanFokCo
2026-09-29 15:34:15 +00:00
committed by GitHub
co-authored by Claude Opus 5.5 aoshen02
parent 8aaeef343a
commit f4917dadc8
2 changed files with 58 additions and 3 deletions
@@ -1208,9 +1208,10 @@ class CompressedTensorsKVCacheMethod(BaseKVCacheMethod):
"""Override the default vLLM placeholder scales with the llm-compressor loaded
scales. Zero points are not used as only symmetric quantization is supported.
"""
layer._k_scale = layer.k_scale
layer._v_scale = layer.v_scale
layer._q_scale = layer.q_scale
# Buffers (not Parameters) so sleep snapshots them; ATTN_HEAD changes shape.
layer.register_buffer("_k_scale", layer.k_scale.detach())
layer.register_buffer("_v_scale", layer.v_scale.detach())
layer.register_buffer("_q_scale", layer.q_scale.detach())
# Set the _float variants that the attention backend uses.
def _to_scalar(tensor: torch.Tensor) -> float: