[Model Runner v2] Fix block table IMA issue (#40648)
Signed-off-by: yewentao256 <zhyanwentao@126.com> Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Nick Hill <nickhill123@gmail.com>
This commit is contained in:
@@ -884,6 +884,9 @@ class GPUModelRunner(
|
||||
self.encoder_cache.clear()
|
||||
self.late_interaction_runner.clear()
|
||||
|
||||
def post_kv_cache_wake_up(self) -> None:
|
||||
self.init_fp8_kv_scales()
|
||||
|
||||
@torch.inference_mode()
|
||||
def init_fp8_kv_scales(self) -> None:
|
||||
"""
|
||||
|
||||
Reference in New Issue
Block a user