[Bugfix][CuMem] Make KV-cache wake cleanup tag-safe (#49857)
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com> Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
This commit is contained in:
@@ -291,6 +291,9 @@ class CuMemAllocator:
|
|||||||
back to GPU memory. If None, all memory allocation will be loaded
|
back to GPU memory. If None, all memory allocation will be loaded
|
||||||
back to GPU memory.
|
back to GPU memory.
|
||||||
"""
|
"""
|
||||||
|
gc.collect()
|
||||||
|
torch.accelerator.empty_cache()
|
||||||
|
|
||||||
for ptr, data in self.pointer_to_data.items():
|
for ptr, data in self.pointer_to_data.items():
|
||||||
if tags is None or data.tag in tags:
|
if tags is None or data.tag in tags:
|
||||||
handle = data.handle
|
handle = data.handle
|
||||||
|
|||||||
Reference in New Issue
Block a user