[Frontend][Core] Add release_kv_cache_memory() API (#44890)
Signed-off-by: Dakai An <dakaian108@gmail.com> Signed-off-by: AlanFokCo <alanfok2868@gmail.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com> Co-authored-by: AlanFokCo <alanfok2868@gmail.com> Co-authored-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: OpenAI Codex <noreply@openai.com>
This commit is contained in:
co-authored by
AlanFokCo
aoshen02
OpenAI Codex
parent
a524f80b1f
commit
092bdd6d57
@@ -186,6 +186,11 @@ class EngineClient(ABC):
|
||||
"""Sleep the engine."""
|
||||
...
|
||||
|
||||
@abstractmethod
|
||||
async def release_kv_cache_memory(self) -> None:
|
||||
"""Discard KV cache physical GPU memory. Requires a completed pause."""
|
||||
...
|
||||
|
||||
@abstractmethod
|
||||
async def wake_up(self, tags: list[str] | None = None) -> None:
|
||||
"""Wake up the engine."""
|
||||
|
||||
Reference in New Issue
Block a user