[Fast Start] Charge daemon-held weights against gpu_memory_utilization (#57298)

Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Xun Sun <UNIDY2002@outlook.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Xun Sun <UNIDY2002@outlook.com>
This commit is contained in:
siyu
2026-09-29 17:58:35 +00:00
committed by GitHub
co-authored by Isotr0py Xun Sun
parent cb4f016c5d
commit f5ed3d9c03
7 changed files with 187 additions and 14 deletions
@@ -28,6 +28,11 @@ class BaseModelLoader(ABC):
def __init__(self, load_config: LoadConfig):
self.load_config = load_config
def get_external_weight_memory(self, vllm_config: VllmConfig) -> int:
"""Get weights memory from external process;
0 when the weights are not external."""
return 0
@abstractmethod
def download_model(self, model_config: ModelConfig) -> None:
"""Download a model so that it can be immediately loaded."""