iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[Model Runner v2] Support reload weights (sleep mode) (#42673)

Browse Source 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This commit is contained in:
Wentao Ye
2026-05-15 16:41:23 +00:00
committed by GitHub
co-authored by gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
parent 0162596603
commit 6147c70224
1 changed files with 8 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
vllm/v1/worker/gpu/model_runner.py
+8
View File
@@ -329,6 +329,14 @@ class GPUModelRunner(LoRAModelRunnerMixin):
def get_model(self) -> nn.Module:
return self.model
def reload_weights(self, *args, **kwargs) -> None:
# TODO(Wentao): Use full version instead of import when fully migrated to v2
from vllm.v1.worker.gpu_model_runner import GPUModelRunner as GPUModelRunnerV1
GPUModelRunnerV1.reload_weights(self, *args, **kwargs) # type: ignore[arg-type]
self.reset_encoder_cache()
self.reset_mm_cache()
@functools.cached_property
def main_stream(self) -> torch.cuda.Stream:
# Cache the default CUDA stream to avoid lookup overhead.