[Performance] Auto-enable prefetch on NFS with RAM guard (#37673)
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
This commit is contained in:
+6
-2
@@ -54,10 +54,14 @@ class LoadConfig:
|
||||
download_dir: str | None = None
|
||||
"""Directory to download and load the weights, default to the default
|
||||
cache directory of Hugging Face."""
|
||||
safetensors_load_strategy: str = "lazy"
|
||||
safetensors_load_strategy: str | None = None
|
||||
"""Specifies the loading strategy for safetensors weights.
|
||||
- "lazy" (default): Weights are memory-mapped from the file. This enables
|
||||
- None (default): Uses memory-mapped (lazy) loading. When an NFS
|
||||
filesystem is detected and the total checkpoint size fits within 90%%
|
||||
of available RAM, prefetching is enabled automatically.
|
||||
- "lazy": Weights are memory-mapped from the file. This enables
|
||||
on-demand loading and is highly efficient for models on local storage.
|
||||
Unlike the default (None), auto-prefetch on NFS is not performed.
|
||||
- "eager": The entire file is read into CPU memory upfront before loading.
|
||||
This is recommended for models on network filesystems (e.g., Lustre, NFS)
|
||||
as it avoids inefficient random reads, significantly speeding up model
|
||||
|
||||
Reference in New Issue
Block a user