[Feature][Perf] Support Selective CPU Weight Offloading (#34535)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
This commit is contained in:
@@ -101,6 +101,17 @@ class CacheConfig:
|
||||
Note that this requires fast CPU-GPU interconnect, as part of the model is
|
||||
loaded from CPU memory to GPU memory on the fly in each model forward pass.
|
||||
"""
|
||||
cpu_offload_params: set[str] = Field(default_factory=set)
|
||||
""" The set of parameter name segments to target for CPU offloading.
|
||||
Unmatched parameters are not offloaded. If this set is empty, parameters
|
||||
are offloaded non-selectively until the memory limit defined by
|
||||
`cpu_offload_gb` is reached.
|
||||
Examples:
|
||||
- For parameter name "mlp.experts.w2_weight":
|
||||
- "experts" or "experts.w2_weight" will match.
|
||||
- "expert" or "w2" will NOT match (must be exact segments).
|
||||
This allows distinguishing parameters like "w2_weight" and "w2_weight_scale".
|
||||
"""
|
||||
calculate_kv_scales: bool = False
|
||||
"""This enables dynamic calculation of `k_scale` and `v_scale` when
|
||||
kv_cache_dtype is fp8. If `False`, the scales will be loaded from the model
|
||||
|
||||
Reference in New Issue
Block a user