[Model] Decoder-side SWA bounded replay for DeepSeek-V4.1 (#58132)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
9af952c555
commit
4b2e1cfa7b
@@ -242,7 +242,9 @@ class CacheConfig:
|
||||
swa_bounded_replay: bool = True
|
||||
"""Keep the sliding-window KV of models that support it (DeepSeek-V4.1)
|
||||
out of prefix caching and rebuild it after a prefix hit by recomputing the
|
||||
hit's last window. Requires model runner V2."""
|
||||
hit's last window. The layers past the last KV-source layer then also run
|
||||
eager prefill steps on each request's trailing window only. Requires model
|
||||
runner V2."""
|
||||
|
||||
kv_cache_memory_bytes: int | None = None
|
||||
"""Size of KV Cache per GPU in bytes. By default, this is set to None
|
||||
|
||||
Reference in New Issue
Block a user