[Model] Decoder-side SWA bounded replay for DeepSeek-V4.1 (#58132)

Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Yifan Qiao
2026-09-29 08:25:46 +00:00
committed by GitHub
co-authored by Claude Fable 5.1
parent 9af952c555
commit 4b2e1cfa7b
9 changed files with 1000 additions and 59 deletions
+3 -1
View File
@@ -242,7 +242,9 @@ class CacheConfig:
swa_bounded_replay: bool = True
"""Keep the sliding-window KV of models that support it (DeepSeek-V4.1)
out of prefix caching and rebuild it after a prefix hit by recomputing the
hit's last window. Requires model runner V2."""
hit's last window. The layers past the last KV-source layer then also run
eager prefill steps on each request's trailing window only. Requires model
runner V2."""
kv_cache_memory_bytes: int | None = None
"""Size of KV Cache per GPU in bytes. By default, this is set to None