[Docs] kv_sharing_fast_prefill correction (#47044)

Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
This commit is contained in:
Nicolò Lucchesi
2026-07-08 01:17:46 -07:00
committed by GitHub
parent 51e5372f3d
commit d79855eaac
+2 -4
View File
@@ -161,13 +161,11 @@ class CacheConfig:
"""Per-DP-engine maximum concurrency at max_model_len tokens.""" """Per-DP-engine maximum concurrency at max_model_len tokens."""
kv_sharing_fast_prefill: bool = False kv_sharing_fast_prefill: bool = False
"""This feature is work in progress and no prefill optimization takes place """In some KV sharing setups, e.g. YOCO (https://arxiv.org/abs/2405.05254),
with this flag enabled currently.
In some KV sharing setups, e.g. YOCO (https://arxiv.org/abs/2405.05254),
some layers can skip tokens corresponding to prefill. This flag enables some layers can skip tokens corresponding to prefill. This flag enables
attention metadata for eligible layers to be overridden with metadata attention metadata for eligible layers to be overridden with metadata
necessary for implementing this optimization in some models (e.g. Gemma3n) necessary for implementing this optimization in some models (e.g. Gemma3n)
NOTE: KV cache sharing is not supported for MRv2 (v2 model runner).
""" """
kv_cache_memory_bytes: int | None = None kv_cache_memory_bytes: int | None = None