[Core] MRV2 support for fast-prefill (#56145)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
This commit is contained in:
@@ -234,7 +234,6 @@ class CacheConfig:
|
||||
some layers can skip tokens corresponding to prefill. This flag enables
|
||||
attention metadata for eligible layers to be overridden with metadata
|
||||
necessary for implementing this optimization in some models (e.g. Gemma3n)
|
||||
NOTE: KV cache sharing is not supported for MRv2 (v2 model runner).
|
||||
"""
|
||||
|
||||
kv_cache_memory_bytes: int | None = None
|
||||
|
||||
Reference in New Issue
Block a user