[Core] MRV2 support for fast-prefill (#56145)

Signed-off-by: Nick Hill <nickhill123@gmail.com>
This commit is contained in:
Nick Hill
2026-09-10 10:36:04 -07:00
committed by GitHub
parent 7470082f57
commit e6cb56337b
12 changed files with 233 additions and 12 deletions
-1
View File
@@ -234,7 +234,6 @@ class CacheConfig:
some layers can skip tokens corresponding to prefill. This flag enables
attention metadata for eligible layers to be overridden with metadata
necessary for implementing this optimization in some models (e.g. Gemma3n)
NOTE: KV cache sharing is not supported for MRv2 (v2 model runner).
"""
kv_cache_memory_bytes: int | None = None