[Misc][Hybrid allocator + kv connector] Optionally enable hybrid allocator + KV cache connector (#29805)

Signed-off-by: NickLucche <nlucches@redhat.com>
This commit is contained in:
Nicolò Lucchesi
2025-12-15 11:17:58 +00:00
committed by GitHub
parent e4806d973a
commit 185c22bf2f
3 changed files with 64 additions and 38 deletions
+3 -1
View File
@@ -122,10 +122,12 @@ class SchedulerConfig:
the default scheduler. Can be a class directly or the path to a class of
form "mod.custom_class"."""
disable_hybrid_kv_cache_manager: bool = False
disable_hybrid_kv_cache_manager: bool | None = None
"""If set to True, KV cache manager will allocate the same size of KV cache
for all attention layers even if there are multiple type of attention layers
like full attention and sliding window attention.
If set to None, the default value will be determined based on the environment
and starting configuration.
"""
async_scheduling: bool = False