[Metrics] Add group-aware KV cache capacity to vllm:cache_config_info (#42206)
The startup log already reports the correct group-aware KV cache capacity for hybrid models, but Prometheus did not expose matching info in 'vllm:cache_config_info`. This PR adds kv_cache_size_tokens and kv_cache_max_concurrency. Signed-off-by: Ethan Feng <ethan.fengch@gmail.com>
This commit is contained in:
@@ -146,6 +146,14 @@ class CacheConfig:
|
||||
num_cpu_blocks: int | None = field(default=None, init=False)
|
||||
"""The number of blocks to allocate for CPU memory."""
|
||||
|
||||
# Set after KV cache initialization.
|
||||
kv_cache_size_tokens: int | None = field(default=None, init=False)
|
||||
"""Per-DP-engine KV cache capacity in tokens (group-aware). Uses
|
||||
group-aware capacity since num_gpu_blocks * block_size can be wrong
|
||||
for hybrid models where requests occupy multiple KV cache groups."""
|
||||
kv_cache_max_concurrency: float | None = field(default=None, init=False)
|
||||
"""Per-DP-engine maximum concurrency at max_model_len tokens."""
|
||||
|
||||
kv_sharing_fast_prefill: bool = False
|
||||
"""This feature is work in progress and no prefill optimization takes place
|
||||
with this flag enabled currently.
|
||||
@@ -204,6 +212,8 @@ class CacheConfig:
|
||||
# Post-init/derived counters
|
||||
"num_gpu_blocks",
|
||||
"num_cpu_blocks",
|
||||
"kv_cache_size_tokens",
|
||||
"kv_cache_max_concurrency",
|
||||
# WIP feature toggle not impacting compiled graph shape
|
||||
"kv_sharing_fast_prefill",
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user