[HiSparse] Log steady-state max concurrency and expose host-tier utilization gauges (#58949)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
Matthew Bonanni
parent
8542c3eff5
commit
2e3154aa18
@@ -37,6 +37,14 @@ memory consumption is therefore topology- and implementation-dependent. The
|
||||
realized capacity may be slightly smaller because the budget is rounded down
|
||||
to complete host blocks.
|
||||
|
||||
Startup logs report two concurrency bounds at `max_model_len`. The generic
|
||||
`Maximum concurrency` line charges each request its full admission footprint,
|
||||
including the in-flight window of every resident group. The `HiSparse
|
||||
steady-state maximum concurrency` line charges running requests that read from
|
||||
host only their active tail pages, plus one request being admitted at its full
|
||||
footprint. Host-pool metrics are listed in
|
||||
[Metrics](../usage/metrics.md#hisparse-kv-connector-metrics).
|
||||
|
||||
## Ownership
|
||||
|
||||
| Thing | Owner | What “owner” means |
|
||||
|
||||
@@ -23,6 +23,10 @@ METRIC_SOURCE_FILES = [
|
||||
"path": "vllm/distributed/kv_transfer/kv_connector/v1/nixl/stats.py",
|
||||
"key": "metrics-nixl",
|
||||
},
|
||||
{
|
||||
"path": "vllm/distributed/kv_transfer/kv_connector/v1/hisparse/stats.py",
|
||||
"key": "metrics-hisparse",
|
||||
},
|
||||
{
|
||||
"path": "vllm/v1/simple_kv_offload/metrics.py",
|
||||
"key": "metrics-simple-kv-offload",
|
||||
|
||||
@@ -45,6 +45,15 @@ The following metrics are exposed:
|
||||
|
||||
--8<-- "gen:metrics-nixl"
|
||||
|
||||
## HiSparse KV Connector Metrics
|
||||
|
||||
These metrics are exposed when the `HiSparseConnector` KV connector is
|
||||
configured. The host-pool gauges are reported by the scheduler once per engine
|
||||
step, for the host pool shared by all tensor-parallel ranks; evictable cached
|
||||
blocks count as free.
|
||||
|
||||
--8<-- "gen:metrics-hisparse"
|
||||
|
||||
## Simple CPU Offload Connector Metrics
|
||||
|
||||
These metrics are exposed when the `SimpleCPUOffloadConnector` KV connector
|
||||
|
||||
Reference in New Issue
Block a user