iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
14,386 Commits 3 Branches 0 Tags
fa6a6be51978bd4b49ba0da17039e60f96dc5b13
Commit Graph
77 Commits
Author SHA1 Message Date
Mark McLoughlinandHarry Mellor 5cc7c4452e [Metrics] Add Prometheus counters for Model FLOPs Utilization (MFU) (#30950) 
Export the existing Model FLOPs Utilization (MFU) metrics via Prometheus.

`--enable-mfu-metrics` is required for these to be exposed.

Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2026-02-23 15:01:07 +00:00
JaewonandLu Fang 4453ba8d9e [Core] Profiler improvements and lazy initialization (#33198) 
Signed-off-by: Jaewon Lee <jaewon@meta.com>
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
 2026-02-12 16:16:38 -08:00
zhanqiuhu 4403e3ed4c [Metrics] Add labeled prompt token metrics for P/D disaggregation (#33290) 
Add labeled Prometheus metrics to distinguish where prompt tokens come
from in P/D disaggregated deployments.

In P/D disaggregation, decode instances receive KV cache from prefill instances.
Currently, decode reports inflated prompt throughput because it counts all
prompt tokens as "computed", even though most were transferred.

This PR adds labeled metrics so users can understand actual compute work vs
transferred work:

vllm:prompt_tokens_by_source_total{source="local_compute"}        # Tokens prefilled locally
vllm:prompt_tokens_by_source_total{source="external_kv_transfer"} # Tokens received via KV transfer  
vllm:prompt_tokens_by_source_total{source="local_cache_hit"}      # Tokens from local prefix cache
vllm:prompt_tokens_cached_total                                    # Total cached (local + external, -1 when all 

Signed-off-by: Zhanqiu Hu <zh338@cornell.edu>
 2026-02-04 07:46:48 +00:00
杨朱 · KikiandClaude Haiku 4.5 bb9172030e [Metrics] Complete removal of deprecated vllm:time_per_output_token_seconds metric (#32661) 
This PR completes the removal of the deprecated vllm:time_per_output_token_seconds
metric that was deprecated in v0.11, hidden in v0.12, scheduled for removal in v0.13,
but delayed until v0.15.

Signed-off-by: carlory <baofa.fan@daocloud.io>
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
 2026-01-20 12:28:41 +00:00
SungMinChoandMark McLoughlin a0b782f9cc [Metrics] Model FLOPs Utilization estimation (#30738) 
Signed-off-by: SungMinCho <tjdals4565@gmail.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
 2025-12-18 01:40:51 +00:00
Victor Ziliang Peng f1599ca55d feat(metrics): Add prefill KV compute metric excluding cached tokens (#30189) 
Signed-off-by: Ziliang Peng <ziliang@character.ai>
 2025-12-09 00:08:48 +00:00
Nick Hill d726a7b0ed [BugFix] Unblock use of LoRA with data parallel mode (#30220) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-12-08 12:21:05 +08:00
Yong Hoon Shin 69520bc695 Add logging for cudagraph related info (#29825) 
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
 2025-12-03 01:01:48 -08:00
shivampr cabc77cc86 [Core][Observability] Add KV cache residency metrics (#27793) 
Introduces three new Prometheus histograms for fine-grained observability of KV cache residency behavior:

vllm:kv_block_lifetime_seconds — total lifetime from allocation to free
vllm:kv_block_idle_before_evict_seconds — idle duration before eviction
vllm:kv_block_reuse_gap_seconds — time between consecutive reuses of the same block

These metrics help operators analyze KV cache efficiency, reuse patterns, and eviction timing beyond simple utilization rates.

Implementation uses monotonic timestamps for accuracy, 1% sampling for minimal overhead (~48 bytes/block), and is fully thread-safe with zero runtime cost when disabled.

Two new runtime flags are introduced:

--kv-cache-metrics – enable KV cache residency metrics
--kv-cache-metrics-sample – control sampling ratio (default: 0.01)

Signed-off-by: Shivam <shivamprasad91@gmail.com>
 2025-12-01 18:27:53 +00:00
Cyrus Leung 9e6bcda3ac [mypy] Enable type checking for more directories (#29674) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-28 08:39:27 -08:00
Mark McLoughlin 9cf4edae6e [Metrics] Scheduled removal of deprecated metrics (#29330) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-11-25 11:15:13 +08:00
wangxiyuan 4050bae417 [Doc] Update plugin doc (#28532) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2025-11-21 14:57:26 +00:00
Jae-Won Chung d4acf518d0 [Metrics] Fix KV cache usage percent metric multiproc (#28792) 
The `vllm:kv_cache_usage_perc` Gauge metric is missing `multiprocess_mode="mostrecent"` and ends up returning

```
vllm:kv_cache_usage_perc{engine="0",model_name="Qwen/Qwen3-VL-8B-Instruct",pid="277"} 0.0
vllm:kv_cache_usage_perc{engine="0",model_name="Qwen/Qwen3-VL-8B-Instruct",pid="275"} 0.0
vllm:kv_cache_usage_perc{engine="0",model_name="Qwen/Qwen3-VL-8B-Instruct",pid="273"} 0.6530455880475035
...
```

The deprecated `vllm:gpu_cache_usage_perc` Gauge metric has `multiprocess_mode="mostrecent"`.

Signed-off-by: Jae-Won Chung <jwnchung@umich.edu>
 2025-11-17 09:54:15 +00:00
lyn610 ecf8230d4d [Metrics] Log number of preempted requests (#28522) 
Add tracking and periodic logging for the number of preempted requests in the
metrics logger. This helps monitor system behavior under load.

Signed-off-by: Yining Liu <610lyn@gmail.com>
 2025-11-14 09:47:45 +00:00
Mark McLoughlin 6f7de33bed [Metrics] Refactor LoRA state tracking (#26801) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-11-10 16:34:36 +08:00
Snehlata e15601789b [Feature]: Add corrupted request metric to V1 metrics system. (#27306) 
Signed-off-by: atalhens <sneh.lata@nutanix.com>
 2025-11-05 13:45:29 -08:00
Mark McLoughlin 380ba6816d [Metrics] Enable sleep state metric outside of dev mode (#27867) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-11-03 20:35:36 -08:00
Sumanth R Hegde 4917002523 [Fix] Skip record_sleep_state logic in PrometheusStatsLogger if not in dev mode (#27789) 
Signed-off-by: SumanthRH <sumanthrh99@gmail.com>
 2025-10-30 19:26:27 +00:00
Nicolò LucchesiandMark McLoughlin accb8fab07 [KVConnector] Add metrics to Prometheus-Grafana dashboard (#26811) 
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
 2025-10-29 18:44:49 +00:00
Braulio Dumba 1da3309ace [Core] Exposing engine sleep & wake_up state as prometheus metrics (#24176) 
Signed-off-by: Braulio Dumba <Braulio.Dumba@ibm.com>
 2025-10-29 09:32:01 -07:00
Wentao Ye 52efc34ebf [Log] Optimize Startup Log (#26740) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2025-10-24 19:27:04 -04:00
Tova Movshovitz 88afa11010 [Metrics] [KVConnector] Add connector prefix cache hit rate stats (#26245) 
Signed-off-by: tovam <tovam@pliops.com>
 2025-10-23 12:21:08 +02:00
Tova Movshovitz 83e760c57d [V1][Metrics][Plugin] Add plugin support for custom StatLoggerBase implementations (#22456) 
Signed-off-by: tovam <tovam@pliops.com>
 2025-10-18 15:12:46 -07:00
Lucia Fang 8317f72354 [Misc][DP] support customized aggregated logger for dp (#24354) 
Signed-off-by: Lu Fang <fanglu@fb.com>
 2025-10-13 17:45:59 -07:00
Harry Mellor 8fcaaf6a16 Update Optional[x] -> x | None and Union[x, y] to x | y (#26633) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-12 09:51:31 -07:00
Mark McLoughlin e519281920 [Metrics] Add test for multi-modal cache stats logging (#26588) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-10-10 16:00:50 +00:00
Cyrus Leung ad430a67ca [Metrics] Log multi-modal cache stats and fix reset (#26285) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-10 01:45:55 -07:00
Harry Mellor d6953beb91 Convert formatting to use ruff instead of yapf + isort (#26247) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-05 07:06:22 -07:00
Nicolò Lucchesi 48f309029a [NIXL][Misc] Expose metrics from NIXL for logging to CLI (#25388) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-10-03 10:47:59 +00:00
Zhuohan LiandYe Qi 8c435c9bce [Core] Enable command line logging for LLMEngine (#25610) 
Co-authored-by: Ye (Charlotte) Qi <yeq@meta.com>
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
 2025-09-25 15:31:17 -07:00
baxingpiaochong d06b5a95cb [V1][Metrics] Add per-request TPOT histogram (#24015) 
Signed-off-by: baxingpiaochong <771405853@qq.com>
 2025-09-23 18:19:04 -06:00
Nicolò Lucchesi a3d087adec [P/D][Nixl] Introduce KVTransferMetrics and aggregation strategy (#22188) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-09-19 11:09:14 +00:00
Mark McLoughlin 2942970d44 [Metrics] Hide deprecated metrics with gpu_ prefix (#24245) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-09-15 20:15:57 -06:00
wuhangandRobert Shaw 90f3f7d73e [Spec Decoding]Support Spec Decoding Metrics in DP Mode (#24049) 
Signed-off-by: wuhang <wuhang6@huawei.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
 2025-09-14 21:11:09 +00:00
ChaunceyandNick Hill e680723eba [Bugfix] Disable the statslogger if the api_server_count is greater than 1 (#22227) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-09-08 15:28:03 -07:00
Seiji EicherandNick Hill 60b755cbcb [Misc] Have AsyncLLM custom_stat_loggers extend default logger list (#20952) 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
Signed-off-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-09-04 14:25:30 -07:00
Mark McLoughlin 2417798471 [Metrics] Deprecate TPOT in favor of ITL (#24110) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-09-02 18:10:10 +00:00
ZiTian.Zhao a8936e5193 Refactor: Remove numpy dependency from LoggingStatLogger (#20529) 
Signed-off-by: zitian.zhao <zitian.zhao@tencentmusic.com>
 2025-07-27 04:06:21 +00:00
Robert ShawandRobert Shaw 29d1ffc5b4 [DP] Fix Prometheus Logging (#21257) 
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
 2025-07-21 09:11:35 -07:00
Maximilien de Bayserand22quinn 799397ee4f Support embedding models in V1 (#16188) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Signed-off-by: Max de Bayser <maxdebayser@gmail.com>
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-06-18 21:36:33 -07:00
Saheli Bhattacharjee d1e34cc9ac [V1][Metrics] Deprecate metrics with gpu_ prefix for non GPU specific metrics. (#18354) 
Signed-off-by: Saheli Bhattacharjee <saheli@krai.ai>
 2025-06-14 11:07:36 +08:00
Simon Mo 02f0c7b220 [Misc] Add SPDX-FileCopyrightText (#19100) 
Signed-off-by: simon-mo <simon.mo@hey.com>
 2025-06-03 11:20:17 -07:00
Nick Hill 2dbe8c0774 [Perf] API-server scaleout with many-to-many server-engine comms (#17546) 2025-05-30 08:17:00 -07:00
Mark McLoughlin 06a0338015 [V1][Metrics] Add API for accessing in-memory Prometheus metrics (#17010) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-05-27 09:37:06 +00:00
Seiji Eicher 541817670c [Misc] Add Ray Prometheus logger to V1 (#17925) 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
 2025-05-16 01:02:42 -07:00
Chen Zhang 302f3aca7e [v1][KVCacheManager] Change prefix caching metric from counting blocks to counting tokens (#18003) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-05-12 13:46:12 -07:00
Mark McLoughlin 7042cc96b0 [V1][Spec Decoding] Log accumulated metrics after system goes idle (#17913) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-05-09 18:23:07 -07:00
rongfu.leng d803786731 [V1][Bugfix]: vllm v1 verison metric num_gpu_blocks is None (#15755) 
Signed-off-by: rongfu.leng <rongfu.leng@daocloud.io>
 2025-04-30 18:20:39 +08:00
Cyrus Leung 4213475ec7 [Metrics] Fix minor inconsistencies in bucket progression (#17262) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-04-27 16:19:39 +00:00
Flex Wang 18445edd0f [Misc] Change buckets of histogram_iteration_tokens to [1, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8096] to represent number of tokens (#17033) 
Signed-off-by: sfc-gh-zhwang <flex.wang@snowflake.com>
 2025-04-27 12:30:53 +00:00