31 Commits
Author SHA1 Message Date
liranschourandKevin H. Luu c5f8adc0b1 [KV Offload] Expose OffloadingManager.lock()/unlock() and service the tiering control plane off the scheduler thread (#58168)
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 08:50:56 +03:00
775fefa804 [Metrics][KV Offload] Add vllm:kv_offload_config_info - generic info metrics for offloading the offloading connector (#56867)
Signed-off-by: Amir Friedman <Amir.Friedman1@ibm.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-10-14 12:34:37 +00:00
e37e51dd24 [Metrics] Expose cached prompt tokens by cache tier (#56318)
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cameron@semianalysis.com>
Co-authored-by: Cam Quilici <cameron@semianalysis.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
2026-10-06 14:52:05 -04:00
bnellnmandItay Etelis 23e26e0588 [KV Offloading] Back-pressure detection and remediation (#50045)
Signed-off-by: Bill Nell <bnell@redhat.com>
Signed-off-by: bnellnm <49004751+bnellnm@users.noreply.github.com>
Co-authored-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
2026-09-18 14:47:48 -04:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
MINJUN GILandOr Ozeri f12fe10c5c [Bugfix][KV Offload] Track cache recency once per request (#51787)
Signed-off-by: mindungil <alswnsrlf12@naver.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-09-16 10:47:26 +03:00
Ronen Schaffer 138d137b5b [Refactor][kv_offload]: rename block→chunk (#52615)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-09-09 17:45:49 +03:00
Ilia Yastrebov 41848caa63 [NIXL] Use int32 array for indices to avoid intermediate conversion (#51952)
Signed-off-by: Ilia Yastrebov <iyastrebov@nvidia.com>
2026-09-02 16:30:16 +02:00
AlexHuangandOr Ozeri c35551f892 [Bugfix][KV Offload] Isolate tiering shutdown failures (#52290)
Signed-off-by: Alex <jihui.huang@daocloud.io>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-09-01 12:24:54 +03:00
Almog Tavor bc39ded3c4 [Bugfix][KV Offload] Defer request-level cascade of in-flight primary keys (#53329)
Signed-off-by: almogtavor <almogtavor@gmail.com>
2026-08-25 23:20:03 +00:00
Varun Sundar RabindranathandVarun Sundar Rabindranath f067737b2a [Bugfix][TieredOffloading] : Return HIT_PENDING when KV promotion is triggered (#51840)
Signed-off-by: <>
Co-authored-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com>
2026-08-12 05:07:16 +03:00
Srinivas Krovvidiandsrinivas_oo7 a123159f7a Add tiering offloading metrics (#48798)
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
2026-08-10 05:23:14 +03:00
41a7e7da0c [KV Offload] Support partial secondary-tier load results (#50321)
Signed-off-by: Moein Khazraee <moein@nvidia.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-08-04 20:48:58 -07:00
Philip Pesic dc1be79031 Add CachePolicyFactory for pluggable/external eviction policies (#49114)
Signed-off-by: Philip Pesic <philippesic06@gmail.com>
2026-07-29 07:34:55 +03:00
Ronen Schaffer 77cba0259f [KV Offloading] Per-request tier filtering with TierFilter/TierMatcher (#48123)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-07-27 13:29:57 +03:00
Srinivas Krovvidiandsrinivas_oo7 ce65385618 [KV Offload] Split tiering_lookup_delay into sync/async histograms (#47679)
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
2026-07-16 20:03:45 +03:00
c46ced1ee3 [kv_offload] Establish tier-owned KV event handling (#46544)
Signed-off-by: Change72 <changg@nvidia.com>
Signed-off-by: Chang Guo <changg@nvidia.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-07 16:55:20 +03:00
Ronen SchafferandOr Ozeri 48fcfc926c [KV Offload] Add ParentManager ABC for secondary tier callbacks (#47274)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-07 13:51:18 +03:00
Ronen SchafferandOr Ozeri 0fc2512094 [KV Offload] Pass ScheduleEndContext to on_schedule_end hook (#46450)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-06-30 17:07:12 +03:00
Ronen SchafferandOr Ozeri bb61177e49 [KV Offloading] Replace bool|None lookup return with LookupResult enum (#46363)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-06-24 18:06:08 +03:00
Rui YinandOr Ozeri cf9fd6457e Fix KV offload request-finished lifecycle contract (#46284)
Signed-off-by: test test <2260891073@qq.com>
Signed-off-by: Rui Yin <2260891073@qq.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-06-24 15:42:40 +03:00
Srinivas Krovvidiandsrinivas_oo7 091bc1026e [KV Offloading] Add tiering metric plumbing (#45959)
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
2026-06-23 15:10:36 +03:00
6d8fff5698 [KV Connector][Offloading] Avoid blocking the engine to flush offloads on idle (#45595)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Signed-off-by: Itay Etelis <Itay.etelis@gmail.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Itay Etelis <Itay.etelis@gmail.com>
2026-06-17 11:35:07 +03:00
Ronen Schafferandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 7e612a0f06 [KV Offloading] Implement reset_cache for TieringOffloadingManager (#44541)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-15 18:42:53 +00:00
Ronen Schaffer 2a2b5ca791 [KV Offload] Add on_schedule_end() hook to separate step lifecycle from event draining (#44206)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-06-02 13:42:52 +03:00
Ronen Schaffer 93da882e73 [kv_offload] Add @override decorators to subclass method implementations (#44177)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-06-02 08:07:47 +00:00
Ronen SchafferOr Ozerimergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
a3ed5ab10c [KV Offload] Add per-request offloading policy via on_new_request lifecycle hook (#43205)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-28 20:45:18 +00:00
Ronen Schaffer 4bfa0f2b14 [KV Offload] Rename SecondaryTierManager.get_finished() to get_finished_jobs() (#43870)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-05-28 16:00:18 +00:00
Ronen Schaffer 84747489de Tier offload followup (#42529)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-05-18 19:41:58 +00:00
Ronen Schaffer 11f6b545d4 [kv_offload] Add multi-tier KV cache offloading framework (#40020)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-05-13 17:21:43 +03:00