41 Commits
Author SHA1 Message Date
liranschourandKevin H. Luu c5f8adc0b1 [KV Offload] Expose OffloadingManager.lock()/unlock() and service the tiering control plane off the scheduler thread (#58168)
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 08:50:56 +03:00
775fefa804 [Metrics][KV Offload] Add vllm:kv_offload_config_info - generic info metrics for offloading the offloading connector (#56867)
Signed-off-by: Amir Friedman <Amir.Friedman1@ibm.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-10-14 12:34:37 +00:00
e37e51dd24 [Metrics] Expose cached prompt tokens by cache tier (#56318)
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cameron@semianalysis.com>
Co-authored-by: Cam Quilici <cameron@semianalysis.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
2026-10-06 14:52:05 -04:00
Karen Chung 986e217158 [Feature] Add first-class KV hints request envelope for programmatic KV management (#53423)
Signed-off-by: Karen Chung <karenc@nvidia.com>
2026-09-21 09:56:11 -07:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
MINJUN GILandOr Ozeri f12fe10c5c [Bugfix][KV Offload] Track cache recency once per request (#51787)
Signed-off-by: mindungil <alswnsrlf12@naver.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-09-16 10:47:26 +03:00
Ronen Schaffer 138d137b5b [Refactor][kv_offload]: rename block→chunk (#52615)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-09-09 17:45:49 +03:00
Ilia Yastrebov 41848caa63 [NIXL] Use int32 array for indices to avoid intermediate conversion (#51952)
Signed-off-by: Ilia Yastrebov <iyastrebov@nvidia.com>
2026-09-02 16:30:16 +02:00
Moein Khazraee 9acbc5360a [KV Offload] Preserve KV event metadata until final residency removal (#52068)
Signed-off-by: Moein Khazraee <moein@nvidia.com>
2026-08-31 13:28:30 +03:00
f9d666f917 [KV Offload] Forward ownership in KV cache events (#52067)
Signed-off-by: Moein Khazraee <moein@nvidia.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-08-31 10:28:41 +03:00
Lucas Wilkinson 8bdc70ec7b [6/N][KV-Cache Layout Refactor] Standardize KV cache layout (#51718)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
2026-08-21 17:47:49 -07:00
2c4d348848 [KV Connector] Add per-layer canonical KV page mappings for parallelism-agnostic offload (#48408)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Itay Etelis <Itay.etelis@gmail.com>
2026-07-31 14:42:54 +03:00
Philip Pesic dc1be79031 Add CachePolicyFactory for pluggable/external eviction policies (#49114)
Signed-off-by: Philip Pesic <philippesic06@gmail.com>
2026-07-29 07:34:55 +03:00
Ronen Schaffer 77cba0259f [KV Offloading] Per-request tier filtering with TierFilter/TierMatcher (#48123)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-07-27 13:29:57 +03:00
da3a252fd1 [KVOffload][P2P] Generic P2P secondary tier: peer lookup and serving via ParentManager (#48021)
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: liranschour <liranschour@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-26 11:45:47 +03:00
Chang GuoandOpenAI Codex 7eca0e1a64 [KV Offload] Deduplicate replicated MLA KV in the shared CPU region (#48906)
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-07-26 08:22:30 +03:00
426d48bfa1 [KV Offload] Add optional tier locality to FS/OBJ KV events (#48281)
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 10:31:52 +03:00
a9531edfa6 [KV Offload] Define clean backend configuration boundary (#48150)
Signed-off-by: Change72 <changg@nvidia.com>
Signed-off-by: Chang Guo <cguo51@asu.edu>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-07-16 13:27:05 +03:00
c46ced1ee3 [kv_offload] Establish tier-owned KV event handling (#46544)
Signed-off-by: Change72 <changg@nvidia.com>
Signed-off-by: Chang Guo <changg@nvidia.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-07 16:55:20 +03:00
Ronen SchafferandOr Ozeri 0fc2512094 [KV Offload] Pass ScheduleEndContext to on_schedule_end hook (#46450)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-06-30 17:07:12 +03:00
Ronen SchafferandOr Ozeri bb61177e49 [KV Offloading] Replace bool|None lookup return with LookupResult enum (#46363)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-06-24 18:06:08 +03:00
Rui YinandOr Ozeri cf9fd6457e Fix KV offload request-finished lifecycle contract (#46284)
Signed-off-by: test test <2260891073@qq.com>
Signed-off-by: Rui Yin <2260891073@qq.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-06-24 15:42:40 +03:00
Martin Hickey f237e16b41 [KV Offload] Replace OffloadingHandler with OffloadingWorker (#45053)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
2026-06-24 14:44:24 +03:00
Change72andClaude a9f7b2d41c [feature][kv_offload] Self-describing KV events for OffloadingConnector (#43468)
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-06-22 07:27:46 +00:00
Srinivas Krovvidiandsrinivas_oo7 c441ad1c07 [KV Offloading] Add labeled metrics support (#45957)
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
2026-06-21 18:04:01 +00:00
Ronen Schaffer f428718ffe [Fix][KV offload] Defer on_request_finished until in-flight transfers drain (#45823)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-06-18 07:05:46 +03:00
6d8fff5698 [KV Connector][Offloading] Avoid blocking the engine to flush offloads on idle (#45595)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Signed-off-by: Itay Etelis <Itay.etelis@gmail.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Itay Etelis <Itay.etelis@gmail.com>
2026-06-17 11:35:07 +03:00
Srinivas Krovvidisrinivas_oo7Srinivasoo7mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Or Ozeri
9dfc313bdc Feature/offloading manager stats (#35669)
Signed-off-by: Sriusa4414@gmail.com
Signed-off-by: srinivas_oo7 <Sriusa4414@gmail.com>
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Signed-off-by: Srinivasoo7 <158864704+Srinivasoo7@users.noreply.github.com>
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: Srinivasoo7 <158864704+Srinivasoo7@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-06-10 12:44:55 +00:00
Ronen Schaffer 2a2b5ca791 [KV Offload] Add on_schedule_end() hook to separate step lifecycle from event draining (#44206)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-06-02 13:42:52 +03:00
Ronen Schaffer 93da882e73 [kv_offload] Add @override decorators to subclass method implementations (#44177)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-06-02 08:07:47 +00:00
Itay EtelisandItay Etelis d63108fb18 [kv_offload] Skip decode-phase blocks in CPU offload (#43797)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
2026-05-29 06:39:43 +03:00
Ronen SchafferOr Ozerimergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
a3ed5ab10c [KV Offload] Add per-request offloading policy via on_new_request lifecycle hook (#43205)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-28 20:45:18 +00:00
Or Ozeri 357fddf614 [kv_offload]: Add DSv4 support (#43142)
Signed-off-by: Or Ozeri <oro@il.ibm.com>
2026-05-24 11:10:12 +03:00
Martin Hickeygemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Or Ozeri
5bd8c71e79 [kv_offload] Implement reset_cache() for the offloading connector (#41956)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
2026-05-14 16:00:10 +03:00
Ronen Schaffer 11f6b545d4 [kv_offload] Add multi-tier KV cache offloading framework (#40020)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-05-13 17:21:43 +03:00
Ronen Schaffer 79fd1bc7ed [kv_offload] Add req_id to ReqContext for per-request tracking (#42507)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-05-13 11:11:10 +00:00
Ronen Schaffer e175192d33 [KV Offload] Pass ReqContext to touch(), complete_load(), and complete_store() (#41366)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-05-10 15:09:25 +03:00
Itay EtelisandItay Etelis 98661fe012 [Bugfix][KVConnector] Support DCP/PCP in OffloadingConnector (#41549)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
2026-05-05 14:54:29 +03:00
Ronen Schaffer 415a879899 [KV Offload] Use Collection instead of Sequence/Iterable for OffloadingManager key parameters (#41361)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-05-01 05:18:38 +03:00
Martin HickeyandOr Ozeri c42981d034 [Refactor][kv_offload] KV Offloading maintainability improvements (#40538)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-04-30 05:55:31 +03:00