29 Commits
Author SHA1 Message Date
e10c10a644 [Core] Add PROMOTION_LATENCY histogram metric for tiering offload (#53910)
Signed-off-by: Wen Zhou <wenzhou@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-10-15 13:47:41 +03:00
liranschourandKevin H. Luu c5f8adc0b1 [KV Offload] Expose OffloadingManager.lock()/unlock() and service the tiering control plane off the scheduler thread (#58168)
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 08:50:56 +03:00
775fefa804 [Metrics][KV Offload] Add vllm:kv_offload_config_info - generic info metrics for offloading the offloading connector (#56867)
Signed-off-by: Amir Friedman <Amir.Friedman1@ibm.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-10-14 12:34:37 +00:00
Tony Lin 2bea22002d [BUG] Fix KV offload shared-memory slot selection using worker rank (#58497)
Signed-off-by: Tony Lin <tony.lin@intel.com>
2026-10-10 20:25:11 +08:00
bnellnmandItay Etelis 23e26e0588 [KV Offloading] Back-pressure detection and remediation (#50045)
Signed-off-by: Bill Nell <bnell@redhat.com>
Signed-off-by: bnellnm <49004751+bnellnm@users.noreply.github.com>
Co-authored-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
2026-09-18 14:47:48 -04:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
Ronen Schaffer 138d137b5b [Refactor][kv_offload]: rename block→chunk (#52615)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-09-09 17:45:49 +03:00
Itay EtelisandItay Etelis 81840a172f [KV Connector] Canonical CPU layout for parallelism-agnostic KV offload (#48414)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
2026-08-10 10:44:54 +03:00
Srinivas Krovvidiandsrinivas_oo7 a123159f7a Add tiering offloading metrics (#48798)
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
2026-08-10 05:23:14 +03:00
AlexHuangandOpenAI Codex 46e6a83ce1 [Bugfix][KV Offload] Clean up resources after initialization failure (#51227)
Signed-off-by: Alex <jihui.huang@daocloud.io>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-08-06 14:42:13 +03:00
Ronen SchafferandOr Ozeri 8543522ca7 [KV Offload] Support out-of-tree secondary tier managers via module_path (#51007)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-08-05 20:30:53 +03:00
Chang GuoandCursor Agent 541128bdeb [KV Offload] Enable single-copy MLA layout for CPUOffloadingSpec (#50301)
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: Cursor Agent <cursor-agent@cursor.com>
2026-07-31 06:20:47 +03:00
Philip Pesic dc1be79031 Add CachePolicyFactory for pluggable/external eviction policies (#49114)
Signed-off-by: Philip Pesic <philippesic06@gmail.com>
2026-07-29 07:34:55 +03:00
Chang GuoandOpenAI Codex 7eca0e1a64 [KV Offload] Deduplicate replicated MLA KV in the shared CPU region (#48906)
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-07-26 08:22:30 +03:00
Chang GuoandOpenAI Codex f007cceb42 [KV Offload] Support self-describing KV events with TieringOffloadingSpec (#48679)
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-07-20 16:41:58 +03:00
Srinivas Krovvidiandsrinivas_oo7 ce65385618 [KV Offload] Split tiering_lookup_delay into sync/async histograms (#47679)
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
2026-07-16 20:03:45 +03:00
a9531edfa6 [KV Offload] Define clean backend configuration boundary (#48150)
Signed-off-by: Change72 <changg@nvidia.com>
Signed-off-by: Chang Guo <cguo51@asu.edu>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-07-16 13:27:05 +03:00
liranschourandOr Ozeri 5c0c987c03 Make tiering offload region DP-replica aware (#47987)
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-12 13:10:21 +03:00
c46ced1ee3 [kv_offload] Establish tier-owned KV event handling (#46544)
Signed-off-by: Change72 <changg@nvidia.com>
Signed-off-by: Chang Guo <changg@nvidia.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-07 16:55:20 +03:00
Martin Hickey f237e16b41 [KV Offload] Replace OffloadingHandler with OffloadingWorker (#45053)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
2026-06-24 14:44:24 +03:00
Srinivas Krovvidiandsrinivas_oo7 091bc1026e [KV Offloading] Add tiering metric plumbing (#45959)
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
2026-06-23 15:10:36 +03:00
Change72andClaude a9f7b2d41c [feature][kv_offload] Self-describing KV events for OffloadingConnector (#43468)
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-06-22 07:27:46 +00:00
Effi Ofer 6a894574bf Add objectstore as a secondary tier to multi-tier kv cache offloading (#41968)
Signed-off-by: Effi Ofer <effi.ofer@gmail.com>
2026-06-05 18:05:41 +03:00
Varun Sundar Rabindranathandvarun sundar rabindranath 3d76f395e3 [SharedOffloadRegion] Align blocks to page-size (#43689)
Signed-off-by: varun sundar rabindranath <vsundarr@redhat.com>
Co-authored-by: varun sundar rabindranath <vsundarr@redhat.com>
2026-06-03 14:25:57 +03:00
Varun Sundar Rabindranathandvarun sundar rabindranath 7268457999 [KV Offloading] Enable HMA models for Tiering Offloading (#44287)
Signed-off-by: varun sundar rabindranath <vsundarr@redhat.com>
Co-authored-by: varun sundar rabindranath <vsundarr@redhat.com>
2026-06-03 10:03:00 +03:00
Ronen Schaffer 4f940896a3 [KV Offload] Pass OffloadingSpec instead of VllmConfig to secondary tiers (#43076)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-05-20 03:32:08 +00:00
Ronen Schaffer 84747489de Tier offload followup (#42529)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-05-18 19:41:58 +00:00
Ronen Schaffer 11f6b545d4 [kv_offload] Add multi-tier KV cache offloading framework (#40020)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-05-13 17:21:43 +03:00