23 Commits
Author SHA1 Message Date
5341b5ba7d [KV Offload] Pin only each worker's own offload slots (#59544)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-10-15 08:49:42 +03:00
Chaojun Zhang b027ac8312 [XPU] Support register KV offload mmap region as pinned host memory on XPU (#51956)
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
2026-10-09 18:58:07 +08:00
ca0df1f122 [Bugfix][Core] Exempt exactly the blocks an async KV load writes from zeroing (#59504)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-03 06:26:12 +00:00
40b40d1d39 [Bugfix][KV Offload] Register the offload region in chunks (#51081)
Signed-off-by: Mikhail Kostryukov <mike@triptrack.net>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Colton Ottley <coltonottley@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-09-17 13:19:56 +03:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
Ronen Schaffer 138d137b5b [Refactor][kv_offload]: rename block→chunk (#52615)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-09-09 17:45:49 +03:00
Itay EtelisItay Etelismergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
bed3280f50 [KV offload] Order CPU->GPU loads against the compute stream (#50696)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-31 14:31:18 +03:00
Andreas KaratzasandOpenAI Codex 1ab2801dde [ROCm] Remove stale SDPA and skinny GEMM workarounds (#50907)
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-08-11 14:19:16 -04:00
AlexHuang 0fec3d652b [Bugfix][KV Offload] Centralize shared mmap cleanup in CPU worker (#51622)
Signed-off-by: Alex <jihui.huang@daocloud.io>
2026-08-11 13:22:15 +03:00
Itay EtelisandItay Etelis 81840a172f [KV Connector] Canonical CPU layout for parallelism-agnostic KV offload (#48414)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
2026-08-10 10:44:54 +03:00
Andreas KaratzasandOpenAI Codex 394beb633b [Bugfix][ROCm] Use batch DMA for CPU KV cache loads (#49843)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-07-27 02:18:01 -07:00
a9531edfa6 [KV Offload] Define clean backend configuration boundary (#48150)
Signed-off-by: Change72 <changg@nvidia.com>
Signed-off-by: Chang Guo <cguo51@asu.edu>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-07-16 13:27:05 +03:00
Varun Sundar RabindranathandVarun Sundar Rabindranath 36bbecd643 [BugFix] Revert "[KV Offload] Use background thread for mmap / cpu_tensors pinning" (#46958)
Signed-off-by: <>
Co-authored-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com>
2026-06-29 07:54:34 -07:00
Yiwei Hu f889325c51 [KV Offload] Use background thread for mmap / cpu_tensors pinning (#45850)
Signed-off-by: Sorryhorizon <arikara6666@gmail.com>
2026-06-24 18:13:27 +03:00
Martin Hickey f237e16b41 [KV Offload] Replace OffloadingHandler with OffloadingWorker (#45053)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
2026-06-24 14:44:24 +03:00
Nick Hill 7df3d7dada [Core] Ensure memory is pinned prior to async h2d copy (#45424)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-06-20 20:02:24 -07:00
Chaojun Zhang 810966453a [XPU] Support cpu kv offloading and tiering offloading on XPU platform (#36423)
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
2026-06-07 09:59:28 +08:00
Itay EtelisItay Etelismergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
1fa9ea09f6 [Perf] Triton fast path for small CPU→GPU swap_blocks_batch in the offloading connector (#42212)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-03 13:38:17 +03:00
Ronen Schaffer 93da882e73 [kv_offload] Add @override decorators to subclass method implementations (#44177)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-06-02 08:07:47 +00:00
Ronen Schaffer 11f6b545d4 [kv_offload] Add multi-tier KV cache offloading framework (#40020)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-05-13 17:21:43 +03:00
Itay EtelisItay EtelisOr OzeriItay Etelismergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
00b0618a03 Use CU_MEMCPY_SRC_ACCESS_ORDER_ANY for batch KV cache swaps (#39306)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Signed-off-by: Itay Etelis <etelis2019@gmail.com>
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Itay Etelis <etelis2019@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-10 05:57:09 +03:00
Martin HickeyandOr Ozeri c42981d034 [Refactor][kv_offload] KV Offloading maintainability improvements (#40538)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-04-30 05:55:31 +03:00