19 Commits
Author SHA1 Message Date
21f4b6dc84 [KV Offload] Split the CPU offload region across shm files (#59546)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 20:17:40 +00:00
5341b5ba7d [KV Offload] Pin only each worker's own offload slots (#59544)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-10-15 08:49:42 +03:00
Itay EtelisandCodex efad400092 [HiSparse] Prefault the shared host pool in parallel (#60869)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-14 10:28:26 -04:00
Kapil AryaandOr Ozeri a68dbbad05 [Bugfix][KV Offload] Retain offload mappings after unsafe KVCR startup failure (#57500)
Signed-off-by: Kapil Arya <kapila@nvidia.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-10-14 15:43:04 +03:00
Itay EtelisRobert ShawCodexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
483e0c24c7 [KV Offload] Split replicated-layout pre-faulting across workers (#59545)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-13 19:18:56 +00:00
Chaojun Zhang b027ac8312 [XPU] Support register KV offload mmap region as pinned host memory on XPU (#51956)
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
2026-10-09 18:58:07 +08:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
0f8fa53acf [Bugfix][KV Offload] Check cgroup memory before SHM allocation (#54014)
Signed-off-by: Alex <jihui.huang@daocloud.io>
Signed-off-by: AlexHuang <alex.tech.lab@outlook.com>
Signed-off-by: Alex <alex.tech.lab@outlook.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
2026-09-16 09:17:50 +03:00
e19a3e172e [5/N] Share HiSparse host cache across TP ranks (reopens #52760) (#56629)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Chao Lei <leichao139636@163.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Chao Lei <leichao139636@163.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-09-12 19:48:25 -04:00
Ronen Schaffer 138d137b5b [Refactor][kv_offload]: rename block→chunk (#52615)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-09-09 17:45:49 +03:00
4c58a0c398 [Bugfix][KV Offload] Unlink /dev/shm region after all workers map it (barrier variant of #51317) (#52596)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-08-31 15:53:00 +03:00
Itay EtelisandItay Etelis 81840a172f [KV Connector] Canonical CPU layout for parallelism-agnostic KV offload (#48414)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
2026-08-10 10:44:54 +03:00
ef2615c2e0 [Bugfix][KV Offload] Fall back when MADV_POPULATE_WRITE is unsupported (#51116)
Signed-off-by: Guy Girmonsky <guygir@gmail.com>
Signed-off-by: Alex <jihui.huang@daocloud.io>
Co-authored-by: Guy Girmonsky <guygir@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-08-06 13:48:01 +03:00
9833aa53d5 [Bugfix] Fail fast with a clear error when CPU offload region exceeds available space (#50358)
Signed-off-by: Alex <jihui.huang@daocloud.io>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-08-05 20:59:55 +03:00
liranschourandOr Ozeri 5c0c987c03 Make tiering offload region DP-replica aware (#47987)
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-12 13:10:21 +03:00
Chaojun Zhang 810966453a [XPU] Support cpu kv offloading and tiering offloading on XPU platform (#36423)
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
2026-06-07 09:59:28 +08:00
Varun Sundar Rabindranathandvarun sundar rabindranath 3d76f395e3 [SharedOffloadRegion] Align blocks to page-size (#43689)
Signed-off-by: varun sundar rabindranath <vsundarr@redhat.com>
Co-authored-by: varun sundar rabindranath <vsundarr@redhat.com>
2026-06-03 14:25:57 +03:00
Ronen Schaffer 11f6b545d4 [kv_offload] Add multi-tier KV cache offloading framework (#40020)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-05-13 17:21:43 +03:00
omerpaz95andOr Ozeri be0c855ebd [KV Offload] Unified memory layout for offloading workers (#37206)
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-04-14 21:33:33 +03:00