  
|
21f4b6dc84
|
[KV Offload] Split the CPU offload region across shm files (#59546)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
|
2026-10-15 20:17:40 +00:00 |
|
   
|
5341b5ba7d
|
[KV Offload] Pin only each worker's own offload slots (#59544)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-10-15 08:49:42 +03:00 |
|
 Itay EtelisandCodex
|
efad400092
|
[HiSparse] Prefault the shared host pool in parallel (#60869)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
|
2026-10-14 10:28:26 -04:00 |
|
 Kapil AryaandOr Ozeri
|
a68dbbad05
|
[Bugfix][KV Offload] Retain offload mappings after unsafe KVCR startup failure (#57500)
Signed-off-by: Kapil Arya <kapila@nvidia.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-10-14 15:43:04 +03:00 |
|
 
|
775fefa804
|
[Metrics][KV Offload] Add vllm:kv_offload_config_info - generic info metrics for offloading the offloading connector (#56867)
Signed-off-by: Amir Friedman <Amir.Friedman1@ibm.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-10-14 12:34:37 +00:00 |
|
![mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>](/assets/img/avatar_default.png)  
|
483e0c24c7
|
[KV Offload] Split replicated-layout pre-faulting across workers (#59545)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-10-13 19:18:56 +00:00 |
|
Tony Lin
|
2bea22002d
|
[BUG] Fix KV offload shared-memory slot selection using worker rank (#58497)
Signed-off-by: Tony Lin <tony.lin@intel.com>
|
2026-10-10 20:25:11 +08:00 |
|
Chaojun Zhang
|
b027ac8312
|
[XPU] Support register KV offload mmap region as pinned host memory on XPU (#51956)
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
|
2026-10-09 18:58:07 +08:00 |
|
  
|
e37e51dd24
|
[Metrics] Expose cached prompt tokens by cache tier (#56318)
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cameron@semianalysis.com>
Co-authored-by: Cam Quilici <cameron@semianalysis.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
|
2026-10-06 14:52:05 -04:00 |
|
 
|
ca0df1f122
|
[Bugfix][Core] Exempt exactly the blocks an async KV load writes from zeroing (#59504)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
|
2026-10-03 06:26:12 +00:00 |
|
Ren Yuzhou
|
7b94293627
|
[Bugfix][ROCm][KV Offload] Use private pinned tensors for CPU KV offload (#57160)
Signed-off-by: Yuzhou Ren <ryzcqu@outlook.com>
|
2026-09-18 02:35:23 +00:00 |
|
  
|
40b40d1d39
|
[Bugfix][KV Offload] Register the offload region in chunks (#51081)
Signed-off-by: Mikhail Kostryukov <mike@triptrack.net>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Colton Ottley <coltonottley@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-09-17 13:19:56 +03:00 |
|
 Harry MellorandClaude Opus 5
|
651a88c09a
|
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
2026-09-16 10:57:24 -07:00 |
|
 Harry MellorandClaude Opus 5
|
8c1557a79c
|
Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
2026-09-16 16:44:29 +00:00 |
|
 MINJUN GILandOr Ozeri
|
f12fe10c5c
|
[Bugfix][KV Offload] Track cache recency once per request (#51787)
Signed-off-by: mindungil <alswnsrlf12@naver.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-09-16 10:47:26 +03:00 |
|
 
|
0f8fa53acf
|
[Bugfix][KV Offload] Check cgroup memory before SHM allocation (#54014)
Signed-off-by: Alex <jihui.huang@daocloud.io>
Signed-off-by: AlexHuang <alex.tech.lab@outlook.com>
Signed-off-by: Alex <alex.tech.lab@outlook.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
|
2026-09-16 09:17:50 +03:00 |
|
    
|
e19a3e172e
|
[5/N] Share HiSparse host cache across TP ranks (reopens #52760) (#56629)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Chao Lei <leichao139636@163.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Chao Lei <leichao139636@163.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
|
2026-09-12 19:48:25 -04:00 |
|
Ronen Schaffer
|
138d137b5b
|
[Refactor][kv_offload]: rename block→chunk (#52615)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
|
2026-09-09 17:45:49 +03:00 |
|
 Zupeng WangandCodex
|
dd0760165c
|
[Bugfix] Fall back to T1 when ARC cannot reclaim enough entries from T2 (#55461)
Signed-off-by: Wang Zupeng <zupenwang@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
|
2026-09-06 13:28:19 +03:00 |
|
 Cheng RuiandCodex
|
2a4e3cc3db
|
[Bugfix][KV Offload] Ensure tracker progress for oversized offers (#54759)
Signed-off-by: Cheng Rui <286040359@qq.com>
Co-authored-by: Codex <noreply@openai.com>
|
2026-09-02 14:09:29 +03:00 |
|
 
|
4c58a0c398
|
[Bugfix][KV Offload] Unlink /dev/shm region after all workers map it (barrier variant of #51317) (#52596)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: NickLucche <nicolo.lucchesi@mistral.ai>
|
2026-08-31 15:53:00 +03:00 |
|
![mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>](/assets/img/avatar_default.png) 
|
bed3280f50
|
[KV offload] Order CPU->GPU loads against the compute stream (#50696)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-08-31 14:31:18 +03:00 |
|
 
|
e6bfe03ad7
|
Count store offers, not lookups, for CPU offload store_threshold (#52227)
Signed-off-by: okorzh-amd <okorzh-amd@users.noreply.github.com>
Co-authored-by: okorzh-amd <okorzh-amd@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
2026-08-27 20:21:06 -07:00 |
|
 Andreas KaratzasandOpenAI Codex
|
1ab2801dde
|
[ROCm] Remove stale SDPA and skinny GEMM workarounds (#50907)
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-08-11 14:19:16 -04:00 |
|
AlexHuang
|
0fec3d652b
|
[Bugfix][KV Offload] Centralize shared mmap cleanup in CPU worker (#51622)
Signed-off-by: Alex <jihui.huang@daocloud.io>
|
2026-08-11 13:22:15 +03:00 |
|
 Itay EtelisandItay Etelis
|
81840a172f
|
[KV Connector] Canonical CPU layout for parallelism-agnostic KV offload (#48414)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
|
2026-08-10 10:44:54 +03:00 |
|
 AlexHuangandOpenAI Codex
|
46e6a83ce1
|
[Bugfix][KV Offload] Clean up resources after initialization failure (#51227)
Signed-off-by: Alex <jihui.huang@daocloud.io>
Co-authored-by: OpenAI Codex <noreply@openai.com>
|
2026-08-06 14:42:13 +03:00 |
|
  
|
ef2615c2e0
|
[Bugfix][KV Offload] Fall back when MADV_POPULATE_WRITE is unsupported (#51116)
Signed-off-by: Guy Girmonsky <guygir@gmail.com>
Signed-off-by: Alex <jihui.huang@daocloud.io>
Co-authored-by: Guy Girmonsky <guygir@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
|
2026-08-06 13:48:01 +03:00 |
|
 
|
9833aa53d5
|
[Bugfix] Fail fast with a clear error when CPU offload region exceeds available space (#50358)
Signed-off-by: Alex <jihui.huang@daocloud.io>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-08-05 20:59:55 +03:00 |
|
 Ronen SchafferandOr Ozeri
|
8543522ca7
|
[KV Offload] Support out-of-tree secondary tier managers via module_path (#51007)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-08-05 20:30:53 +03:00 |
|
MINJUN GIL
|
b92352ca2c
|
[Perf][KV Offload] Avoid quadratic ARC batch eviction (#50992)
Signed-off-by: MINJUN GIL <alswnsrlf12@naver.com>
|
2026-08-05 10:58:36 +03:00 |
|
 Chang GuoandCursor Agent
|
541128bdeb
|
[KV Offload] Enable single-copy MLA layout for CPUOffloadingSpec (#50301)
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: Cursor Agent <cursor-agent@cursor.com>
|
2026-07-31 06:20:47 +03:00 |
|
 Chang GuoandCursor Agent
|
542a8fad6d
|
[KV Offload] Move CPUOffloadingSpec onto SharedOffloadRegion (#50094)
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: Cursor Agent <cursor-agent@cursor.com>
|
2026-07-29 13:05:50 +03:00 |
|
Philip Pesic
|
dc1be79031
|
Add CachePolicyFactory for pluggable/external eviction policies (#49114)
Signed-off-by: Philip Pesic <philippesic06@gmail.com>
|
2026-07-29 07:34:55 +03:00 |
|
Ronen Schaffer
|
77cba0259f
|
[KV Offloading] Per-request tier filtering with TierFilter/TierMatcher (#48123)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
|
2026-07-27 13:29:57 +03:00 |
|
 Andreas KaratzasandOpenAI Codex
|
394beb633b
|
[Bugfix][ROCm] Use batch DMA for CPU KV cache loads (#49843)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
|
2026-07-27 02:18:01 -07:00 |
|
 Chang GuoandOpenAI Codex
|
7eca0e1a64
|
[KV Offload] Deduplicate replicated MLA KV in the shared CPU region (#48906)
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
|
2026-07-26 08:22:30 +03:00 |
|
   
|
a9531edfa6
|
[KV Offload] Define clean backend configuration boundary (#48150)
Signed-off-by: Change72 <changg@nvidia.com>
Signed-off-by: Chang Guo <cguo51@asu.edu>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
|
2026-07-16 13:27:05 +03:00 |
|
 
|
cdaa40d2a8
|
[KV Offload] Split cpu_cache_usage_perc into write/read usage gauges (#47666)
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-07-14 20:13:41 +03:00 |
|
 liranschourandOr Ozeri
|
5c0c987c03
|
Make tiering offload region DP-replica aware (#47987)
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-07-12 13:10:21 +03:00 |
|
 Posedge_LinandOr Ozeri
|
0ca6eee743
|
[Core] Pass request context to CPU offload cache policy touch (#47744)
Signed-off-by: jacklin78911-collab <jacklin78911@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-07-08 05:56:25 +03:00 |
|
 
|
c46ced1ee3
|
[kv_offload] Establish tier-owned KV event handling (#46544)
Signed-off-by: Change72 <changg@nvidia.com>
Signed-off-by: Chang Guo <changg@nvidia.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-07-07 16:55:20 +03:00 |
|
  
|
e040899a00
|
[KV Offloading] Add basic offloading metrics (#45958)
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Signed-off-by: Srinivas Krovvidi <194645829+Srinivasoo7@users.noreply.github.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-07-07 09:26:28 +03:00 |
|
 Varun Sundar RabindranathandVarun Sundar Rabindranath
|
36bbecd643
|
[BugFix] Revert "[KV Offload] Use background thread for mmap / cpu_tensors pinning" (#46958)
Signed-off-by: <>
Co-authored-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com>
|
2026-06-29 07:54:34 -07:00 |
|
Yiwei Hu
|
f889325c51
|
[KV Offload] Use background thread for mmap / cpu_tensors pinning (#45850)
Signed-off-by: Sorryhorizon <arikara6666@gmail.com>
|
2026-06-24 18:13:27 +03:00 |
|
 Ronen SchafferandOr Ozeri
|
bb61177e49
|
[KV Offloading] Replace bool|None lookup return with LookupResult enum (#46363)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-06-24 18:06:08 +03:00 |
|
Martin Hickey
|
f237e16b41
|
[KV Offload] Replace OffloadingHandler with OffloadingWorker (#45053)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
|
2026-06-24 14:44:24 +03:00 |
|
 Change72andClaude
|
a9f7b2d41c
|
[feature][kv_offload] Self-describing KV events for OffloadingConnector (#43468)
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: Claude <noreply@anthropic.com>
|
2026-06-22 07:27:46 +00:00 |
|
 Varun Sundar RabindranathandVarun Sundar Rabindranath
|
68567ef2df
|
[CPUOffloadingManager] Maintain evictable list in LRUCachePolicy (#46216)
Signed-off-by: <>
Co-authored-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com>
|
2026-06-22 06:54:44 +00:00 |
|
Nick Hill
|
7df3d7dada
|
[Core] Ensure memory is pinned prior to async h2d copy (#45424)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
|
2026-06-20 20:02:24 -07:00 |
|