[KV Connector][Offloading] Flush all pending jobs on last step (#42611)

Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: liranschour <liranschour@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
liranschour
2026-05-18 12:59:42 +00:00
committed by GitHub
co-authored by Or Ozeri mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent 2e40faf08b
commit e5417657e5
3 changed files with 71 additions and 14 deletions
@@ -782,6 +782,14 @@ class OffloadingConnectorScheduler:
assert self._jobs[any_jid].is_store
self._current_batch_jobs_to_flush.update(req_status.transfer_jobs)
# If all tracked requests are finished, flush all pending jobs
# (both store and load) - there might not be a future scheduler
# step to trigger their completion.
if self._req_status and all(
rs.req.is_finished() for rs in self._req_status.values()
):
self._current_batch_jobs_to_flush.update(self._jobs.keys())
meta = OffloadingConnectorMetadata(
load_jobs=self._current_batch_load_jobs,
store_jobs=self._build_store_jobs(scheduler_output),