[Bugfix][MoRIIO] Exclude synchronous READ destinations from KV zeroing (#59164)
Signed-off-by: Yichao Zhu <Yichao.Zhu@amd.com> Signed-off-by: whx-sjtu <xiaowang990929@gmail.com> Signed-off-by: Hexiang Wang <56632993+whx-sjtu@users.noreply.github.com> Co-authored-by: Yichao Zhu <Yichao.Zhu@amd.com> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
This commit is contained in:
co-authored by
Yichao Zhu
Codex
Yifan Qiao
parent
42ecc8eebe
commit
df417f780a
@@ -583,7 +583,11 @@ class KVConnectorBase_V1(ABC):
|
||||
"""Build the connector metadata for this step.
|
||||
|
||||
This function should NOT modify fields in the scheduler_output.
|
||||
Also, calling this function will reset the state of the connector.
|
||||
FIXME: one exception:
|
||||
synchronous READ connectors may remove attention blocks they fully
|
||||
overwrite this step from new_block_ids_to_zero. They must complete
|
||||
those loads before the blocks are used and fail the step if a load fails.
|
||||
Calling this function will reset the state of the connector.
|
||||
|
||||
Args:
|
||||
scheduler_output (SchedulerOutput): the scheduler output object.
|
||||
|
||||
@@ -1232,6 +1232,24 @@ class MoRIIOConnectorScheduler:
|
||||
local_block_ids=block_ids,
|
||||
kv_transfer_params=kv_params,
|
||||
)
|
||||
if (
|
||||
self.mode == MoRIIOMode.READ
|
||||
and self._has_mamba
|
||||
and scheduler_output.new_block_ids_to_zero
|
||||
):
|
||||
# Hybrid models zero recycled attention pages that held Mamba state.
|
||||
# Host-submitted READs overwrite these pages and can race zeroing.
|
||||
# Hybrid READ metadata puts the aligned attention pages first.
|
||||
read_dst_block_ids = {
|
||||
b
|
||||
for _, block_ids in self._reqs_need_recv.values()
|
||||
for b in block_ids[0]
|
||||
}
|
||||
scheduler_output.new_block_ids_to_zero = [
|
||||
b
|
||||
for b in scheduler_output.new_block_ids_to_zero
|
||||
if b not in read_dst_block_ids
|
||||
]
|
||||
|
||||
for req_id, (req, block_ids) in self._reqs_need_save.items():
|
||||
kv_params = self._req_kv_params.get(req_id, req.kv_transfer_params or {})
|
||||
|
||||
Reference in New Issue
Block a user