[Bugfix][HiSparse] Fix TP group hang on steps without scheduled tokens (#60941)
Signed-off-by: Maroon Ayoub <mayoub@redhat.com> Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com> Co-authored-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Robert Shaw
Matthew Bonanni
Claude Opus 5.5
parent
a7b7bc1fbe
commit
5b0b8f251e
@@ -416,6 +416,10 @@ class HiSparseConnectorWorker:
|
||||
self.host_write_event = self.host_write_events[self._next_host_write_event]
|
||||
self._next_host_write_event ^= 1
|
||||
current_stream().wait_event(previous_host_write_event)
|
||||
if self.shared_host_region is not None and not num_tokens:
|
||||
# Without a forward, no TP collective keeps rank 0 from recording
|
||||
# a shared event again before every rank has issued its wait.
|
||||
get_tp_group().barrier()
|
||||
self._release_completed_dma_descriptors()
|
||||
self._dma_submitted = False
|
||||
mirrors = _flatten_row_mirrors(metadata.row_mirrors, request_ids)
|
||||
|
||||
Reference in New Issue
Block a user