[Bugfix][HiSparse] Fix TP group hang on steps without scheduled tokens (#60941)

Signed-off-by: Maroon Ayoub <mayoub@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Maroon Ayoub
2026-10-12 17:38:12 +00:00
committed by GitHub
co-authored by Robert Shaw Matthew Bonanni Claude Opus 5.5
parent a7b7bc1fbe
commit 5b0b8f251e
2 changed files with 51 additions and 0 deletions
@@ -416,6 +416,10 @@ class HiSparseConnectorWorker:
self.host_write_event = self.host_write_events[self._next_host_write_event]
self._next_host_write_event ^= 1
current_stream().wait_event(previous_host_write_event)
if self.shared_host_region is not None and not num_tokens:
# Without a forward, no TP collective keeps rank 0 from recording
# a shared event again before every rank has issued its wait.
get_tp_group().barrier()
self._release_completed_dma_descriptors()
self._dma_submitted = False
mirrors = _flatten_row_mirrors(metadata.row_mirrors, request_ids)