[Feature] Detect all2all peer fault with fault tolerance backend and prevent corrupted output (#43637)

Signed-off-by: fangyuchu <fangyuchu@qq.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
fangyuchu
2026-06-30 09:00:25 -07:00
committed by GitHub
co-authored by Tyler Michael Smith mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent e840f0d3f5
commit 245888ff77
5 changed files with 82 additions and 4 deletions
+5 -1
View File
@@ -939,7 +939,11 @@ class WorkerProc:
converted to a FAILURE response.
"""
if isinstance(output, AsyncModelRunnerOutput):
output = output.get_output()
try:
output = output.get_output()
except Exception as e:
logger.exception("Error getting async model runner output")
output = e
if isinstance(output, Exception):
result = (WorkerProc.ResponseStatus.FAILURE, str(output))