ple, tests: debug-gated pull-flag check; diagnostics report that they ran

The pull-transport flag check is wired in (rank-gated, off by default, warns when enabled). Both
diagnostic checks log that they ran. Test repairs for helpers v0.30.0 renamed.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
2026-10-09 20:01:26 +07:00
co-authored by Hinata
parent fb52250c11
commit 7f6055caa2
2 changed files with 236 additions and 0 deletions
@@ -476,3 +476,88 @@ class PleOffloadLayer(nn.Module, ABC):
"""Mark the cross-process output buffer reusable on ``stream``."""
if self._is_cpu_offloaded:
self._sem.reset(stream)
# ---------------------------------------------------------------------------
# Pre-launch staleness check for the pull transport (#53899 class)
# ---------------------------------------------------------------------------
# WHAT IT TESTS. The transport's correctness rests on one precondition: when a
# real request is published, every layer's semaphore must be CLEAR, so the
# ``cuStreamWaitValue32(flag == DONE_VALUE)`` inside the forward (line 279 for
# the pull path, line 215 for the IPC push path) can only be satisfied by the
# copy THIS step publishes. If a leftover ``DONE_VALUE`` survives into a real
# step, that wait is satisfied immediately by the PREVIOUS step's signal and
# the forward consumes the previous step's PLE rows. That is vllm#53899 on
# GB10/sm_121 exactly: "every forward consumes the previous step's PLE
# outputs", for the life of the server, with ``cudagraph_mode=NONE`` correct.
#
# In this tree the precondition is held structurally -- the dummy signal is
# consumed by the dummy forward, ``_release_outputs_impl`` resets every layer
# and synchronizes the model stream while ``_after_dummy`` is set, and the
# dummy->real transition takes a TP barrier -- but nothing ever READ the flag
# back, so the structural argument had no instrument. This is that instrument.
#
# WHY NOT AN EPOCH WORD. The stronger fix is a monotone epoch: the launch
# writes ``epoch = step``, the worker signals ``epoch``, and the placeholder
# waits ``EQ(epoch)``, so a stale signal carries the previous epoch and cannot
# satisfy the wait. It is not implementable on this transport as it stands.
# The compare value is an IMMEDIATE OPERAND of ``cuStreamWaitValue32``
# (``_ple_offload_wait_copy_impl``, this file, line 279), and that call is
# issued inside the model forward, which is captured: PIECEWISE and
# FULL_AND_PIECEWISE both bake the wait into the graph with the value frozen at
# capture time. Making the compare value change per replay needs the captured
# node's parameters rewritten before each launch
# (``cuGraphExecBatchMemOpNodeSetParams``); nothing in this tree's capture path
# exposes the graph exec handle to the connector, and the wait is reached
# through a torch custom op inside a compiled region. An epoch word would
# therefore work only with ``cudagraph_mode=NONE``, i.e. only in the
# configuration where #53899 does not reproduce. So: binary flag, read back.
#
# The check is HOST-SIDE and needs a stream synchronize to be ordered against
# a pending reset, so it is never run inside a capture and is off by default.
# ---------------------------------------------------------------------------
def flag_check_is_authoritative(tp_rank: int) -> bool:
"""Whether ``flag == RESET_VALUE`` before a launch is an INVARIANT here.
Only on the publishing rank (TP0). Why: the offload worker fans a
result out to every registered TP target, each gated only on that target's
own flag having been reset, so a non-publishing rank's flag can legitimately
carry THIS step's signal before that rank reaches its own pre-launch check.
Reading a set flag there and calling it stale terminates a healthy run.
On TP0 the invariant holds by construction: the worker learns of a request
only from ``_launch``, which runs after the check, and the progress monitor
refuses overlapping serving submissions, so the previous transaction has
been released and (after a stream synchronize) its reset has retired.
A non-publishing rank cannot tell "this step's fan-out arrived early" from
"last step's signal was never consumed", because the flag is BINARY and
carries no transaction identity -- the same representational limit that
rules out an epoch word. So those ranks are skipped, not checked weakly.
"""
return tp_rank == 0
def stale_offload_flags(
layers: "dict[str, PleOffloadLayer]",
) -> list[tuple[str, int]]:
"""Return ``(layer_name, flag_value)`` for every layer whose flag is set.
An empty list is the precondition holding. A non-empty list means at least
one semaphore still carries a signal that no forward consumed, so the next
``wait`` would be satisfied by the previous step's copy.
Pure and side-effect free: the caller owns the ordering (synchronize the
model stream first, and never call this inside a stream capture). Layers
that are not cross-process offloaded have no semaphore and are skipped.
"""
stale: list[tuple[str, int]] = []
for name, layer in layers.items():
if not getattr(layer, "_is_cpu_offloaded", False):
continue
value = int(layer._sem.flag_tensor.reshape(-1)[0].item())
if value != CpuGpuSemaphore.RESET_VALUE:
stale.append((name, value))
return stale