ple, tests: debug-gated pull-flag check; diagnostics report that they ran
The pull-transport flag check is wired in (rank-gated, off by default, warns when enabled). Both diagnostic checks log that they ran. Test repairs for helpers v0.30.0 renamed. Co-Authored-By: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
@@ -476,3 +476,88 @@ class PleOffloadLayer(nn.Module, ABC):
|
||||
"""Mark the cross-process output buffer reusable on ``stream``."""
|
||||
if self._is_cpu_offloaded:
|
||||
self._sem.reset(stream)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pre-launch staleness check for the pull transport (#53899 class)
|
||||
# ---------------------------------------------------------------------------
|
||||
# WHAT IT TESTS. The transport's correctness rests on one precondition: when a
|
||||
# real request is published, every layer's semaphore must be CLEAR, so the
|
||||
# ``cuStreamWaitValue32(flag == DONE_VALUE)`` inside the forward (line 279 for
|
||||
# the pull path, line 215 for the IPC push path) can only be satisfied by the
|
||||
# copy THIS step publishes. If a leftover ``DONE_VALUE`` survives into a real
|
||||
# step, that wait is satisfied immediately by the PREVIOUS step's signal and
|
||||
# the forward consumes the previous step's PLE rows. That is vllm#53899 on
|
||||
# GB10/sm_121 exactly: "every forward consumes the previous step's PLE
|
||||
# outputs", for the life of the server, with ``cudagraph_mode=NONE`` correct.
|
||||
#
|
||||
# In this tree the precondition is held structurally -- the dummy signal is
|
||||
# consumed by the dummy forward, ``_release_outputs_impl`` resets every layer
|
||||
# and synchronizes the model stream while ``_after_dummy`` is set, and the
|
||||
# dummy->real transition takes a TP barrier -- but nothing ever READ the flag
|
||||
# back, so the structural argument had no instrument. This is that instrument.
|
||||
#
|
||||
# WHY NOT AN EPOCH WORD. The stronger fix is a monotone epoch: the launch
|
||||
# writes ``epoch = step``, the worker signals ``epoch``, and the placeholder
|
||||
# waits ``EQ(epoch)``, so a stale signal carries the previous epoch and cannot
|
||||
# satisfy the wait. It is not implementable on this transport as it stands.
|
||||
# The compare value is an IMMEDIATE OPERAND of ``cuStreamWaitValue32``
|
||||
# (``_ple_offload_wait_copy_impl``, this file, line 279), and that call is
|
||||
# issued inside the model forward, which is captured: PIECEWISE and
|
||||
# FULL_AND_PIECEWISE both bake the wait into the graph with the value frozen at
|
||||
# capture time. Making the compare value change per replay needs the captured
|
||||
# node's parameters rewritten before each launch
|
||||
# (``cuGraphExecBatchMemOpNodeSetParams``); nothing in this tree's capture path
|
||||
# exposes the graph exec handle to the connector, and the wait is reached
|
||||
# through a torch custom op inside a compiled region. An epoch word would
|
||||
# therefore work only with ``cudagraph_mode=NONE``, i.e. only in the
|
||||
# configuration where #53899 does not reproduce. So: binary flag, read back.
|
||||
#
|
||||
# The check is HOST-SIDE and needs a stream synchronize to be ordered against
|
||||
# a pending reset, so it is never run inside a capture and is off by default.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def flag_check_is_authoritative(tp_rank: int) -> bool:
|
||||
"""Whether ``flag == RESET_VALUE`` before a launch is an INVARIANT here.
|
||||
|
||||
Only on the publishing rank (TP0). Why: the offload worker fans a
|
||||
result out to every registered TP target, each gated only on that target's
|
||||
own flag having been reset, so a non-publishing rank's flag can legitimately
|
||||
carry THIS step's signal before that rank reaches its own pre-launch check.
|
||||
Reading a set flag there and calling it stale terminates a healthy run.
|
||||
|
||||
On TP0 the invariant holds by construction: the worker learns of a request
|
||||
only from ``_launch``, which runs after the check, and the progress monitor
|
||||
refuses overlapping serving submissions, so the previous transaction has
|
||||
been released and (after a stream synchronize) its reset has retired.
|
||||
|
||||
A non-publishing rank cannot tell "this step's fan-out arrived early" from
|
||||
"last step's signal was never consumed", because the flag is BINARY and
|
||||
carries no transaction identity -- the same representational limit that
|
||||
rules out an epoch word. So those ranks are skipped, not checked weakly.
|
||||
"""
|
||||
return tp_rank == 0
|
||||
|
||||
|
||||
def stale_offload_flags(
|
||||
layers: "dict[str, PleOffloadLayer]",
|
||||
) -> list[tuple[str, int]]:
|
||||
"""Return ``(layer_name, flag_value)`` for every layer whose flag is set.
|
||||
|
||||
An empty list is the precondition holding. A non-empty list means at least
|
||||
one semaphore still carries a signal that no forward consumed, so the next
|
||||
``wait`` would be satisfied by the previous step's copy.
|
||||
|
||||
Pure and side-effect free: the caller owns the ordering (synchronize the
|
||||
model stream first, and never call this inside a stream capture). Layers
|
||||
that are not cross-process offloaded have no semaphore and are skipped.
|
||||
"""
|
||||
stale: list[tuple[str, int]] = []
|
||||
for name, layer in layers.items():
|
||||
if not getattr(layer, "_is_cpu_offloaded", False):
|
||||
continue
|
||||
value = int(layer._sem.flag_tensor.reshape(-1)[0].item())
|
||||
if value != CpuGpuSemaphore.RESET_VALUE:
|
||||
stale.append((name, value))
|
||||
return stale
|
||||
|
||||
Reference in New Issue
Block a user