Port of the upstream vLLM change #58863 (Apache-2.0). The final-state folds run before the boundary-state folds, because in align
mode a boundary state can be written into the slot a final fold still reads.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Port of the upstream vLLM change #58863 (Apache-2.0). The GDN and PLE RecoverSSM metadata builders declare UNIFORM_BATCH, so
--use-replayssm resolves FULL_AND_PIECEWISE; graph padding rows get a null state slot and write nothing.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Port of the upstream vLLM change #58863 (Apache-2.0). Each program folds either the final or the block-boundary state, not both,
which halves the live accumulator tiles.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Port of the upstream vLLM change #58863 (Apache-2.0). One check read use_gdn_recoverssm directly; it now uses getattr with a False
default like its neighbours, so layers built without __init__ in tests work.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Port of the upstream vLLM change #58863 (Apache-2.0). The step recorded during graph-capture dummy runs held the profiling KV cache
alive after profiling; it is now released at teardown.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Port of the upstream vLLM change #58863 (Apache-2.0). With --use-replayssm and speculative decoding, Gated DeltaNet layers verify the
draft window from a read-only checkpoint and keep a per-token replay record; after sampling one commit replays
the accepted tokens and writes the recurrent state once. The PLE short conv gets the same treatment.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The FP8 KV clip counter's reader thread used to start on the first eager forward, before CUDA graph capture, so
its device-to-host copies could overlap a capture. _qsa_clip_count now only allocates the per-layer counter; the
reader is started by a hook that guard_telemetry runs after warmup and capture (or at once if capture is already
done). Counting is unchanged. Tests cover the order, idempotence and a failing control that restores the old start.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
VLLM_E2_GUARD_MODE now defaults to warn in code, matching what the shipped launchers set; v2 through v2.2.0
defaulted to off. The comment above it, the MODES docstring, the vllm/envs.py default and its annotation follow.
tests/v1/worker/gpu/test_guard_default_mode.py checks that code, comment and envs.py agree, from the sources, so
it runs without CUDA, and fails on the earlier source.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
A fault on a non-zero TP rank could hang the engine: the worker's sentinel pipe
was inherited by a helper process and stayed open, so the monitor never saw
the exit. Earlier releases carry the same monitor and fail-closed exit path.
- Worker exit is detected by pidfd, with the sentinel as fallback, before and
after READY and for every worker through the whole initialisation window.
- A per-worker fault pipe: ple_fail_closed writes one byte before os._exit and
the monitor starts the failure ladder at once; the fd stays registered
through worker teardown.
- worker_main marks inherited pipes close-on-exec.
- Any unexpected busy-loop failure exits 1; the termination ladder uses
time.monotonic(); forced-exit paths no longer wait on logging.
- Tests on real fork/spawn workers for rank 0 and rank 1, with and without an
fd-inheriting helper.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
- EngineCore survives a late SIGTERM during teardown (bounded deferral) so
every worker drains its guard telemetry.
- A worker exits right after its orderly shutdown.
- A worker failure racing shutdown stays a failure (exit 1).
- Rank 0 waits for the PLE child's whole thread group, bounded, instead of its
reap; drain-aware failure ladder with a bounded grace for survivors whose
shutdown already started.
- tests: the sorted-emit graph arm takes max_len on the host before capture.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Emits the selected indices in canonical ascending order inside the kernel so
the Python QSA sort can be skipped. Output-equivalent to the Python sort;
reference model, property tests and GPU arms included. Default off in this
commit.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
ple_fail_closed formatted its traceback with traceback.format_exception, which
runs exception-defined __str__/__getattribute__; if one blocked, the rank never
reached os._exit(1) and kept its GPUs. A safe formatter reads frames and plain
string arguments only.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The debug PLE row dump is deleted; ruff findings on fork lines cleared; a
read-site census test pins which fork knobs are read and where.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The pull-transport flag check is wired in (rank-gated, off by default, warns when enabled). Both
diagnostic checks log that they ran. Test repairs for helpers v0.30.0 renamed.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The fused pre-indexer could read a stale or uncommitted raw-key ring row and
pass its int64 tail to RoPE as a position (illegal address). The writer now
tags each row with logical_position + 1; the reader checks the tag of every
source row and, on a mismatch, masks the pooled load, the RoPE-tail read and
the compressed-slot store. New guard site 11 (always fatal under warn). The
KV pool size is unchanged.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Landed 0027 with mamba_utils.py and guard_telemetry.py under the
ours-newer rule; that rule saw the slot-ownership check and the
diagnostic gate, not the bias-encoding helpers. decode_slot(),
encode_slot(), GUARD_MAX_ENCODABLE and GUARD_MIN_ENCODABLE are new
names, present in v2.5.1 and absent here, and patch 0049's ring-row
report calls decode_slot(). Replays 0027's guard_telemetry.py section
(3 hunks, patch -F 8) onto the current tree.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
A schema walk fails on any pointer-like int parameter of a custom op on the
PLE path. The FULL_K test child calls the Marlin MoE GEMM by keyword against
its v0.30.0 signature.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Each worker logs its launch identity and, after warm-up and capture, one
e2-toggles line with the effective toggle state read back from the C++
extensions through two new tensor-free query ops. Both extensions must be
rebuilt.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Compile-time defines (default 0, device code unchanged by default) for the
delayed-publication stress, the pre-repair control and a forced-fault cell;
test cells for the OFF-arm controls, the QSA_SORT contract, FULL_K attribution
on the Marlin MoE GEMM and legacy-tail repeated-row membership.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
- Counter enum declared above the decode template (two-phase lookup).
- Emission-fault flag set atomically.
- Close the legacy large tie loop with a block barrier (inherited).
- Large-path fault injectors registered as hashed passthroughs.
- The tensor-free counter-slots op registered for CompositeExplicitAutograd.
- Tie measurements recorded only on rows neither fault verdict failed.
- Tests scoped to the properties the build promises.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
- MERLIN_FULL_K: deterministic Marlin MoE split-K (after unmerged upstream #46639).
- MERLIN_QSA_SORT: canonical order for the QSA selected blocks.
- MERLIN_TIE_DET: deterministic tie-break at the persistent_topk cutoff.
- MERLIN_TIE_RECENT: cutoff ties keep the highest indices.
- Decode candidate truncation is detected; one tie direction across the
medium path; the kernel switches are registered.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
- Inline-PTX acquire/release helpers declare their memory effects.
- The fault verdict is aggregated across the CTA group before emission, so a
faulted row is failed closed by every CTA.
- 64-bit device counters and an int64 reader (counter protocol v2).
- Tests: adversarial distributions, slot-residue coverage, an sm_90 gate for
tests that need it.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Above RADIX_THRESHOLD a row is split across a group of CTAs. The pivot was
already order-independent; cross-CTA placement, intra-CTA placement and tie
membership were arrival-order races. They are now deterministic behind
MERLIN_TOPK_DET_LARGE (default on).
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
In radix_topk's large-path round loop the leader published arrival with no
block barrier after the histogram merge, so a peer CTA could observe an
incomplete histogram. Inherited by earlier releases; fixed below the placement
work so both arms of any comparison carry it.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Reads persistent_topk_exact_counters into {slot: value}, logs one line on a
cadence, and degrades to a one-time warning when the op is absent.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The wait/copy op took host pointers as Python-int arguments, which were baked
into the AOT compile cache as the tracing process's addresses; the first
forward after a cache-load boot then failed. The op now carries only
process-stable values and resolves pointers from a per-process registry.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
A KV cache group with no layer on a pipeline stage was still given tensors
there, and allocate_kv_cache raised StopIteration on the second stage. The
fix was lost in the v0.30.0 rebase. Test drives the real get_kv_cache_configs
with the production layout.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
With the wait/copy inside a compiled piece, Inductor's post-grad
auto-functionalization decomposition failed on the stage-0 ranks.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
- torch.compile restored on the NVIDIA Qwen4Exp target and MTP so the
production profile (VLLM_COMPILE + FULL_AND_PIECEWISE, breakable off) runs.
- The production pin requires VLLM_USE_BREAKABLE_CUDAGRAPH to be exactly "0".
- assert_resolved_cudagraph_modes() runs at the top of capture_model.
- CPU tests for the combined profile, the exact pin and the resolved modes.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
- VLLM_E2_PLE_PULL_TRANSPORT replaces the fork's old switch name, which
v0.30.0 reuses for its UVA backend with the opposite default; strict parser;
compile/capture factor.
- The pull transport is reachable on v0.30.0 through a pull embedding class
that inherits upstream's without editing it; the upstream UVA backend is
provably off while the pull switch is on.
- resolve_ple_mode()/assert_ple_mode() log and assert the effective PLE mode
for target and drafter, and refuse a mixed configuration.
- Producer log lines for the PLE backend and weight dtype.
- CPU tests for mode resolution and the mixed-config refusal.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Env-gated diagnostics, inert unless set. Allocator telemetry is read only at
step boundaries, armed after warm-up and capture.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Restores the fork's PLE offload package (not upstream) and its pull
transport: the offload process holds no CUDA context; the captured op waits on
a device-mapped host flag and copies host->GPU on the model stream; publication
is fenced; releases are synchronous across dummy/capture work; failures close
the rank. Includes the test-only fault-injection cuts for the transport.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
New vllm/v1/e2_guards.py, called from the target and MTP constructors on every
rank before profiling or capture: the breakable-cudagraph pin is asserted
(three readings of the switch), a graph profile that cannot execute on v0.30.0
is refused, and VLLM_E2_EXPECTED_CUDAGRAPH_MODE is checked by name. Tests
cover the drafter. An inert knob is no longer hashed into the cache key.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
- Drop the PLE-offload half of ple_layer.py that upstream #54371 deleted and a
3-way merge resurrected (it no longer imported on v0.30.0).
- Drop the structured-output helper duplicate that v0.30.0 already provides,
and an unused import.
- Re-sort the import blocks the merges left unsorted (ruff I001).
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
- Every fork knob is registered in envs.py and classified as a compile/capture
factor or runtime-only; envs.py documents that capture identity and AOT
cache identity are different things.
- ruff findings cleared on lines this fork authored; upstream lines untouched.
- A `gpu` pytest marker so the GPU-free tier can be selected, not just skipped.
- Duplicate knob-table entries from two branches reconciled onto one
off|warn|raise normaliser.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
- Host-side Mamba slot-ownership check (off|warn|raise, default off) with a
failing control that proves it detects a wrong block-table row.
- Guard telemetry: int64 encoding limits and the extreme-value pair closed by
test; a decoded-None counter field is now named in the warning; a
captured-replay sentinel probe.
- Both knobs registered in envs.py.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
When the persistent top-k medium-path candidate buffer overflows, the row is
restarted exactly instead of being truncated. Capture-safe counters with atomic
publication and a coherent snapshot record every restart.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Ports upstream vLLM PRs #54442 and #56802 (issue #54437): never sample from an
unmasked row, and key draft-token hand-off snapshots by scheduler step so an
alternating PP microbatch cannot overwrite the slot.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Env-gated per-step log of blocks allocated and freed and scheduled tokens per
request; prefix-cache hits are read behind a schema check so an upstream field
rename fails visibly.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
A circular-buffer ring block could be freed while an earlier, still-executing
step was reading it. The free is deferred until the overlapping step has
committed.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Both callers of the guarded kernel (fused recurrent and KDA) are updated in the
same commit.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
MOVZXandHinata