Commit Graph
22567 Commits
Author SHA1 Message Date
MOVZXandHinata f5d9021745 [Scheduler] Release a waiter's pin on the deferred free path too
The scheduler takes pop_blocks_for_free when a step may still be in
flight, for example when a preempted request is aborted before its
fence drained. A waiting request's pin is never written by any step, so
it is now released there at once. Before, it stayed until the next
yield or reset, keeping its blocks pinned and a waiter slot taken.

pin_waiting_hit touches each block in the pool it belongs to; private
pool groups route like free_blocks. Its docstring states the known gap:
a pin is a snapshot of the hit at pin time. It is not refreshed if the
hit grows while the request waits, or if a producer moves a pinned live
block's hash into its copy. The admission lookup still sees whatever is
cached, so the cost is a smaller hit, never a wrong one.
MAX_PINNED_WAITERS goes from 8 to 16.

Test: a pinned waiter popped through the deferred path leaves no pin,
no waiter slot and the pool at its baseline.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:16:37 +07:00
MOVZXandHinata 633c758d4c [Scheduler] Pin a waiting request's prefix-cache hit until it is admitted
A follow-up turn that waits for admission holds no reference on its
cached prefix. With a prefix unit finer than the block, its whole hit
depends on every one of those blocks: the resume key sits in the top
attention block, and the only GDN state is the held tail. So the first
of them that another request's allocation popped from the free queue
turned the next turn into a cold prefill; that was usually its own top
block, which is freed first. Under a long-context multi-turn agent load
(8 conversations, 80-100K-token prompts) this gave 4-5 total misses per
300 s at unit 64/512 and none at unit 4096. All of them happened while
the pool was at 88-99 % with one or two conversations waiting, and the
conversation that missed was usually the next victim too.

Now the manager can pin a waiter's hit:

- pin_waiting_hit looks the request up and takes a ref on every
  non-null hit block in every group. It also pins the block holding the
  EAGLE/MTP proof unit: one unit past the key, dropped from the returned
  hit, and a different block when the key ends a block. The pinned
  blocks stay out of the free queue.
- release_pinned_hit drops a pin; yield_pinned_hits drops the most
  recently pinned waiters' first.
- allocate_slots releases the request's pin when it succeeds (its table
  holds the hit from then on), free releases it (abort, finish), and
  reset_prefix_cache yields every pin before resetting.

Pinned blocks are referenced, so admission and the watermark see them
as used.

The scheduler pins the front of the waiting queue after each scheduling
pass: arrival order, at most MAX_PINNED_WAITERS = 8, only requests that
have not computed anything yet, never with a KV connector. It does this
also while the running set is full, when a waiter is not even looked
up. A pin never costs anyone blocks. A running request that fails to
allocate yields one waiter's pin and retries before any preemption
victim is chosen. A head-of-line waiter whose admission fails yields the
pins of the waiters behind it and retries once. When pins are active
the scheduler logs one startup line saying so, with the bound, the hash
unit and the block size.

On a closed-loop CPU model of that load (6-8 conversations, 1-2
waiting, pool 88-100 %), total misses went from 3/1/1/5 to 2/0/0/0 with
every waiter pinned. The remaining two sit at 99.2 % use, where the
running set needed the pinned blocks themselves (two yields, 65
blocks). Unit 4096 is unaffected except where a waiter was losing its
boundary block.

Tests:
- A pinned waiter survives another follow-up's admission and a burst,
  and resumes at its tail (a total miss without the pin).
- Pool full and a running request needs a block: the pin yields, nobody
  is preempted, and the waiter is still admitted later.
- Pins are released on abort, admission, yield and reset without
  leaks; the bound is respected; yield order is newest first.
- The scheduler's pass pins in arrival order up to the bound and skips
  partly computed requests and connector setups.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:16:37 +07:00
MOVZXandHinata 95ba99ed29 [Mamba] Key the prompt tail when it lands on a block end
With a hash unit finer than the block, the prompt's tail key lands
exactly on a block end once every block/unit turns. The tail key is the
position the next turn resumes at: the last unit boundary, one unit
lower under the eagle drop. It is then a full-block key, so the
partial-tail registration does nothing, and under retention interval 0
the mask kept only the replay boundary one block below it. That state
was never hashed. The next turn therefore resumed one block lower after
a cold turn, or one turn lower (at the held source) after a partial-tail
resume, re-prefilling the previous turn's extension.

MambaManager.cache_blocks now adds that block-aligned tail to the
reachable boundaries, so the block ending there is hashed like any
boundary state, retired into the held checkpoint and freed last. At the
block unit the tail is the replay boundary itself and nothing changes.
When the hash unit is below the Mamba block, MambaManager logs one
startup line saying the block-aligned tail key is hashed.

Tests: a cold and a partial-tail-resume case whose tail lands on a block
end, under a concurrent burst, with the no-leak check. Both fail on the
previous commit (the tail state is not keyed and the next turn resumes
lower). A block-unit case asserts the hashed set is exactly the replay
boundary.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:16:37 +07:00
MOVZXandHinata 944007d9e5 [Mamba] Hold the partial-tail CoW checkpoint until the request is freed
With --prefix-match-unit below the block size, a follow-up turn that
resumes at a partial tail (mid-block) never hashes a block-boundary GDN
state. Under retention interval 0 the only boundary kept is the replay
boundary, which lies below the resume point. The request's own tail is
published on its running block and moved into a CoW block by the next
chunk, and the scheduler releases both copy endpoints once the copy has
run. That block then sat at ref 0 in the free queue for the rest of the
request, ahead of its attention blocks, and concurrent prefills popped
it. The hybrid hit is the shortest group's, so the next turn resumed at
0. Under a long-context multi-turn agent load, 7 of 61 follow-ups at
unit 64 (6 at 256/512/1024) came back with cached_tokens 0 on 80-100K
prompts whose whole attention prefix was still cached. Every one
followed a partial-tail resume, none followed a block-aligned one, and
unit 4096 never missed.

The existing retention (VLLM_E2_MAMBA_RETAIN_CHECKPOINT) only held
hashed blocks retired from the table, so it never covered that copy.
Now, under the same switch:

- The running-request CoW branch holds the moved tail block in
  _retained_checkpoints with a ref of its own; the copy retention pair
  is unchanged. The new-request branch holds the hit source after
  _apply_cow for the same reason: a request that stops before its own
  tail still leaves the state it resumed from.
- _retained_checkpoints is keyed by block_hash_num_tokens instead of
  block index, so a sub-block tail key orders correctly against full
  block states; _retire_state_blocks uses the same key. The newest
  position supersedes and releases the older one, so a request still
  holds at most one block per group.
- When a producer moves its tail hashes into its CoW block, another
  request holding that source (a sibling that resumed at the tail while
  it was the producer's live block) follows the hashes to the copy.
- Admission reserves one extra block per Mamba group for a partial-tail
  hit: the held source and the moved tail coexist at the tail step.

The hold is released on every free path (finish, abort, preemption,
deferred pop_blocks_for_free) by the coordinator, last, after the
request's attention blocks, as before. When the hold is active (switch
on, hash unit below the Mamba block) MambaManager logs one startup line
saying so, with the hash unit and block size.

Tests (CPU): a replay of the miss that fails on v2.5.0 with the
retention switch on (next-turn hit 0 instead of the tail) and passes
here; the structural variant; admission accounting; no-leak checks for
a request freed with its copy still queued, in both release orders; and
the moved-hash follow. The retention suite's one block-index assertion
now reads the token position.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:16:36 +07:00
MOVZXandHinata db2851e54a [GDN] Keep the fused packed forward under RecoverSSM
RecoverSSM's verify needs the Triton decode kernel for its speculative rows, and the previous code met that by turning
off the fused CUDA decode for the whole layer. That also turned off the fused packed forward, which carries every
prefill chunk, so prefill ran the unfused path. The decode kernel stays Triton and the packed recurrent decode stays
off; the fused packed forward stays on. A small helper makes the choice, with CPU tests for each combination.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:16:36 +07:00
MOVZXandHinata 69aed12cf8 config, envs: pass use_replayssm to the breakable-graph fallback; register VLLM_FAIL_ON_PROFILING_KV_LEAK
CompilationConfig.resolve_cudagraph_mode_and_sizes read self.cache_config.use_replayssm, but CompilationConfig has
no cache_config, so with VLLM_USE_BREAKABLE_CUDAGRAPH=1 the fallback branch raised AttributeError. Both model
runners now pass cache_config.use_replayssm as an argument. The shipped launchers set VLLM_USE_BREAKABLE_CUDAGRAPH=0
and never reach this branch.

VLLM_FAIL_ON_PROFILING_KV_LEAK was read with a raw os.getenv, so setting it printed the unknown-variable warning. It
is now registered in vllm.envs with the same spellings (unset fails the boot; 0, false or no warn instead), listed
as an ignored compile factor and read through vllm.envs. Tests cover both.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:15:12 +07:00
MOVZXandHinata 86d314050f [Config] The MERLIN_* kernel toggles default to the shipped value when unset
MERLIN_FULL_K, MERLIN_QSA_SORT, MERLIN_TIE_DET, MERLIN_TIE_RECENT and MERLIN_TOPK_SORTED_EMIT defaulted to off in
code while every shipped launcher set them to 1, so a launch without them ran a different kernel configuration.
vllm/env_override.py now sets each to 1 when unset; 0 still turns one off. tests/test_merlin_toggle_defaults.py
covers it.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:15:11 +07:00
MOVZXandHinata b5f9fdbbac [PLE] VLLM_E2_PLE_PULL_TRANSPORT defaults to on when unset
With the switch unset, the PLE took upstream's UVA backend, whose Triton lookup kernel needs fp8e4nv, which sm_86
does not have, so a launcher without the switch failed at profile_run. Unset now means on, the shipped value;
0 still selects upstream's path.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:15:11 +07:00
MOVZXandHinata 72485816d9 [KV] Side-cache layout -- fold the PLE state into a GDN group, pack QSA rings
The packed pool strides every block by the widest group, and every group draws whole blocks. The PLE short-conv
state now shares a GDN cache group, and the QSA raw-key rings are packed as equal slots inside shared blocks, so
neither holds a whole block per request. VLLM_E2_SIDE_CACHE_LAYOUT (default 1) selects the old layout with 0.
On its own the layout raises the Flash-Next pool from 806,792 to 853,994 tokens at the same budget.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:15:11 +07:00
MOVZXandHinata 21fb7de7bf [Scheduler] Don't double-count the checkpoint of a prefix-hit admission
The admission gate reserved one retained-checkpoint block per Mamba group for every request not holding one. A
follow-up turn with a prefix hit already brings that block, so it was counted twice; the gate now counts it once.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:13:06 +07:00
MOVZXandHinata 010521f10d [Scheduler] Keep the running request's Mamba prefix checkpoint referenced until it finishes
In Mamba align mode a state block leaves the block table once the request has computed one block past it. When
that block is hashed it is the request's prefix checkpoint, and releasing it let a long concurrent session evict
it, so the next turn re-prefilled the whole conversation. The manager now keeps the newest hashed checkpoint
referenced until the request finishes, and admission reserves that block. VLLM_E2_MAMBA_RETAIN_CHECKPOINT
(default 1) switches it off. Scheduler tests cover eviction, admission, preemption and freeing.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:13:06 +07:00
MOVZXandHinata 0f72060d66 [Scheduler] Preserve per-group eviction order in deferred block free (port of upstream #48532)
Repair: the port driver logged this patch as resolved but produced no
commit; the -3 merge dropped the coordinator code hunk. Reapplied to HEAD.

Co-Authored-By: Hinata <hinata@localhost>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:13:04 +07:00
MOVZXandHinata 7c0a916c5d recoverssm: --use-replayssm without MTP falls back to the stock GDN path
On Qwen4Exp the flag selects RecoverSSM, which needs speculative decoding. With num_speculative_tokens=0 the
validator refused to start; it now logs a warning, clears use_replayssm (the no-flag path) and continues, so one
launch line serves with and without MTP. Other architectures keep the error. Config tests cover both.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:09:17 +07:00
MOVZXandHinata 8c518d4f09 [Worker] Gate the graph fallback and fail on profiling leaks
The breakable-CUDA-graph fallback in the compilation config now applies only with --use-replayssm, and a
profiling KV cache that survives the teardown fails the boot instead of only being logged
(VLLM_FAIL_ON_PROFILING_KV_LEAK=0 turns it back into a warning).

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:57 +07:00
MOVZXandHinata 4cfd09ebda [Worker] Report a profiling KV cache that survives the teardown
_teardown_profiling_state measures the storage behind the profiling KV caches on entry and the allocated memory
before and after each release, and logs any profiling KV cache that is still alive afterwards.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:57 +07:00
MOVZXandHinata 812d662602 recoverssm: allow pipeline parallelism for Qwen4Exp on the V2 runner
Upstream RecoverSSM commits a step in the same step's postprocess, hence PP=1. Under pipeline parallelism a
non-last rank learns a step's accepted counts only when the PP broadcast arrives pp_size steps later, while later
builds overwrite the persistent commit inputs. The commit is now deferred with its own copy of those inputs, and
the PP=1 check is lifted for Qwen4Exp on Model Runner V2. Config and CPU tests cover the deferral.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:56 +07:00
MOVZXandHinata 8918d40b59 recoverssm: GDN: commit forward with the verify step's own decay, bit-identical to per-token (port of upstream #58863, 514102a1db)
Port of the upstream vLLM change #58863 (Apache-2.0). The verify kernel records the decay it applied and the commit replays the
accepted tokens forward with the same update, so the committed state equals the per-token state bit for bit.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:56 +07:00
MOVZXandHinata 7e9cd428a1 recoverssm: align commit: fold final states before boundary states (port of upstream #58863, 3388ba1a25)
Port of the upstream vLLM change #58863 (Apache-2.0). The final-state folds run before the boundary-state folds, because in align
mode a boundary state can be written into the slot a final fold still reads.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:56 +07:00
MOVZXandHinata e5e541446f recoverssm: FULL cudagraphs for uniform spec-decode batches (port of upstream #58863, 3aa3d5b586)
Port of the upstream vLLM change #58863 (Apache-2.0). The GDN and PLE RecoverSSM metadata builders declare UNIFORM_BATCH, so
--use-replayssm resolves FULL_AND_PIECEWISE; graph padding rows get a null state slot and write nothing.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:56 +07:00
MOVZXandHinata 2c6efaabe8 recoverssm: one fold per program in the GDN RecoverSSM state commit (port of upstream #58863, af0d2b45d7)
Port of the upstream vLLM change #58863 (Apache-2.0). Each program folds either the final or the block-boundary state, not both,
which halves the live accumulator tiles.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:55 +07:00
MOVZXandHinata eb68fb9105 recoverssm: read the GDN RecoverSSM flag defensively in _forward_core (port of upstream #58863, c1d2da8773)
Port of the upstream vLLM change #58863 (Apache-2.0). One check read use_gdn_recoverssm directly; it now uses getattr with a False
default like its neighbours, so layers built without __init__ in tests work.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:55 +07:00
MOVZXandHinata cb27582783 recoverssm: release the RecoverSSM step after CUDA graph memory profiling (port of upstream #58863, fdcc43cf3e)
Port of the upstream vLLM change #58863 (Apache-2.0). The step recorded during graph-capture dummy runs held the profiling KV cache
alive after profiling; it is now released at teardown.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:55 +07:00
MOVZXandHinata 24bcbc5087 recoverssm: RecoverSSM for Qwen GDN and the PLE short conv (port of upstream #58863, 03cae7837f)
Port of the upstream vLLM change #58863 (Apache-2.0). With --use-replayssm and speculative decoding, Gated DeltaNet layers verify the
draft window from a read-only checkpoint and keep a per-token replay record; after sampling one commit replays
the accepted tokens and writes the recurrent state once. The PLE short conv gets the same treatment.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:08:55 +07:00
MOVZXandHinata 5df639bb4e [QSA] Start the clip-counter reader after capture, not on the first forward
The FP8 KV clip counter's reader thread used to start on the first eager forward, before CUDA graph capture, so
its device-to-host copies could overlap a capture. _qsa_clip_count now only allocates the per-layer counter; the
reader is started by a hook that guard_telemetry runs after warmup and capture (or at once if capture is already
done). Counting is unchanged. Tests cover the order, idempotence and a failing control that restores the old start.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:05:28 +07:00
MOVZXandHinata ef55e095c3 [Guards] The bounds-guard telemetry code default is warn
VLLM_E2_GUARD_MODE now defaults to warn in code, matching what the shipped launchers set; v2 through v2.2.0
defaulted to off. The comment above it, the MODES docstring, the vllm/envs.py default and its annotation follow.
tests/v1/worker/gpu/test_guard_default_mode.py checks that code, comment and envs.py agree, from the sources, so
it runs without CUDA, and fails on the earlier source.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:04:39 +07:00
MOVZXandHinata 036608b012 [Engine] Detect a fail-closed fault or death on any rank
A fault on a non-zero TP rank could hang the engine: the worker's sentinel pipe
was inherited by a helper process and stayed open, so the monitor never saw
the exit. Earlier releases carry the same monitor and fail-closed exit path.

- Worker exit is detected by pidfd, with the sentinel as fallback, before and
  after READY and for every worker through the whole initialisation window.
- A per-worker fault pipe: ple_fail_closed writes one byte before os._exit and
  the monitor starts the failure ladder at once; the fd stays registered
  through worker teardown.
- worker_main marks inherited pipes close-on-exec.
- Any unexpected busy-loop failure exits 1; the termination ladder uses
  time.monotonic(); forced-exit paths no longer wait on logging.
- Tests on real fork/spawn workers for rank 0 and rank 1, with and without an
  fd-inheriting helper.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:04:37 +07:00
MOVZXandHinata 6c956ee3bd [Engine] Drain-order shutdown
- EngineCore survives a late SIGTERM during teardown (bounded deferral) so
  every worker drains its guard telemetry.
- A worker exits right after its orderly shutdown.
- A worker failure racing shutdown stays a failure (exit 1).
- Rank 0 waits for the PLE child's whole thread group, bounded, instead of its
  reap; drain-aware failure ladder with a bounded grace for survivors whose
  shutdown already started.
- tests: the sorted-emit graph arm takes max_len on the host before capture.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:04:37 +07:00
MOVZXandHinata eeea036c12 [TopK] In-kernel canonical emission (MERLIN_TOPK_SORTED_EMIT)
Emits the selected indices in canonical ascending order inside the kernel so
the Python QSA sort can be skipped. Output-equivalent to the Python sort;
reference model, property tests and GPU arms included. Default off in this
commit.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:04:37 +07:00
MOVZXandHinata a4aecf9ec7 [PLE] The dying-path traceback runs no exception code
ple_fail_closed formatted its traceback with traceback.format_exception, which
runs exception-defined __str__/__getattribute__; if one blocked, the rank never
reached os._exit(1) and kept its GPUs. A safe formatter reads frames and plain
string arguments only.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:03:19 +07:00
MOVZXandHinata 3575cdb126 [Guards] Draft identity and draft PLE guard report the drafter's own state
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:03:19 +07:00
MOVZXandHinata e3fcd7e3ae [Rebase] Row dump removed, lint, knob read-site census
The debug PLE row dump is deleted; ruff findings on fork lines cleared; a
read-site census test pins which fork knobs are read and where.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:01:43 +07:00
MOVZXandHinata 7f6055caa2 ple, tests: debug-gated pull-flag check; diagnostics report that they ran
The pull-transport flag check is wired in (rank-gated, off by default, warns when enabled). Both
diagnostic checks log that they ran. Test repairs for helpers v0.30.0 renamed.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:01:26 +07:00
MOVZXandHinata fb52250c11 [Scheduler] Drop the one-victim preemption guard that upstream #49675 made dead
The cascade test now expects zero needless victims.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:01:26 +07:00
MOVZXandHinata fba504d5bb [QSA] Tag every raw-key ring row and validate it before pooling
The fused pre-indexer could read a stale or uncommitted raw-key ring row and
pass its int64 tail to RoPE as a position (illegal address). The writer now
tags each row with logical_position + 1; the reader checks the tag of every
source row and, on a mismatch, masks the pooled load, the RoPE-tail read and
the compressed-slot store. New guard site 11 (always fatal under warn). The
KV pool size is unchanged.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 20:01:26 +07:00
MOVZXandHinata ad80cac06d [Guards] Repair: carry patch 0027's slot encode/decode helpers
Landed 0027 with mamba_utils.py and guard_telemetry.py under the
ours-newer rule; that rule saw the slot-ownership check and the
diagnostic gate, not the bias-encoding helpers. decode_slot(),
encode_slot(), GUARD_MAX_ENCODABLE and GUARD_MIN_ENCODABLE are new
names, present in v2.5.1 and absent here, and patch 0049's ring-row
report calls decode_slot(). Replays 0027's guard_telemetry.py section
(3 hunks, patch -F 8) onto the current tree.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:58:32 +07:00
MOVZXandHinata dec8be19de [PLE] The pull registry refuses unbound or clashing names; test fixes
A schema walk fails on any pointer-like int parameter of a custom op on the
PLE path. The FULL_K test child calls the Marlin MoE GEMM by keyword against
its v0.30.0 signature.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:56:01 +07:00
MOVZXandHinata 281491646d [Worker] E2-identity and rank-tagged e2-toggles startup lines
Each worker logs its launch identity and, after warm-up and capture, one
e2-toggles line with the effective toggle state read back from the C++
extensions through two new tensor-free query ops. Both extensions must be
rebuilt.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:56:00 +07:00
MOVZXandHinata bef464a69d [TopK] Diagnostic-only build defines and qualification test cells
Compile-time defines (default 0, device code unchanged by default) for the
delayed-publication stress, the pre-repair control and a forced-fault cell;
test cells for the OFF-arm controls, the QSA_SORT contract, FULL_K attribution
on the Marlin MoE GEMM and legacy-tail repeated-row membership.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:56:00 +07:00
MOVZXandHinata cff46af086 [TopK] Follow-up fixes to the persistent top-k kernels
- Counter enum declared above the decode template (two-phase lookup).
- Emission-fault flag set atomically.
- Close the legacy large tie loop with a block barrier (inherited).
- Large-path fault injectors registered as hashed passthroughs.
- The tensor-free counter-slots op registered for CompositeExplicitAutograd.
- Tie measurements recorded only on rows neither fault verdict failed.
- Tests scoped to the properties the build promises.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:56:00 +07:00
MOVZXandHinata 2918e3fda1 [TopK] Determinism toggles for greedy repeatability
- MERLIN_FULL_K: deterministic Marlin MoE split-K (after unmerged upstream #46639).
- MERLIN_QSA_SORT: canonical order for the QSA selected blocks.
- MERLIN_TIE_DET: deterministic tie-break at the persistent_topk cutoff.
- MERLIN_TIE_RECENT: cutoff ties keep the highest indices.
- Decode candidate truncation is detected; one tie direction across the
  medium path; the kernel switches are registered.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:56:00 +07:00
MOVZXandHinata 512ca2fc0b [TopK] Large-path hardening
- Inline-PTX acquire/release helpers declare their memory effects.
- The fault verdict is aggregated across the CTA group before emission, so a
  faulted row is failed closed by every CTA.
- 64-bit device counters and an int64 reader (counter protocol v2).
- Tests: adversarial distributions, slot-residue coverage, an sm_90 gate for
  tests that need it.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:59 +07:00
MOVZXandHinata 63e2d8be20 [TopK] Deterministic placement and tie membership for the large-path radix_topk
Above RADIX_THRESHOLD a row is split across a group of CTAs. The pivot was
already order-independent; cross-CTA placement, intra-CTA placement and tie
membership were arrival-order races. They are now deterministic behind
MERLIN_TOPK_DET_LARGE (default on).

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:59 +07:00
MOVZXandHinata daa56356c8 [TopK] Barrier the histogram round before the leader publishes arrival
In radix_topk's large-path round loop the leader published arrival with no
block barrier after the histogram merge, so a peer CTA could observe an
incomplete histogram. Inherited by earlier releases; fixed below the placement
work so both arms of any comparison carry it.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:59 +07:00
MOVZXandHinata 66c3d11016 [TopK] Python reader for the exact-restart counters
Reads persistent_topk_exact_counters into {slot: value}, logs one line on a
cadence, and degrades to a one-time warning when the op is absent.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:59 +07:00
MOVZXandHinata 7f4385a422 [PLE] The pull op resolves live pointers by layer name
The wait/copy op took host pointers as Python-int arguments, which were baked
into the AOT compile cache as the tracing process's addresses; the first
forward after a cache-load boot then failed. The op now carries only
process-stable values and resolves pointers from a per-process registry.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:23 +07:00
MOVZXandHinata 54d4ab1b9a [Guards] The draft guard reports the draft model's own architecture
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:23 +07:00
MOVZXandHinata 09ea3a66e9 [KV] Restore PP fix #54795 -- PP stages need only share a KV layout
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:23 +07:00
MOVZXandHinata e2eaefdec9 [KV] Restore PP fix #54793 -- no KV tensors for another stage's layers
A KV cache group with no layer on a pipeline stage was still given tensors
there, and allocate_kv_cache raised StopIteration on the second stage. The
fix was lost in the v0.30.0 rebase. Test drives the real get_kv_cache_configs
with the production layout.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:23 +07:00
MOVZXandHinata 6705330924 [PLE] The pull wait/copy op is a graph splitting op
With the wait/copy inside a compiled piece, Inductor's post-grad
auto-functionalization decomposition failed on the stage-0 ranks.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:22 +07:00
MOVZXandHinata 2e68aba2a7 [Qwen4Exp] Restore torch.compile on the NVIDIA path; exact breakable pin; runtime graph-mode assertion
- torch.compile restored on the NVIDIA Qwen4Exp target and MTP so the
  production profile (VLLM_COMPILE + FULL_AND_PIECEWISE, breakable off) runs.
- The production pin requires VLLM_USE_BREAKABLE_CUDAGRAPH to be exactly "0".
- assert_resolved_cudagraph_modes() runs at the top of capture_model.
- CPU tests for the combined profile, the exact pin and the resolved modes.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-09 19:55:22 +07:00