Upstream e3cae8d2ac (#57387) deleted vllm/transformers_utils/configs/
qwen4_exp.py and moved the config family into the model package, so
that module path no longer resolves. transformers 5.18 exports the
config itself, which is how the rest of this fork imports it
(nvidia/model.py, amd/mtp.py, and nvidia/qsa.py at the base).
vllm/models/qwen4_exp/common/mtp.py carried the dead path at the base,
which made the Qwen4Exp model package fail to import at all; the PLE
pull transport and the FP8-KV reader added two more users of it.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
With a prefix unit finer than the block, a hybrid follow-up resumes from
a single GDN state at the previous prompt's tail, and that key sits in
the top attention block. Losing any one block of that prefix is a total
miss. The waiting-request pins only protect a hit that is still whole
when the follow-up is first looked up. Tracing the remaining misses
under a long-context multi-turn agent load shows they lost their top
attention block before that, in the gap between the previous turn
finishing and the next turn arriving. In that gap nothing referenced
the prefix. LRU then consumed it ahead of dead blocks freed later
(other conversations' handed-over CoW sources and superseded states).
A finished turn (stop, length, repetition, or an abort after decoding)
now keeps one extra ref on each of its hashed blocks, including its GDN
holds, as a grace pin. The ref is taken on both free paths, the
immediate free and the deferred pop_blocks_for_free behind an in-flight
step, so the prefix never passes through the free queue. A grace is
released when:
- the next turn is admitted on it. That is the grace holding the
deepest hit block of every group, with a hit that reaches its prompt
to within two prefix units, so a new turn sharing only a system
prompt releases nothing. The table takes the blocks over, with no
double ref.
- anyone needs blocks. Graces yield before waiter pins, oldest first and
one per failed allocation, so they never cause a preemption.
- the bound (16) is exceeded. The oldest grace goes.
- reset_prefix_cache runs.
A head waiter whose admission fails pins its own hit first. It then
yields graces and other pins one at a time, and only when everything
yieldable covers its shortfall. Otherwise yielding would drop other
conversations' prefixes and still not admit it. The shortfall is
measured after the pin: the pin takes the proof block past the hit (and
any hit block the lookup did not return) out of the free queue, so the
shortfall of the failed attempt can be short by those blocks and the
guard would yield another conversation's grace for nothing. Its own
pinned blocks outside the hit it is admitted on, the proof block past
it, go back first: once the hit is taken over they protect nothing, and
they cost nobody else anything. At an exact fit they are the blocks it
needs, and nothing else ever yields a head waiter's own pin, so without
this a request that needs the whole pool waited forever.
While any pin or grace is held the scheduler reports no common prefix
to cascade attention. Those refs can make a block's count equal the
number of running requests when only some of them share it, and cascade
attention would then read it for all of them.
Graces and waiter pins now follow a producer's GDN hash into its CoW
copy, as holds already do. Otherwise they would keep a second ref on a
running state block that its owner must own exclusively. Graces apply
only to fine-grained hybrid hits (coordinator.enable_partial_hash_hits)
and are off with a KV connector; a block-sized unit (the 4096 lane) is
unchanged. When graces are active the scheduler logs one startup line
saying so, with the bound, the hash unit and the block size.
Tests: the arrival-gap reproduction fails on the previous commit (hit 0
instead of 168, both free paths). It passes here. Further tests cover
the handover refcounts, yield order and bound, finish statuses, a
running request at an exactly full pool, yieldable accounting,
moved-hash following, the pin's extras contract, and six runs on the
real scheduler: among them a head waiter whose pin raises its shortfall
(it gives its proof block back and the one grace that covers the rest
yields), a request that needs the whole pool (admitted on its own proof
block; it waited forever before), and the common-prefix count under a
grace (none reported; the count returns once the pins are gone).
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The scheduler takes pop_blocks_for_free when a step may still be in
flight, for example when a preempted request is aborted before its
fence drained. A waiting request's pin is never written by any step, so
it is now released there at once. Before, it stayed until the next
yield or reset, keeping its blocks pinned and a waiter slot taken.
pin_waiting_hit touches each block in the pool it belongs to; private
pool groups route like free_blocks. Its docstring states the known gap:
a pin is a snapshot of the hit at pin time. It is not refreshed if the
hit grows while the request waits, or if a producer moves a pinned live
block's hash into its copy. The admission lookup still sees whatever is
cached, so the cost is a smaller hit, never a wrong one.
MAX_PINNED_WAITERS goes from 8 to 16.
Test: a pinned waiter popped through the deferred path leaves no pin,
no waiter slot and the pool at its baseline.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
A follow-up turn that waits for admission holds no reference on its
cached prefix. With a prefix unit finer than the block, its whole hit
depends on every one of those blocks: the resume key sits in the top
attention block, and the only GDN state is the held tail. So the first
of them that another request's allocation popped from the free queue
turned the next turn into a cold prefill; that was usually its own top
block, which is freed first. Under a long-context multi-turn agent load
(8 conversations, 80-100K-token prompts) this gave 4-5 total misses per
300 s at unit 64/512 and none at unit 4096. All of them happened while
the pool was at 88-99 % with one or two conversations waiting, and the
conversation that missed was usually the next victim too.
Now the manager can pin a waiter's hit:
- pin_waiting_hit looks the request up and takes a ref on every
non-null hit block in every group. It also pins the block holding the
EAGLE/MTP proof unit: one unit past the key, dropped from the returned
hit, and a different block when the key ends a block. The pinned
blocks stay out of the free queue.
- release_pinned_hit drops a pin; yield_pinned_hits drops the most
recently pinned waiters' first.
- allocate_slots releases the request's pin when it succeeds (its table
holds the hit from then on), free releases it (abort, finish), and
reset_prefix_cache yields every pin before resetting.
Pinned blocks are referenced, so admission and the watermark see them
as used.
The scheduler pins the front of the waiting queue after each scheduling
pass: arrival order, at most MAX_PINNED_WAITERS = 8, only requests that
have not computed anything yet, never with a KV connector. It does this
also while the running set is full, when a waiter is not even looked
up. A pin never costs anyone blocks. A running request that fails to
allocate yields one waiter's pin and retries before any preemption
victim is chosen. A head-of-line waiter whose admission fails yields the
pins of the waiters behind it and retries once. When pins are active
the scheduler logs one startup line saying so, with the bound, the hash
unit and the block size.
On a closed-loop CPU model of that load (6-8 conversations, 1-2
waiting, pool 88-100 %), total misses went from 3/1/1/5 to 2/0/0/0 with
every waiter pinned. The remaining two sit at 99.2 % use, where the
running set needed the pinned blocks themselves (two yields, 65
blocks). Unit 4096 is unaffected except where a waiter was losing its
boundary block.
Tests:
- A pinned waiter survives another follow-up's admission and a burst,
and resumes at its tail (a total miss without the pin).
- Pool full and a running request needs a block: the pin yields, nobody
is preempted, and the waiter is still admitted later.
- Pins are released on abort, admission, yield and reset without
leaks; the bound is respected; yield order is newest first.
- The scheduler's pass pins in arrival order up to the bound and skips
partly computed requests and connector setups.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
With a hash unit finer than the block, the prompt's tail key lands
exactly on a block end once every block/unit turns. The tail key is the
position the next turn resumes at: the last unit boundary, one unit
lower under the eagle drop. It is then a full-block key, so the
partial-tail registration does nothing, and under retention interval 0
the mask kept only the replay boundary one block below it. That state
was never hashed. The next turn therefore resumed one block lower after
a cold turn, or one turn lower (at the held source) after a partial-tail
resume, re-prefilling the previous turn's extension.
MambaManager.cache_blocks now adds that block-aligned tail to the
reachable boundaries, so the block ending there is hashed like any
boundary state, retired into the held checkpoint and freed last. At the
block unit the tail is the replay boundary itself and nothing changes.
When the hash unit is below the Mamba block, MambaManager logs one
startup line saying the block-aligned tail key is hashed.
Tests: a cold and a partial-tail-resume case whose tail lands on a block
end, under a concurrent burst, with the no-leak check. Both fail on the
previous commit (the tail state is not keyed and the next turn resumes
lower). A block-unit case asserts the hashed set is exactly the replay
boundary.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
With --prefix-match-unit below the block size, a follow-up turn that
resumes at a partial tail (mid-block) never hashes a block-boundary GDN
state. Under retention interval 0 the only boundary kept is the replay
boundary, which lies below the resume point. The request's own tail is
published on its running block and moved into a CoW block by the next
chunk, and the scheduler releases both copy endpoints once the copy has
run. That block then sat at ref 0 in the free queue for the rest of the
request, ahead of its attention blocks, and concurrent prefills popped
it. The hybrid hit is the shortest group's, so the next turn resumed at
0. Under a long-context multi-turn agent load, 7 of 61 follow-ups at
unit 64 (6 at 256/512/1024) came back with cached_tokens 0 on 80-100K
prompts whose whole attention prefix was still cached. Every one
followed a partial-tail resume, none followed a block-aligned one, and
unit 4096 never missed.
The existing retention (VLLM_E2_MAMBA_RETAIN_CHECKPOINT) only held
hashed blocks retired from the table, so it never covered that copy.
Now, under the same switch:
- The running-request CoW branch holds the moved tail block in
_retained_checkpoints with a ref of its own; the copy retention pair
is unchanged. The new-request branch holds the hit source after
_apply_cow for the same reason: a request that stops before its own
tail still leaves the state it resumed from.
- _retained_checkpoints is keyed by block_hash_num_tokens instead of
block index, so a sub-block tail key orders correctly against full
block states; _retire_state_blocks uses the same key. The newest
position supersedes and releases the older one, so a request still
holds at most one block per group.
- When a producer moves its tail hashes into its CoW block, another
request holding that source (a sibling that resumed at the tail while
it was the producer's live block) follows the hashes to the copy.
- Admission reserves one extra block per Mamba group for a partial-tail
hit: the held source and the moved tail coexist at the tail step.
The hold is released on every free path (finish, abort, preemption,
deferred pop_blocks_for_free) by the coordinator, last, after the
request's attention blocks, as before. When the hold is active (switch
on, hash unit below the Mamba block) MambaManager logs one startup line
saying so, with the hash unit and block size.
Tests (CPU): a replay of the miss that fails on v2.5.0 with the
retention switch on (next-turn hit 0 instead of the tail) and passes
here; the structural variant; admission accounting; no-leak checks for
a request freed with its copy still queued, in both release orders; and
the moved-hash follow. The retention suite's one block-index assertion
now reads the token position.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
RecoverSSM's verify needs the Triton decode kernel for its speculative rows, and the previous code met that by turning
off the fused CUDA decode for the whole layer. That also turned off the fused packed forward, which carries every
prefill chunk, so prefill ran the unfused path. The decode kernel stays Triton and the packed recurrent decode stays
off; the fused packed forward stays on. A small helper makes the choice, with CPU tests for each combination.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
CompilationConfig.resolve_cudagraph_mode_and_sizes read self.cache_config.use_replayssm, but CompilationConfig has
no cache_config, so with VLLM_USE_BREAKABLE_CUDAGRAPH=1 the fallback branch raised AttributeError. Both model
runners now pass cache_config.use_replayssm as an argument. The shipped launchers set VLLM_USE_BREAKABLE_CUDAGRAPH=0
and never reach this branch.
VLLM_FAIL_ON_PROFILING_KV_LEAK was read with a raw os.getenv, so setting it printed the unknown-variable warning. It
is now registered in vllm.envs with the same spellings (unset fails the boot; 0, false or no warn instead), listed
as an ignored compile factor and read through vllm.envs. Tests cover both.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
MERLIN_FULL_K, MERLIN_QSA_SORT, MERLIN_TIE_DET, MERLIN_TIE_RECENT and MERLIN_TOPK_SORTED_EMIT defaulted to off in
code while every shipped launcher set them to 1, so a launch without them ran a different kernel configuration.
vllm/env_override.py now sets each to 1 when unset; 0 still turns one off. tests/test_merlin_toggle_defaults.py
covers it.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
With the switch unset, the PLE took upstream's UVA backend, whose Triton lookup kernel needs fp8e4nv, which sm_86
does not have, so a launcher without the switch failed at profile_run. Unset now means on, the shipped value;
0 still selects upstream's path.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The packed pool strides every block by the widest group, and every group draws whole blocks. The PLE short-conv
state now shares a GDN cache group, and the QSA raw-key rings are packed as equal slots inside shared blocks, so
neither holds a whole block per request. VLLM_E2_SIDE_CACHE_LAYOUT (default 1) selects the old layout with 0.
On its own the layout raises the Flash-Next pool from 806,792 to 853,994 tokens at the same budget.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The admission gate reserved one retained-checkpoint block per Mamba group for every request not holding one. A
follow-up turn with a prefix hit already brings that block, so it was counted twice; the gate now counts it once.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
In Mamba align mode a state block leaves the block table once the request has computed one block past it. When
that block is hashed it is the request's prefix checkpoint, and releasing it let a long concurrent session evict
it, so the next turn re-prefilled the whole conversation. The manager now keeps the newest hashed checkpoint
referenced until the request finishes, and admission reserves that block. VLLM_E2_MAMBA_RETAIN_CHECKPOINT
(default 1) switches it off. Scheduler tests cover eviction, admission, preemption and freeing.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Repair: the port driver logged this patch as resolved but produced no
commit; the -3 merge dropped the coordinator code hunk. Reapplied to HEAD.
Co-Authored-By: Hinata <hinata@localhost>
Signed-off-by: MOVZX <movzx@yahoo.com>
On Qwen4Exp the flag selects RecoverSSM, which needs speculative decoding. With num_speculative_tokens=0 the
validator refused to start; it now logs a warning, clears use_replayssm (the no-flag path) and continues, so one
launch line serves with and without MTP. Other architectures keep the error. Config tests cover both.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The breakable-CUDA-graph fallback in the compilation config now applies only with --use-replayssm, and a
profiling KV cache that survives the teardown fails the boot instead of only being logged
(VLLM_FAIL_ON_PROFILING_KV_LEAK=0 turns it back into a warning).
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
_teardown_profiling_state measures the storage behind the profiling KV caches on entry and the allocated memory
before and after each release, and logs any profiling KV cache that is still alive afterwards.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Upstream RecoverSSM commits a step in the same step's postprocess, hence PP=1. Under pipeline parallelism a
non-last rank learns a step's accepted counts only when the PP broadcast arrives pp_size steps later, while later
builds overwrite the persistent commit inputs. The commit is now deferred with its own copy of those inputs, and
the PP=1 check is lifted for Qwen4Exp on Model Runner V2. Config and CPU tests cover the deferral.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Port of the upstream vLLM change #58863 (Apache-2.0). The verify kernel records the decay it applied and the commit replays the
accepted tokens forward with the same update, so the committed state equals the per-token state bit for bit.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Port of the upstream vLLM change #58863 (Apache-2.0). The final-state folds run before the boundary-state folds, because in align
mode a boundary state can be written into the slot a final fold still reads.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Port of the upstream vLLM change #58863 (Apache-2.0). The GDN and PLE RecoverSSM metadata builders declare UNIFORM_BATCH, so
--use-replayssm resolves FULL_AND_PIECEWISE; graph padding rows get a null state slot and write nothing.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Port of the upstream vLLM change #58863 (Apache-2.0). Each program folds either the final or the block-boundary state, not both,
which halves the live accumulator tiles.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Port of the upstream vLLM change #58863 (Apache-2.0). One check read use_gdn_recoverssm directly; it now uses getattr with a False
default like its neighbours, so layers built without __init__ in tests work.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Port of the upstream vLLM change #58863 (Apache-2.0). The step recorded during graph-capture dummy runs held the profiling KV cache
alive after profiling; it is now released at teardown.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Port of the upstream vLLM change #58863 (Apache-2.0). With --use-replayssm and speculative decoding, Gated DeltaNet layers verify the
draft window from a read-only checkpoint and keep a per-token replay record; after sampling one commit replays
the accepted tokens and writes the recurrent state once. The PLE short conv gets the same treatment.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The FP8 KV clip counter's reader thread used to start on the first eager forward, before CUDA graph capture, so
its device-to-host copies could overlap a capture. _qsa_clip_count now only allocates the per-layer counter; the
reader is started by a hook that guard_telemetry runs after warmup and capture (or at once if capture is already
done). Counting is unchanged. Tests cover the order, idempotence and a failing control that restores the old start.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
VLLM_E2_GUARD_MODE now defaults to warn in code, matching what the shipped launchers set; v2 through v2.2.0
defaulted to off. The comment above it, the MODES docstring, the vllm/envs.py default and its annotation follow.
tests/v1/worker/gpu/test_guard_default_mode.py checks that code, comment and envs.py agree, from the sources, so
it runs without CUDA, and fails on the earlier source.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
A fault on a non-zero TP rank could hang the engine: the worker's sentinel pipe
was inherited by a helper process and stayed open, so the monitor never saw
the exit. Earlier releases carry the same monitor and fail-closed exit path.
- Worker exit is detected by pidfd, with the sentinel as fallback, before and
after READY and for every worker through the whole initialisation window.
- A per-worker fault pipe: ple_fail_closed writes one byte before os._exit and
the monitor starts the failure ladder at once; the fd stays registered
through worker teardown.
- worker_main marks inherited pipes close-on-exec.
- Any unexpected busy-loop failure exits 1; the termination ladder uses
time.monotonic(); forced-exit paths no longer wait on logging.
- Tests on real fork/spawn workers for rank 0 and rank 1, with and without an
fd-inheriting helper.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
- EngineCore survives a late SIGTERM during teardown (bounded deferral) so
every worker drains its guard telemetry.
- A worker exits right after its orderly shutdown.
- A worker failure racing shutdown stays a failure (exit 1).
- Rank 0 waits for the PLE child's whole thread group, bounded, instead of its
reap; drain-aware failure ladder with a bounded grace for survivors whose
shutdown already started.
- tests: the sorted-emit graph arm takes max_len on the host before capture.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Emits the selected indices in canonical ascending order inside the kernel so
the Python QSA sort can be skipped. Output-equivalent to the Python sort;
reference model, property tests and GPU arms included. Default off in this
commit.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
ple_fail_closed formatted its traceback with traceback.format_exception, which
runs exception-defined __str__/__getattribute__; if one blocked, the rank never
reached os._exit(1) and kept its GPUs. A safe formatter reads frames and plain
string arguments only.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The debug PLE row dump is deleted; ruff findings on fork lines cleared; a
read-site census test pins which fork knobs are read and where.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The pull-transport flag check is wired in (rank-gated, off by default, warns when enabled). Both
diagnostic checks log that they ran. Test repairs for helpers v0.30.0 renamed.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The fused pre-indexer could read a stale or uncommitted raw-key ring row and
pass its int64 tail to RoPE as a position (illegal address). The writer now
tags each row with logical_position + 1; the reader checks the tag of every
source row and, on a mismatch, masks the pooled load, the RoPE-tail read and
the compressed-slot store. New guard site 11 (always fatal under warn). The
KV pool size is unchanged.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Landed 0027 with mamba_utils.py and guard_telemetry.py under the
ours-newer rule; that rule saw the slot-ownership check and the
diagnostic gate, not the bias-encoding helpers. decode_slot(),
encode_slot(), GUARD_MAX_ENCODABLE and GUARD_MIN_ENCODABLE are new
names, present in v2.5.1 and absent here, and patch 0049's ring-row
report calls decode_slot(). Replays 0027's guard_telemetry.py section
(3 hunks, patch -F 8) onto the current tree.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
A schema walk fails on any pointer-like int parameter of a custom op on the
PLE path. The FULL_K test child calls the Marlin MoE GEMM by keyword against
its v0.30.0 signature.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Each worker logs its launch identity and, after warm-up and capture, one
e2-toggles line with the effective toggle state read back from the C++
extensions through two new tensor-free query ops. Both extensions must be
rebuilt.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Compile-time defines (default 0, device code unchanged by default) for the
delayed-publication stress, the pre-repair control and a forced-fault cell;
test cells for the OFF-arm controls, the QSA_SORT contract, FULL_K attribution
on the Marlin MoE GEMM and legacy-tail repeated-row membership.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
- Counter enum declared above the decode template (two-phase lookup).
- Emission-fault flag set atomically.
- Close the legacy large tie loop with a block barrier (inherited).
- Large-path fault injectors registered as hashed passthroughs.
- The tensor-free counter-slots op registered for CompositeExplicitAutograd.
- Tie measurements recorded only on rows neither fault verdict failed.
- Tests scoped to the properties the build promises.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
- MERLIN_FULL_K: deterministic Marlin MoE split-K (after unmerged upstream #46639).
- MERLIN_QSA_SORT: canonical order for the QSA selected blocks.
- MERLIN_TIE_DET: deterministic tie-break at the persistent_topk cutoff.
- MERLIN_TIE_RECENT: cutoff ties keep the highest indices.
- Decode candidate truncation is detected; one tie direction across the
medium path; the kernel switches are registered.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
- Inline-PTX acquire/release helpers declare their memory effects.
- The fault verdict is aggregated across the CTA group before emission, so a
faulted row is failed closed by every CTA.
- 64-bit device counters and an int64 reader (counter protocol v2).
- Tests: adversarial distributions, slot-residue coverage, an sm_90 gate for
tests that need it.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Above RADIX_THRESHOLD a row is split across a group of CTAs. The pivot was
already order-independent; cross-CTA placement, intra-CTA placement and tie
membership were arrival-order races. They are now deterministic behind
MERLIN_TOPK_DET_LARGE (default on).
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
In radix_topk's large-path round loop the leader published arrival with no
block barrier after the histogram merge, so a peer CTA could observe an
incomplete histogram. Inherited by earlier releases; fixed below the placement
work so both arms of any comparison carry it.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Reads persistent_topk_exact_counters into {slot: value}, logs one line on a
cadence, and degrades to a one-time warning when the op is absent.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The wait/copy op took host pointers as Python-int arguments, which were baked
into the AOT compile cache as the tracing process's addresses; the first
forward after a cache-load boot then failed. The op now carries only
process-stable values and resolves pointers from a per-process registry.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
A KV cache group with no layer on a pipeline stage was still given tensors
there, and allocate_kv_cache raised StopIteration on the second stage. The
fix was lost in the v0.30.0 rebase. Test drives the real get_kv_cache_configs
with the production layout.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>