[Spec Decode][Qwen4Exp] RecoverSSM for Qwen GDN and the PLE short conv
With --use-replayssm and speculative decoding, Qwen4Exp's Gated DeltaNet layers verify the draft window from a read-only checkpoint and keep a per-token replay record [correction(V), normalized key(K), gate(1)] in fp32. After sampling, one commit replays the accepted tokens and writes the recurrent state once (plus the block-boundary state in align mode), instead of snapshotting the state after every draft position. The PLE dilated short conv follows the same protocol (accepted-suffix compaction). num_speculative_blocks = 0, so no per-draft Mamba blocks are reserved. - GDN verify/commit kernels: model_executor/layers/mamba/gdn/recoverssm_gdn.py - metadata/backends: v1/attention/backends/gdn_recoverssm.py, ple_recoverssm.py - the verify reads strided q/k/v views of the conv output and writes in place - builders declare AttentionCGSupport.NEVER: FULL_AND_PIECEWISE falls back to PIECEWISE; the fallback now also counts breakable CUDA graphs (empty splitting_ops, with or without torch.compile) as piecewise attention instead of dropping to NONE - the generic Mamba metadata builder skips the Mamba2 ReplaySSM ring-buffer workspaces when --use-replayssm selects RecoverSSM (the PLE short conv uses that builder) - tests: GDN verify/commit vs the native kernels (fp32 and bf16 state, align boundaries, mixed and ragged batches, extreme gates), PLE vs token-by-token decode, config selection Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -1424,10 +1424,15 @@ class CompilationConfig:
|
||||
f"with {min_cg_attn_backend} backend (support: "
|
||||
f"{min_cg_support})"
|
||||
)
|
||||
if self.mode == CompilationMode.VLLM_COMPILE and (
|
||||
self.splitting_ops_contain_attention()
|
||||
or self.use_inductor_graph_partition
|
||||
):
|
||||
if (
|
||||
self.mode == CompilationMode.VLLM_COMPILE
|
||||
and (
|
||||
self.splitting_ops_contain_attention()
|
||||
or self.use_inductor_graph_partition
|
||||
)
|
||||
) or envs.VLLM_USE_BREAKABLE_CUDAGRAPH:
|
||||
# Breakable CUDA graphs break at the attention kernels at runtime,
|
||||
# with or without torch.compile.
|
||||
msg += (
|
||||
"; setting cudagraph_mode=PIECEWISE because "
|
||||
"attention is compiled piecewise"
|
||||
|
||||
Reference in New Issue
Block a user