[Bugfix][Spec Decode] Cache the Mamba state at the block-grid position of EAGLE resume (#53945)

Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Signed-off-by: Adam Shaver <ashaver@nvidia.com>
Signed-off-by: akshaver <168006157+akshaver@users.noreply.github.com>
Co-authored-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: roikoren755 <26850796+roikoren755@users.noreply.github.com>
This commit is contained in:
akshaver
2026-09-08 09:59:58 -04:00
committed by GitHub
co-authored by wzhao18 Claude Opus 5 roikoren755
parent 414057a3d3
commit 263c4ff95f
12 changed files with 710 additions and 21 deletions
+7
View File
@@ -195,6 +195,12 @@ class CacheConfig:
when the token is at position i * block_size. This is the default when prefix
caching is enabled.
"""
enable_mamba_fine_grained_prefix_cache: bool = False
"""Also register a Mamba "align" checkpoint at the shared-prefix junction --
where an EAGLE/MTP sibling was observed to resume -- instead of only at the
prompt tail. Off by default; only takes effect with `mamba_cache_mode`
"align", EAGLE on the Mamba group, and a prefix match unit smaller than the
Mamba block size."""
replayssm_buffer_len: int = Field(default=16, gt=0)
"""ReplaySSM logical history length B for Mamba2. Triton uses B physical
rows and FlashInfer uses B+1. Kimi-K3 speculative decode does not use B.
@@ -274,6 +280,7 @@ class CacheConfig:
"prefix_cache_retention_interval",
# Prefix-caching implementation detail (doesn't affect compiled graph).
"prefix_match_unit",
"enable_mamba_fine_grained_prefix_cache",
"mamba_page_size_padded",
"skip_page_size_padded",
"user_specified_block_size",