[Feature][Spec] Support disabling trailing prefix-cache block dropping (#53388)

Signed-off-by: Ziming Huang <zelda.huanghuang@gmail.com>
Signed-off-by: Ziming Huang <48115868+ZeldaHuang@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
This commit is contained in:
Ziming Huang
2026-09-01 19:12:36 +08:00
committed by GitHub
co-authored by OpenAI Codex
parent 9e905f7450
commit 481839ad9e
13 changed files with 112 additions and 21 deletions
+9
View File
@@ -436,6 +436,11 @@ class SpeculativeConfig:
speculative input batches can contain sequences of different lengths,
which may only be supported by certain attention backends. This currently
only affects the EAGLE method of speculation."""
disable_eagle_block_drop: bool = False
"""Disable dropping the trailing prefix-cache block for EAGLE-like
speculative methods. This is an experimental option for measuring the
acceptance-rate impact of reusing that block. It does not disable the
speculative drafter itself."""
use_local_argmax_reduction: bool = False
"""Use vocab-parallel local argmax instead of all-gathering full logits
for draft token generation. Reduces communication from O(vocab_size) to
@@ -1846,6 +1851,10 @@ class SpeculativeConfig:
# TODO(ben): Refactor this so the naming is clearer
return self.method in ("eagle", "eagle3", "mtp", "dflash", "dspark")
def use_eagle_block_drop(self) -> bool:
"""Whether volatile trailing cache blocks should be discarded."""
return self.use_eagle() and not self.disable_eagle_block_drop
def use_dflash(self) -> bool:
return self.method == "dflash"