[Bugfix][HiSparse] Fix MTP acceptance collapse under FULL graphs with a saturated GPU pool (#59309)
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> Signed-off-by: Matthew Bonanni <mbonanni@redhat.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
Matthew Bonanni
parent
0cbac6cd13
commit
4056c8ac1f
@@ -21,6 +21,8 @@ class AttentionLayerBase(ABC):
|
||||
|
||||
impl: "AttentionImpl"
|
||||
supports_dcp: bool = True
|
||||
# Set by the speculator on its draft model's layers.
|
||||
is_draft_layer: bool = False
|
||||
|
||||
def bind_kv_cache(self, kv_cache: torch.Tensor) -> None:
|
||||
"""Bind the allocated KV cache tensor to this layer.
|
||||
|
||||
Reference in New Issue
Block a user