[BugFix] fix speculative decoding memory leak when speculation is disabled (#15506)

Signed-off-by: Noah Yoshida <noahcy117@gmail.com>
This commit is contained in:
Noah Yoshida
2025-04-30 23:28:17 -07:00
committed by GitHub
parent 90d0a54c4d
commit 13cf6b6236
2 changed files with 91 additions and 0 deletions
+1
View File
@@ -695,6 +695,7 @@ class SpecDecodeWorker(LoRANotSupportedWorkerBase):
seq_group_meta_with_hidden):
self.previous_hidden_states.update(hidden_states,
seq_group_meta_with_hidden)
self.previous_hidden_states.prune(seq_group_meta_with_hidden)
if not skip_proposer:
# We prepare the prefill hidden states here so that there no