[Spec-Decode] Support piecewise cudagraphs for Eagle head (#25109)

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
This commit is contained in:
Lucas Wilkinson
2025-10-10 01:20:31 -04:00
committed by GitHub
co-authored by Benjamin Chislett
parent da4455609d
commit 29255cfc3b
6 changed files with 84 additions and 16 deletions
+2 -1
View File
@@ -3441,7 +3441,8 @@ class GPUModelRunner(LoRAModelRunnerMixin, KVConnectorModelRunnerMixin):
if self.speculative_config and self.speculative_config.use_eagle():
assert isinstance(self.drafter, EagleProposer)
self.drafter.dummy_run(num_tokens)
use_cudagraphs = cudagraph_runtime_mode == CUDAGraphMode.PIECEWISE
self.drafter.dummy_run(num_tokens, use_cudagraphs=use_cudagraphs)
# This is necessary to avoid blocking DP.
# For dummy runs, we typically skip EPLB since we don't have any real