[Spec-Decode] Support piecewise cudagraphs for Eagle head (#25109)

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
This commit is contained in:
Lucas Wilkinson
2025-10-10 01:20:31 -04:00
committed by GitHub
co-authored by Benjamin Chislett
parent da4455609d
commit 29255cfc3b
6 changed files with 84 additions and 16 deletions
+6
View File
@@ -283,6 +283,12 @@ def set_forward_context(
vllm_config.parallel_config, num_tokens or 0, num_tokens_across_dp
)
# Convenience: if cudagraph is used and num_tokens is given, we can just
# create a batch descriptor here if not given (there's no harm since if it
# doesn't match in the wrapper it'll fall through).
if cudagraph_runtime_mode != CUDAGraphMode.NONE and num_tokens is not None:
batch_descriptor = batch_descriptor or BatchDescriptor(num_tokens=num_tokens)
forward_context = create_forward_context(
attn_metadata,
vllm_config,