[Bugfix][Spec Decode] Keep default CUDA graph sizes memory-safe (#54418)

Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
This commit is contained in:
Kevin H. Luu
2026-08-30 18:02:31 +00:00
committed by GitHub
parent 9d0fe9bac8
commit b2dc864bb6
3 changed files with 73 additions and 67 deletions
+2 -3
View File
@@ -703,9 +703,8 @@ class CompilationConfig:
If not specified, max_cudagraph_capture_size is capped at 512 by default,
or 1024 on data center Blackwell GPUs. This avoids OOM in tight memory
scenarios with small max_num_seqs, and limits capture of large graphs that
increase startup time and memory usage. When the uniform decode query length
exceeds one, uniform decode sizes are appended and can raise the final value
above this default.
increase startup time and memory usage. Uniform decode sizes are appended
only within this default ceiling.
"""
dynamic_shapes_config: DynamicShapesConfig = field(
+7 -8
View File
@@ -1953,10 +1953,9 @@ class VllmConfig:
`max_num_batched_tokens` is also appended to the list if it fits
within `max_cudagraph_capture_size`, so the max batch size is captured
even when off-stride. Likewise, when the uniform decode query length
exceeds one, the widest uniform decode batch (`max_num_seqs *
decode_query_len`) is appended when it fits, since it need not land on
an 8- or 16-token stride.
even when off-stride. Uniform decode sizes are appended when they fit
within the platform's default capture ceiling, since they need not land
on an 8- or 16-token stride.
In the end, `vllm_config.compilation_config.cudagraph_capture_sizes`
will be the final sizes to capture cudagraph (in ascending order).
@@ -2085,6 +2084,7 @@ class VllmConfig:
n * query_len
for query_len, tier_max_reqs in decode_tiers
for n in request_counts(tier_max_reqs)
if n * query_len <= max_cudagraph_capture_size
}
)
max_num_tokens = self.scheduler_config.max_num_batched_tokens
@@ -2134,10 +2134,9 @@ class VllmConfig:
and max_num_tokens not in cudagraph_capture_sizes
):
cudagraph_capture_sizes.append(max_num_tokens)
# These extend past the token-strided ceiling, which counts one
# token per request. valid_max_size below raises the final
# max_cudagraph_capture_size to the widest of them. They remain
# filtered by max_num_tokens.
# Preserve the platform's default capture ceiling. Larger
# uniform decode batches fall back to eager execution unless
# users explicitly configure wider capture sizes.
cudagraph_capture_sizes += [
size for size in uniform_decode_sizes if size <= max_num_tokens
]