[Bugfix][Spec Decode] Keep default CUDA graph sizes memory-safe (#54418)
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
This commit is contained in:
@@ -703,9 +703,8 @@ class CompilationConfig:
|
||||
If not specified, max_cudagraph_capture_size is capped at 512 by default,
|
||||
or 1024 on data center Blackwell GPUs. This avoids OOM in tight memory
|
||||
scenarios with small max_num_seqs, and limits capture of large graphs that
|
||||
increase startup time and memory usage. When the uniform decode query length
|
||||
exceeds one, uniform decode sizes are appended and can raise the final value
|
||||
above this default.
|
||||
increase startup time and memory usage. Uniform decode sizes are appended
|
||||
only within this default ceiling.
|
||||
"""
|
||||
|
||||
dynamic_shapes_config: DynamicShapesConfig = field(
|
||||
|
||||
+7
-8
@@ -1953,10 +1953,9 @@ class VllmConfig:
|
||||
|
||||
`max_num_batched_tokens` is also appended to the list if it fits
|
||||
within `max_cudagraph_capture_size`, so the max batch size is captured
|
||||
even when off-stride. Likewise, when the uniform decode query length
|
||||
exceeds one, the widest uniform decode batch (`max_num_seqs *
|
||||
decode_query_len`) is appended when it fits, since it need not land on
|
||||
an 8- or 16-token stride.
|
||||
even when off-stride. Uniform decode sizes are appended when they fit
|
||||
within the platform's default capture ceiling, since they need not land
|
||||
on an 8- or 16-token stride.
|
||||
|
||||
In the end, `vllm_config.compilation_config.cudagraph_capture_sizes`
|
||||
will be the final sizes to capture cudagraph (in ascending order).
|
||||
@@ -2085,6 +2084,7 @@ class VllmConfig:
|
||||
n * query_len
|
||||
for query_len, tier_max_reqs in decode_tiers
|
||||
for n in request_counts(tier_max_reqs)
|
||||
if n * query_len <= max_cudagraph_capture_size
|
||||
}
|
||||
)
|
||||
max_num_tokens = self.scheduler_config.max_num_batched_tokens
|
||||
@@ -2134,10 +2134,9 @@ class VllmConfig:
|
||||
and max_num_tokens not in cudagraph_capture_sizes
|
||||
):
|
||||
cudagraph_capture_sizes.append(max_num_tokens)
|
||||
# These extend past the token-strided ceiling, which counts one
|
||||
# token per request. valid_max_size below raises the final
|
||||
# max_cudagraph_capture_size to the widest of them. They remain
|
||||
# filtered by max_num_tokens.
|
||||
# Preserve the platform's default capture ceiling. Larger
|
||||
# uniform decode batches fall back to eager execution unless
|
||||
# users explicitly configure wider capture sizes.
|
||||
cudagraph_capture_sizes += [
|
||||
size for size in uniform_decode_sizes if size <= max_num_tokens
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user