[Bugfix][Spec Decode] Keep default CUDA graph sizes memory-safe (#54418)

Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
This commit is contained in:
Kevin H. Luu
2026-08-30 18:02:31 +00:00
committed by GitHub
parent 9d0fe9bac8
commit b2dc864bb6
3 changed files with 73 additions and 67 deletions
+2 -3
View File
@@ -703,9 +703,8 @@ class CompilationConfig:
If not specified, max_cudagraph_capture_size is capped at 512 by default,
or 1024 on data center Blackwell GPUs. This avoids OOM in tight memory
scenarios with small max_num_seqs, and limits capture of large graphs that
increase startup time and memory usage. When the uniform decode query length
exceeds one, uniform decode sizes are appended and can raise the final value
above this default.
increase startup time and memory usage. Uniform decode sizes are appended
only within this default ceiling.
"""
dynamic_shapes_config: DynamicShapesConfig = field(