[Bugfix][Spec Decode] Capture the widest uniform decode batch by default (#50488)
Signed-off-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com> Signed-off-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Signed-off-by: Rahul Chalamala <rahul@modal.com> Co-authored-by: Janelle Cai <janelle.cai@modal.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Devin <devin@cognition.ai>
This commit is contained in:
co-authored by
Janelle Cai
Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Devin
parent
a758a9f671
commit
d3d79ffc1e
@@ -703,7 +703,9 @@ class CompilationConfig:
|
||||
If not specified, max_cudagraph_capture_size is capped at 512 by default,
|
||||
or 1024 on data center Blackwell GPUs. This avoids OOM in tight memory
|
||||
scenarios with small max_num_seqs, and limits capture of large graphs that
|
||||
increase startup time and memory usage.
|
||||
increase startup time and memory usage. When the uniform decode query length
|
||||
exceeds one, uniform decode sizes are appended and can raise the final value
|
||||
above this default.
|
||||
"""
|
||||
|
||||
dynamic_shapes_config: DynamicShapesConfig = field(
|
||||
|
||||
Reference in New Issue
Block a user