[Bugfix] Cap FULL decode cudagraph sizes for Mamba/hybrid models (#34094) (#34571)

Signed-off-by: haosdent <haosdent@gmail.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
This commit is contained in:
haosdent
2026-03-04 11:56:22 +01:00
committed by GitHub
co-authored by zjy0516
parent a8f66cbde8
commit d6e04f4c43
4 changed files with 230 additions and 0 deletions
+16
View File
@@ -5700,6 +5700,22 @@ class GPUModelRunner(
self.uniform_decode_query_len, self.parallel_config.tensor_parallel_size
)
# If the model has Mamba layers and cudagraph mode includes FULL
# decode, cap cudagraph capture sizes to the number of available
# Mamba cache blocks. Each decode request needs one conv_state
# cache line, so capture batch sizes cannot exceed num_blocks.
# Only FULL decode graphs are affected because PIECEWISE captures
# run GDN/Mamba ops eagerly (prefill path, no causal_conv1d_update).
# See: https://github.com/vllm-project/vllm/issues/34094
if cudagraph_mode.has_full_cudagraphs():
has_mamba = any(
isinstance(g.kv_cache_spec, MambaSpec) for g in kv_cache_groups
)
if has_mamba and self.kv_cache_config is not None:
self.compilation_config.adjust_cudagraph_sizes_for_mamba_cache(
self.kv_cache_config.num_blocks
)
# Trigger cudagraph dispatching keys initialization after
# resolved cudagraph mode.
self.compilation_config.cudagraph_mode = cudagraph_mode