[Perf] Support minimum KV splits configuration for TokenSpeed MLA decode (#60345)

Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
Wei Zhao
2026-10-09 23:33:47 +00:00
committed by GitHub
co-authored by Codex
parent 04b02fc8a5
commit 03a7d3776f
4 changed files with 83 additions and 22 deletions
+5
View File
@@ -61,6 +61,11 @@ class AttentionConfig:
flash_attn_max_num_splits_for_cuda_graph: int = 32
"""Flash Attention max number splits for cuda graph decode."""
tokenspeed_mla_min_split_kv: int = Field(default=1, ge=1, le=256)
"""Minimum KV splits for TokenSpeed MLA decode. The default of 1 preserves
automatic scheduling. Larger values request a split floor and may increase
workspace memory. Only applies to the TokenSpeed MLA backend."""
tq_max_kv_splits_for_cuda_graph: int = 32
"""TurboQuant max NUM_KV_SPLITS for cuda graph decode.
Fixes the split count so grid dimensions are constant across captures,