[Perf] Support minimum KV splits configuration for TokenSpeed MLA decode (#60345)
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com> Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
@@ -61,6 +61,11 @@ class AttentionConfig:
|
||||
flash_attn_max_num_splits_for_cuda_graph: int = 32
|
||||
"""Flash Attention max number splits for cuda graph decode."""
|
||||
|
||||
tokenspeed_mla_min_split_kv: int = Field(default=1, ge=1, le=256)
|
||||
"""Minimum KV splits for TokenSpeed MLA decode. The default of 1 preserves
|
||||
automatic scheduling. Larger values request a split floor and may increase
|
||||
workspace memory. Only applies to the TokenSpeed MLA backend."""
|
||||
|
||||
tq_max_kv_splits_for_cuda_graph: int = 32
|
||||
"""TurboQuant max NUM_KV_SPLITS for cuda graph decode.
|
||||
Fixes the split count so grid dimensions are constant across captures,
|
||||
|
||||
Reference in New Issue
Block a user