[Perf][Kernel] Tune cooperative topk for medium batch-sizes (#53382)
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com> Co-authored-by: Codex <codex@openai.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
Codex
mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent
85c1365bd9
commit
f5c3cc240b
@@ -617,7 +617,7 @@ def sparse_attn_indexer(
|
||||
use_cooperative_topk = (
|
||||
current_platform.is_cuda()
|
||||
and topk_tokens in (512, 1024, 2048)
|
||||
and num_rows <= 32
|
||||
and num_rows <= 64
|
||||
and logits.stride(0) % 4 == 0 # TMA 16-byte alignment
|
||||
and current_platform.has_device_capability(90)
|
||||
and not current_platform.is_device_capability_family(120)
|
||||
|
||||
Reference in New Issue
Block a user