[Bugfix][MLA] Size arange_buffer to max_num_batched_tokens to prevent CUDA IMA (#39277)

Signed-off-by: UranusSeven <109661872+UranusSeven@users.noreply.github.com>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
This commit is contained in:
Uranus
2026-04-29 16:14:50 -07:00
committed by GitHub
co-authored by Claude Opus 4.6 Woosuk Kwon
parent 6841f5dc77
commit a966aaed30
+4 -1
View File
@@ -303,7 +303,10 @@ class DeepseekV32IndexerMetadataBuilder(AttentionMetadataBuilder):
device=self.device,
)
self.arange_buffer = torch.arange(
scheduler_config.max_num_seqs * next_n,
max(
scheduler_config.max_num_seqs * next_n,
scheduler_config.max_num_batched_tokens,
),
dtype=torch.int32,
device=self.device,
)