[Bugfix][MLA] Size arange_buffer to max_num_batched_tokens to prevent CUDA IMA (#39277)
Signed-off-by: UranusSeven <109661872+UranusSeven@users.noreply.github.com> Signed-off-by: Woosuk Kwon <woosuk@inferact.ai> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
This commit is contained in:
co-authored by
Claude Opus 4.6
Woosuk Kwon
parent
6841f5dc77
commit
a966aaed30
@@ -303,7 +303,10 @@ class DeepseekV32IndexerMetadataBuilder(AttentionMetadataBuilder):
|
||||
device=self.device,
|
||||
)
|
||||
self.arange_buffer = torch.arange(
|
||||
scheduler_config.max_num_seqs * next_n,
|
||||
max(
|
||||
scheduler_config.max_num_seqs * next_n,
|
||||
scheduler_config.max_num_batched_tokens,
|
||||
),
|
||||
dtype=torch.int32,
|
||||
device=self.device,
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user