[Bugfix] Zero stale is_prefilling in padded CUDA graph rows for Mamba (#41873)
Signed-off-by: Lanze Liu <lanzetech@gmail.com>
This commit is contained in:
@@ -2268,6 +2268,9 @@ class GPUModelRunner(
|
||||
# Used by mamba backends to distinguish actual decodes from
|
||||
# short extends.
|
||||
is_prefilling = num_computed_tokens_cpu < num_prompt_tokens_cpu
|
||||
# Zero out padded rows so stale data from condense() doesn't
|
||||
# misclassify padding as prefill in CUDA graph mode.
|
||||
is_prefilling[num_reqs:] = False
|
||||
|
||||
if self.use_async_spec_decode:
|
||||
# GPU tensors are authoritative in async mode.
|
||||
|
||||
Reference in New Issue
Block a user