[Bugfix] Fix ReplaySSM ring shape assertion when TP does not divide Mamba2 n_groups (#60704)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude
parent
e6e84f350c
commit
cd7517fb17
@@ -238,13 +238,14 @@ class MambaStateShapeCalculator:
|
||||
"""Append the physical ReplaySSM ring shapes.
|
||||
|
||||
``base_shapes[1]`` is ``(nheads // tp, head_dim, state_size)``;
|
||||
B_cache uses the un-extended ``n_groups``.
|
||||
B_cache extends ``n_groups`` for head shards as in ``mamba2_state_shape``.
|
||||
"""
|
||||
ring_buffer_len = logical_window
|
||||
if backend == MambaBackendEnum.FLASHINFER:
|
||||
# FlashInfer keeps the live window and current verify window together.
|
||||
ring_buffer_len += 1 + num_speculative_tokens
|
||||
local_nheads, head_dim, state_size = base_shapes[1]
|
||||
n_groups = n_groups + cls.extra_groups_for_head_shards(n_groups, tp_world_size)
|
||||
local_ngroups = divide(n_groups, tp_world_size)
|
||||
return (
|
||||
*base_shapes,
|
||||
|
||||
Reference in New Issue
Block a user