[Mamba] Add FlashInfer ReplaySSM support for MTP (#52928)
Signed-off-by: Andrii Skliar <askliar@nvidia.com> Co-authored-by: Andrii Skliar <askliar@nvidia.com>
This commit is contained in:
co-authored by
Andrii Skliar
parent
7e583e615c
commit
deb148e730
@@ -815,6 +815,7 @@ class NemotronHForCausalLM(
|
||||
tp_world_size=parallel_config.tensor_parallel_size,
|
||||
logical_window=cache_config.replayssm_buffer_len,
|
||||
backend=vllm_config.mamba_config.backend,
|
||||
num_speculative_tokens=vllm_config.num_speculative_tokens,
|
||||
)
|
||||
return base_shape
|
||||
|
||||
|
||||
Reference in New Issue
Block a user