[PCP][Spec Decode] Adds PCP support for single-module MTP and replicated DSpark. (#56107)
Signed-off-by: Summer Yang <girasoleyang@gmail.com> Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: Summer Yang <girasoleyang@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: OpenAI Codex <noreply@openai.com>
This commit is contained in:
co-authored by
Summer Yang
OpenAI Codex
OpenAI Codex
parent
e7edf17cea
commit
980c16c8e4
@@ -401,9 +401,10 @@ def sparse_attn_indexer(
|
||||
# out-of-bounds reads in the kernel.
|
||||
# Keep PCP padding so every rank contributes the same all-gather shape.
|
||||
num_tokens = slot_mapping.shape[0]
|
||||
if use_pcp:
|
||||
num_tokens //= get_pcp_group().world_size
|
||||
if k is not None:
|
||||
# MTP draft decodes are replicated and use an unexpanded slot mapping.
|
||||
if use_pcp and num_tokens > k.shape[0]:
|
||||
num_tokens //= get_pcp_group().world_size
|
||||
k = k[:num_tokens]
|
||||
|
||||
if not skip_k_cache_insert:
|
||||
|
||||
Reference in New Issue
Block a user