[PCP] Support prefill context parallelism with data parallelism (#57075)
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com> Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: QiuChunshuo <qiuchunshuo@huawei.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com> Co-authored-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
QiuChunshuo
Claude Fable 5.1
Codex
Robert Shaw
Nick Hill
mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent
6697a7dd30
commit
e6dc16cebd
@@ -406,13 +406,14 @@ class Worker(WorkerBase):
|
||||
if dp_local_rank is None:
|
||||
dp_local_rank = self.parallel_config.data_parallel_index
|
||||
|
||||
tp_pp_world_size = (
|
||||
tp_pcp_pp_world_size = (
|
||||
self.parallel_config.pipeline_parallel_size
|
||||
* self.parallel_config.prefill_context_parallel_size
|
||||
* self.parallel_config.tensor_parallel_size
|
||||
)
|
||||
|
||||
# DP_LOCAL_RANK * TP_PP_WORLD_SIZE + TP_LOCAL_RANK
|
||||
self.local_rank += dp_local_rank * tp_pp_world_size
|
||||
# DP_LOCAL_RANK * TP_PCP_PP_WORLD_SIZE + TP_LOCAL_RANK
|
||||
self.local_rank += dp_local_rank * tp_pcp_pp_world_size
|
||||
|
||||
# Publish the logical-to-physical mapping for topology queries
|
||||
# such as NIC affinity and P2P checks.
|
||||
|
||||
Reference in New Issue
Block a user