[Feat] Support PP with PCP in GPU Model Runner V2 (#59139)
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
This commit is contained in:
co-authored by
Lucas Wilkinson
parent
2e52a558f0
commit
98b45c1a94
@@ -2064,6 +2064,8 @@ class GPUModelRunner(LoRAModelRunnerMixin):
|
||||
# Non-last PP rank: hidden_states is None because this rank produced
|
||||
# IntermediateTensors instead of final hidden states. Receive the
|
||||
# sampled tokens broadcast from the last rank and update local state.
|
||||
if self.pcp_manager is not None:
|
||||
input_batch = self.pcp_manager.global_batch
|
||||
assert self.pp_handler is not None
|
||||
all_decode_next = self.pp_handler.receive(input_batch)
|
||||
# Optimistically update num_computed_tokens for entire batch here.
|
||||
|
||||
Reference in New Issue
Block a user