[Bugfix] Unify rank computation across regular decoding and speculative decoding (#7899)

This commit is contained in:
Jonas M. Kübler
2024-08-28 22:18:13 -07:00
committed by GitHub
parent ef99a78760
commit f205c09854
2 changed files with 22 additions and 3 deletions
+2 -2
View File
@@ -43,8 +43,8 @@ def get_sampled_token_logprobs(
sampled_token_ids, ]
expanded_selected_logprobs = selected_logprobs.unsqueeze(-1).expand(
-1, -1, vocab_size)
sampled_token_ids_ranks = (logprob_tensor >=
expanded_selected_logprobs).sum(-1)
sampled_token_ids_ranks = (logprob_tensor >
expanded_selected_logprobs).sum(-1).add_(1)
return sampled_token_ids_ranks, selected_logprobs