[Bugfix][V1/V2] Fix prompt_logprobs to respect logprobs_mode (#47680)
Signed-off-by: Wojciech Wais <wojciech.wais@gmail.com> Signed-off-by: Federico Kamelhar <209537060+fede-kamel@users.noreply.github.com> Signed-off-by: Allen Shen <aoshen@inferact.ai> Co-authored-by: Wojciech Wais <wojciech.wais@gmail.com> Co-authored-by: Federico Kamelhar <209537060+fede-kamel@users.noreply.github.com>
This commit is contained in:
co-authored by
Wojciech Wais
Federico Kamelhar
parent
088c0be268
commit
41ea2dd44a
@@ -5635,10 +5635,15 @@ class GPUModelRunner(
|
||||
# to gather the logprob for.
|
||||
tgt_token_ids = prompt_token_ids[start_tok : start_tok + num_logits]
|
||||
|
||||
# Compute prompt logprobs.
|
||||
logprobs = self.sampler.compute_logprobs(logits)
|
||||
# Compute prompt scores respecting logprobs_mode.
|
||||
# NOTE: prompt tokens skip sampling processors, so
|
||||
# processed_* and raw_* yield the same scores here.
|
||||
if self.model_config.logprobs_mode in ("raw_logits", "processed_logits"):
|
||||
scores = logits.to(torch.float32)
|
||||
else:
|
||||
scores = self.sampler.compute_logprobs(logits)
|
||||
token_ids, logprobs, ranks, _ = self.sampler.gather_logprobs(
|
||||
logprobs, num_prompt_logprobs, tgt_token_ids
|
||||
scores, num_prompt_logprobs, tgt_token_ids
|
||||
)
|
||||
|
||||
# Transfer GPU->CPU async.
|
||||
|
||||
Reference in New Issue
Block a user