[Bugfix][V1/V2] Fix prompt_logprobs to respect logprobs_mode (#47680)

Signed-off-by: Wojciech Wais <wojciech.wais@gmail.com>
Signed-off-by: Federico Kamelhar <209537060+fede-kamel@users.noreply.github.com>
Signed-off-by: Allen Shen <aoshen@inferact.ai>
Co-authored-by: Wojciech Wais <wojciech.wais@gmail.com>
Co-authored-by: Federico Kamelhar <209537060+fede-kamel@users.noreply.github.com>
This commit is contained in:
aoshen02
2026-07-17 21:58:59 +01:00
committed by GitHub
co-authored by Wojciech Wais Federico Kamelhar
parent 088c0be268
commit 41ea2dd44a
11 changed files with 106 additions and 40 deletions
+8 -3
View File
@@ -5635,10 +5635,15 @@ class GPUModelRunner(
# to gather the logprob for.
tgt_token_ids = prompt_token_ids[start_tok : start_tok + num_logits]
# Compute prompt logprobs.
logprobs = self.sampler.compute_logprobs(logits)
# Compute prompt scores respecting logprobs_mode.
# NOTE: prompt tokens skip sampling processors, so
# processed_* and raw_* yield the same scores here.
if self.model_config.logprobs_mode in ("raw_logits", "processed_logits"):
scores = logits.to(torch.float32)
else:
scores = self.sampler.compute_logprobs(logits)
token_ids, logprobs, ranks, _ = self.sampler.gather_logprobs(
logprobs, num_prompt_logprobs, tgt_token_ids
scores, num_prompt_logprobs, tgt_token_ids
)
# Transfer GPU->CPU async.