[Bugfix][V1/V2] Fix prompt_logprobs to respect logprobs_mode (#47680)
Signed-off-by: Wojciech Wais <wojciech.wais@gmail.com> Signed-off-by: Federico Kamelhar <209537060+fede-kamel@users.noreply.github.com> Signed-off-by: Allen Shen <aoshen@inferact.ai> Co-authored-by: Wojciech Wais <wojciech.wais@gmail.com> Co-authored-by: Federico Kamelhar <209537060+fede-kamel@users.noreply.github.com>
This commit is contained in:
co-authored by
Wojciech Wais
Federico Kamelhar
parent
088c0be268
commit
41ea2dd44a
@@ -341,7 +341,10 @@ class GPUModelRunner(LoRAModelRunnerMixin):
|
||||
self.speculative_config,
|
||||
self.device,
|
||||
)
|
||||
self.prompt_logprobs_worker = PromptLogprobsWorker(self.max_num_reqs)
|
||||
self.prompt_logprobs_worker = PromptLogprobsWorker(
|
||||
self.max_num_reqs,
|
||||
logprobs_mode=self.model_config.logprobs_mode,
|
||||
)
|
||||
self.structured_outputs_worker = StructuredOutputsWorker(
|
||||
max_num_logits=self.max_num_reqs * self.decode_query_len,
|
||||
vocab_size=self.vocab_size,
|
||||
|
||||
Reference in New Issue
Block a user