[Perf] Skip detokenization in online beam search (#46422)
Signed-off-by: Guy Stone <guys@spotify.com> Signed-off-by: Guy Stone <guystone3@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -61,6 +61,7 @@ class BeamSearchOnlineMixin(ABC):
|
|||||||
logprobs=logprobs_num,
|
logprobs=logprobs_num,
|
||||||
max_tokens=1,
|
max_tokens=1,
|
||||||
temperature=temperature,
|
temperature=temperature,
|
||||||
|
detokenize=False,
|
||||||
)
|
)
|
||||||
all_beams = [
|
all_beams = [
|
||||||
BeamSearchSequence(
|
BeamSearchSequence(
|
||||||
|
|||||||
Reference in New Issue
Block a user