[Bugfix][Frontend] Keep logprobs of parser-suppressed streaming chunks (#58583)
Signed-off-by: errmakov <ide404@gmail.com> Co-authored-by: Yanxiao Zhao <39199723+sdpkjc@users.noreply.github.com> Co-authored-by: Prakhar Agarwal <270064960+agarwalprakhar2511@users.noreply.github.com> Co-authored-by: Flora Feng <4florafeng@gmail.com>
This commit is contained in:
co-authored by
Yanxiao Zhao
Prakhar Agarwal
Flora Feng
parent
6491f481a7
commit
f339e0e757
@@ -693,11 +693,13 @@ class OpenAIServingChat(GenerateBaseServing):
|
||||
logprobs = None
|
||||
|
||||
if delta_message is None:
|
||||
# NOTE: If return_token_ids is enabled, we still need to
|
||||
# send a chunk with token_ids even if delta_message is None
|
||||
# NOTE: If return_token_ids or logprobs are enabled, we
|
||||
# still need to send a chunk even if delta_message is None
|
||||
# to ensure all tokens are included in the response
|
||||
if output.finish_reason is None and (
|
||||
not request.return_token_ids or hide_stream_metadata
|
||||
if (
|
||||
output.finish_reason is None
|
||||
and logprobs is None
|
||||
and (not request.return_token_ids or hide_stream_metadata)
|
||||
):
|
||||
continue
|
||||
delta_message = DeltaMessage()
|
||||
|
||||
Reference in New Issue
Block a user