[Frontend] Resolve logprobs on the parsed streaming derender path (#60437)

Signed-off-by: Shimi Bandiel <shimib@google.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
This commit is contained in:
Shimi Bandiel
2026-10-16 18:04:15 +08:00
committed by GitHub
co-authored by Claude wang.yuqi Kevin H. Luu
parent 9a16d43ef0
commit 024c5188bf
6 changed files with 404 additions and 20 deletions
+2 -2
View File
@@ -105,7 +105,7 @@ The server keeps no state between calls. Everything the next call needs is in `s
- **Send `prompt_token_ids` on the first chunk, even without a parser.** It seeds detokenization so the first token's leading space matches the coupled endpoint (see [Request format](#request-format)). Later chunks already carry that context in `stream_state`.
- **Don't forward `[DONE]`.** It marks the end of the generate stream and isn't a chunk.
`/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Generate returns every top-k candidate the engine produced; derender keeps the ones the coupled endpoint would, with the same helpers: the first `top_logprobs` on chat (all of them for `top_logprobs=-1`) and the first `logprobs + 1` on completions (all of them for `logprobs=-1`). It reads those values from `chat_request` / `completion_request`, so without the original request, or with one that does not ask for logprobs, it returns every candidate. Requests that use `logprob_token_ids` get no output logprobs from generate (their `sampling_params.logprobs` is unset), so derender has none to return for them either. Streaming derender does the same per chunk on the plain detokenization path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them.
`/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Generate returns every top-k candidate the engine produced; derender keeps the ones the coupled endpoint would, with the same helpers: the first `top_logprobs` on chat (all of them for `top_logprobs=-1`) and the first `logprobs + 1` on completions (all of them for `logprobs=-1`). It reads those values from `chat_request` / `completion_request`, so without the original request, or with one that does not ask for logprobs, it returns every candidate. Requests that use `logprob_token_ids` get no output logprobs from generate (their `sampling_params.logprobs` is unset), so derender has none to return for them either. Streaming derender does the same per chunk on both the plain detokenization path and the parser path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). As on `/v1/chat/completions`, a request with `include_reasoning: false` and a parser configured gets no logprobs at all, batch or streaming, since the decoded logprob tokens would reveal the hidden reasoning.
## Streaming cost
@@ -129,7 +129,7 @@ Streaming derender is stateless on the server side: all mutable state lives in t
- `logprob_context_token_ids`: the trailing sampled token IDs (at most 4) from previous chunks, used to seed byte-fallback (U+FFFD) correction so multi-byte characters whose tokens split across chunk boundaries still resolve to real strings
- `logprob_text_offset`: the cumulative emitted text length, so `text_offset` in completion streaming logprobs stays absolute across chunks instead of restarting at 0
When a streamed `GenerateTokensStreamChoice` carries `logprobs`, the integer `token_id`s are decoded per chunk and the resolved logprobs are attached to the corresponding streamed choice. For chat it is as `ChatCompletionLogProbs` and for completions it is converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`.
When a streamed `GenerateTokensStreamChoice` carries `logprobs`, the integer `token_id`s are decoded per chunk and the resolved logprobs are attached to the corresponding streamed choice. For chat it is as `ChatCompletionLogProbs` and for completions it is converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`. On the parser path the chunk's logprobs are attached to whatever delta the parser emits for that chunk (reasoning, content or a tool call delta, or an empty delta while the parser is still buffering markup), which is what the coupled endpoint does; replay never touches them, since each chunk's `logprobs` cover exactly that chunk's tokens.
## Example
@@ -1133,6 +1133,50 @@ async def test_e2e_parsed_reasoning(parser_client, parser_tokenizer):
assert "<think>" not in msg["content"]
@pytest.mark.asyncio
@pytest.mark.parametrize("include_reasoning", [True, False])
async def test_e2e_parsed_logprobs_follow_include_reasoning(
parser_client, parser_tokenizer, include_reasoning
):
"""With a parser configured, batch derender returns logprobs only when
reasoning is shown. Hidden reasoning drops them, as the coupled chat
endpoint does, because decoded logprob tokens would reveal it."""
messages = [{"role": "user", "content": "What is 2+3?"}]
gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages)
output_ids = _require_markers_survive(
parser_tokenizer, "<think>Two plus three.</think>Five.", "</think>"
)
generate_response = _e2e_generate_response(output_ids)
generate_response["choices"][0]["logprobs"] = {
"content": [
{"token_id": tid, "logprob": -0.5, "rank": 1, "top_logprobs": []}
for tid in output_ids
]
}
resp = await parser_client.post(
"/v1/chat/completions/derender",
json={
"model": PARSER_MODEL,
"generate_response": generate_response,
"prompt_tokens": len(gen_req["token_ids"]),
"chat_request": {
"model": PARSER_MODEL,
"messages": messages,
"include_reasoning": include_reasoning,
},
},
)
assert resp.status_code == 200, resp.text
choice = resp.json()["choices"][0]
assert choice["message"]["content"] == "Five."
if include_reasoning:
assert choice["message"]["reasoning"]
assert len(choice["logprobs"]["content"]) == len(output_ids)
else:
assert choice["message"]["reasoning"] is None
assert choice["logprobs"] is None
@pytest.mark.asyncio
async def test_e2e_parsed_tool_call(parser_client, parser_tokenizer):
"""<tool_call> extracted into tool_calls field."""
@@ -316,13 +316,18 @@ async def test_parity_logprobs(client):
async def _coupled_stream(
client: httpx.AsyncClient, messages: list[dict], **extra
) -> tuple[list[dict], list[int]]:
client: httpx.AsyncClient,
messages: list[dict],
require_token_ids: bool = True,
**extra,
) -> tuple[list[dict], list[int] | None]:
"""Stream `/v1/chat/completions` and return the choices that carry
`token_ids` along with the prompt token IDs.
The leading role only chunk has no `token_ids` and no counterpart on
`/inference/v1/generate`, so it is dropped.
`/inference/v1/generate`, so it is dropped. With `require_token_ids`
off every choice is kept and `prompt_token_ids` may be `None`, for
requests where the coupled server hides token ids (hidden reasoning).
"""
choices: list[dict] = []
prompt_token_ids: list[int] | None = None
@@ -351,9 +356,12 @@ async def _coupled_stream(
if chunk.get("prompt_token_ids") is not None:
prompt_token_ids = chunk["prompt_token_ids"]
choices.extend(
ch for ch in chunk["choices"] if ch.get("token_ids") is not None
ch
for ch in chunk["choices"]
if not require_token_ids or ch.get("token_ids") is not None
)
assert prompt_token_ids is not None
if require_token_ids:
assert prompt_token_ids is not None
return choices, prompt_token_ids
@@ -473,6 +481,151 @@ async def test_stream_parity_reasoning_and_tool_call(client, chunking):
pytest.skip("Model did not emit both a <think> block and a tool call")
# ---------------------------------------------------------------------------
# Streaming logprobs on the parser path
# ---------------------------------------------------------------------------
def _logprob_tokens(choice: dict) -> list[tuple[str, list[int] | None]] | None:
lp = choice.get("logprobs")
if lp is None or lp.get("content") is None:
return None
return [(e["token"], e.get("bytes")) for e in lp["content"]]
def _sampled_generate_logprobs(choice: dict) -> dict | None:
"""`GenerateLogProbs` for one coupled stream chunk from its `token_ids`
and per-token logprob values, without `top_logprobs`.
The coupled stream's `token_ids` are the integer ids a GPU-less worker
would have sent, so one stream is both the derender input and the string
reference. A second greedy run is not needed, and greedy decoding is not
bit-reproducible across calls on every platform anyway.
"""
lp = choice.get("logprobs")
if lp is None or lp.get("content") is None:
return None
content = lp["content"]
assert len(content) == len(choice["token_ids"]), choice
return {
"content": [
{"token_id": tid, "logprob": entry["logprob"], "top_logprobs": []}
for tid, entry in zip(choice["token_ids"], content)
]
}
async def _run_stream_logprob_parity_case(
client: httpx.AsyncClient, messages: list[dict], **extra
) -> tuple[dict, list[dict], list[dict]]:
"""Stream with logprobs through the coupled server, replay the same
chunks through the streaming derender endpoint and compare per chunk.
Returns the assembled coupled message, the coupled choices and the
derender choices.
"""
extra = {"logprobs": True, **extra}
coupled_choices, prompt_token_ids = await _coupled_stream(client, messages, **extra)
finish_reason = coupled_choices[-1]["finish_reason"]
derender_choices: list[dict] = []
disagg = await stream_chat_derender(
client,
[tid for ch in coupled_choices for tid in ch["token_ids"]],
[len(ch["token_ids"]) for ch in coupled_choices],
{"model": MODEL, "messages": messages, **extra},
len(prompt_token_ids),
prompt_token_ids,
finish_reason="stop" if finish_reason == "tool_calls" else finish_reason,
logprobs=[_sampled_generate_logprobs(ch) for ch in coupled_choices],
raw_choices=derender_choices,
)
coupled = assemble_stream(coupled_choices)
for message in (coupled, disagg):
for tc in message["tool_calls"]:
tc["id"] = _tool_call_id_shape(tc["id"])
assert disagg == coupled
assert len(derender_choices) == len(coupled_choices)
for ref, der in zip(coupled_choices, derender_choices):
assert _logprob_tokens(der) == _logprob_tokens(ref), (ref, der)
return coupled, coupled_choices, derender_choices
@pytest.mark.asyncio
async def test_stream_parity_logprobs_reasoning(client):
"""Per-chunk logprobs on the parser path match the coupled stream across
the reasoning -> content transition."""
messages = [{"role": "user", "content": "What is 17 times 23? Think it through."}]
coupled, _, derender_choices = await _run_stream_logprob_parity_case(
client, messages, include_reasoning=True, max_tokens=256
)
if not coupled["reasoning"]:
pytest.skip("Model did not emit a <think> block")
assert any(
ch["delta"].get("reasoning") and ch.get("logprobs") for ch in derender_choices
)
@pytest.mark.asyncio
async def test_stream_parity_logprobs_tool_call(client):
"""Chunks the parser swallows (tool call markup) still carry their
logprobs, on both sides, so no token's logprobs are lost."""
messages = [{"role": "user", "content": "What's the weather in Paris?"}]
coupled, _, derender_choices = await _run_stream_logprob_parity_case(
client,
messages,
tools=TOOLS,
tool_choice=FORCE_WEATHER_TOOL,
include_reasoning=True,
max_tokens=512,
)
if not coupled["tool_calls"]:
pytest.skip("Model did not emit a tool call")
swallowed = [
ch
for ch in derender_choices
if ch.get("logprobs")
and not ch["delta"].get("content")
and not ch["delta"].get("reasoning")
and not ch["delta"].get("tool_calls")
]
assert swallowed, "expected at least one buffered-markup chunk with logprobs"
@pytest.mark.asyncio
async def test_stream_parity_logprobs_hidden_reasoning(client):
"""With `include_reasoning=False` and a parser configured, neither side
returns logprobs on any chunk; derender still emits the content."""
messages = [{"role": "user", "content": "What is 17 times 23? Think it through."}]
extra = {"logprobs": True, "max_tokens": 256}
# Token ids come from a shown-reasoning run: the coupled server hides
# them along with the logprobs when reasoning is hidden.
shown_choices, prompt_token_ids = await _coupled_stream(
client, messages, include_reasoning=True, **extra
)
hidden_choices, _ = await _coupled_stream(
client, messages, require_token_ids=False, include_reasoning=False, **extra
)
assert all(ch.get("logprobs") is None for ch in hidden_choices)
assert assemble_stream(hidden_choices)["reasoning"] is None
finish_reason = shown_choices[-1]["finish_reason"]
derender_choices: list[dict] = []
disagg = await stream_chat_derender(
client,
[tid for ch in shown_choices for tid in ch["token_ids"]],
[len(ch["token_ids"]) for ch in shown_choices],
{"model": MODEL, "messages": messages, "include_reasoning": False, **extra},
len(prompt_token_ids),
prompt_token_ids,
finish_reason="stop" if finish_reason == "tool_calls" else finish_reason,
logprobs=[_sampled_generate_logprobs(ch) for ch in shown_choices],
raw_choices=derender_choices,
)
assert all(ch.get("logprobs") is None for ch in derender_choices)
shown = assemble_stream(shown_choices)
assert disagg["reasoning"] is None
assert disagg["content"] == shown["content"]
# ---------------------------------------------------------------------------
# Inline text level: generate(output_mode="text") == derender(generate(...))
# ---------------------------------------------------------------------------
@@ -1427,6 +1427,159 @@ class TestDerenderChatStreamParsed:
assert delta.reasoning is None
assert delta.content == "c"
@pytest.mark.asyncio
async def test_logprobs_attached_to_each_parsed_delta(
self, parsed_derenderer, tokenizer
):
"""A chunk's logprobs ride on whatever delta the parser emits for it
(reasoning, content or tool call), with token ids decoded, as on
`/v1/chat/completions`."""
chunks = [
[_FakeParser.REASON],
[_FakeParser.CONTENT],
[_FakeParser.TOOL_START, _FakeParser.TOOL_ARG],
]
state = None
seen = []
for ids in chunks:
chunk, state = await parsed_derenderer.derender_chat_stream(
model=MODEL_NAME,
generate_chunk=_make_stream_chunk(
ids, logprobs=_generate_logprobs(ids)
),
state=state,
chat_request=_chat_request(),
prompt_token_ids=[1, 2],
)
choice = chunk.choices[0]
assert choice.logprobs is not None and choice.logprobs.content is not None
assert [e.token for e in choice.logprobs.content] == (
convert_ids_list_to_tokens(tokenizer, ids)
)
seen.append(choice.delta)
assert seen[0].reasoning == "r"
assert seen[1].content == "c"
assert (
seen[2].tool_calls and seen[2].tool_calls[0].function.name == "get_weather"
)
@pytest.mark.asyncio
@pytest.mark.parametrize(
("request_kwargs", "expected_candidates"),
[({"logprobs": True, "top_logprobs": 2}, 2), ({}, 4)],
)
async def test_top_logprobs_follow_the_chat_cut(
self, parsed_derenderer, request_kwargs, expected_candidates
):
"""The parsed path trims generate's candidate list to the request's
`top_logprobs` like the plain path does, and keeps every candidate
when the request does not ask for logprobs."""
ids = [_FakeParser.CONTENT]
logprobs = _generate_logprobs(ids)
logprobs["content"][0]["top_logprobs"] = [
{"token_id": tid, "logprob": -float(rank)}
for rank, tid in enumerate([ids[0], 11, 12, 13])
]
chunk, _ = await parsed_derenderer.derender_chat_stream(
model=MODEL_NAME,
generate_chunk=_make_stream_chunk(ids, logprobs=logprobs),
chat_request=_chat_request(**request_kwargs),
prompt_token_ids=[1, 2],
)
content = chunk.choices[0].logprobs.content
assert len(content) == 1
assert len(content[0].top_logprobs) == expected_candidates
@pytest.mark.asyncio
async def test_logprobs_kept_when_parser_swallows_chunk(
self, parsed_derenderer, tokenizer
):
"""A chunk the parser emits nothing for (buffered markup) still
carries its logprobs on an empty delta, so no token's logprobs are
lost. Mirrors the coupled endpoint, which emits a chunk whenever
logprobs are present even if the delta is empty."""
swallowed = [_FakeParser.CONTENT + 1] # unknown to _FakeParser -> None
chunk, _ = await parsed_derenderer.derender_chat_stream(
model=MODEL_NAME,
generate_chunk=_make_stream_chunk(
swallowed, logprobs=_generate_logprobs(swallowed)
),
chat_request=_chat_request(),
prompt_token_ids=[1, 2],
)
choice = chunk.choices[0]
assert choice.delta.content is None and not choice.delta.tool_calls
assert [e.token for e in choice.logprobs.content] == (
convert_ids_list_to_tokens(tokenizer, swallowed)
)
@pytest.mark.asyncio
async def test_include_reasoning_false_drops_logprobs_on_every_chunk(
self, parsed_derenderer
):
"""Hidden reasoning suppresses logprobs on reasoning *and* content
chunks (decoded logprob tokens would leak the reasoning), the same
rule `/v1/chat/completions` applies with a parser configured."""
state = None
for ids in ([_FakeParser.REASON], [_FakeParser.CONTENT]):
chunk, state = await parsed_derenderer.derender_chat_stream(
model=MODEL_NAME,
generate_chunk=_make_stream_chunk(
ids, logprobs=_generate_logprobs(ids)
),
state=state,
chat_request=_chat_request(include_reasoning=False),
prompt_token_ids=[1, 2],
)
assert chunk.choices[0].logprobs is None
assert chunk.choices[0].delta.content == "c"
@pytest.mark.asyncio
async def test_logprobs_multibyte_across_chunks(self, parsed_derenderer, tokenizer):
"""Byte-fallback correction crosses chunk boundaries on the parser
path too, via the carried `logprob_context_token_ids`, which starts
empty like the engine's and never includes the prompt."""
token_ids = tokenizer.encode("👍", add_special_tokens=False)
if len(token_ids) < 2 or "�" not in tokenizer.decode([token_ids[-1]]):
pytest.skip("Tokenizer does not byte-split this character")
state = None
last_chunk = None
for tid in token_ids:
last_chunk, state = await parsed_derenderer.derender_chat_stream(
model=MODEL_NAME,
generate_chunk=_make_stream_chunk(
[tid], logprobs=_generate_logprobs([tid])
),
state=state,
chat_request=_chat_request(),
prompt_token_ids=[1, 2, 3, 4, 5],
)
assert last_chunk is not None and state is not None
final_entry = last_chunk.choices[0].logprobs.content[0]
assert not final_entry.token.endswith("�"), final_entry.token
assert state.logprob_context_token_ids == token_ids[-4:]
@pytest.mark.asyncio
async def test_logprob_context_advances_without_logprobs(self, parsed_derenderer):
"""The context tail tracks sampled ids whether or not the chunk
carries logprobs, so a later chunk that does can still be corrected."""
ids = [_FakeParser.REASON, _FakeParser.CONTENT, _FakeParser.CONTENT]
_, state = await parsed_derenderer.derender_chat_stream(
model=MODEL_NAME,
generate_chunk=_make_stream_chunk(ids),
chat_request=_chat_request(),
prompt_token_ids=[1, 2],
)
assert state.logprob_context_token_ids == ids
_, state = await parsed_derenderer.derender_chat_stream(
model=MODEL_NAME,
generate_chunk=_make_stream_chunk([], finish_reason="stop"),
state=state,
chat_request=_chat_request(),
prompt_token_ids=[1, 2],
)
assert state.logprob_context_token_ids == ids
# ---------------------------------------------------------------------------
# Harmony / GPT-OSS replay — unit, no server
@@ -52,6 +52,8 @@ async def stream_chat_derender(
prompt_token_ids: list[int],
finish_reason: str = "stop",
on_chunk: Callable[[list[dict]], None] | None = None,
logprobs: list[dict | None] | None = None,
raw_choices: list[dict] | None = None,
) -> dict:
"""Feed `output_ids` through the streaming chat derender endpoint in
the given `chunk_sizes`, threading `stream_state` across calls and
@@ -64,7 +66,13 @@ async def stream_chat_derender(
`tool_calls` assembled so far, letting callers assert properties of the
intermediate deltas (e.g. monotonic argument growth) rather than only
the final assembled result.
`logprobs` supplies one `GenerateLogProbs` payload (or `None`) per chunk
and `raw_choices`, if given, collects every derendered choice so callers
can compare per-chunk fields such as `logprobs`.
"""
if logprobs is not None:
assert len(logprobs) == len(chunk_sizes)
state = None
choices: list[dict] = []
pos = 0
@@ -84,6 +92,7 @@ async def stream_chat_derender(
"index": 0,
"token_ids": tids,
"finish_reason": finish_reason if is_last else None,
"logprobs": logprobs[i] if logprobs is not None else None,
}
],
},
@@ -101,6 +110,8 @@ async def stream_chat_derender(
assert data["chunk"]["choices"][0]["token_ids"] == tids
state = data["stream_state"]
choices.extend(data["chunk"]["choices"])
if raw_choices is not None:
raw_choices.extend(data["chunk"]["choices"])
if on_chunk is not None:
on_chunk(assemble_stream(choices)["tool_calls"])
+36 -13
View File
@@ -202,13 +202,21 @@ class OnlineDerenderer:
if has_parser and chat_request is not None
else {}
)
# Like the coupled chat endpoint, hide per-token metadata (logprobs,
# token ids) when a parser is configured and reasoning is hidden:
# either would otherwise leak the reasoning text.
hide_metadata = (
has_parser
and chat_request is not None
and not chat_request.include_reasoning
)
return_token_ids = (
chat_request is not None
and chat_request.return_token_ids
and (not has_parser or chat_request.include_reasoning)
and not hide_metadata
)
top_limit = _chat_top_logprobs_limit(chat_request)
for choice in generate_response.choices:
if not choice.token_ids:
raise ValueError(f"choice {choice.index} has empty or null token_ids")
@@ -219,7 +227,7 @@ class OnlineDerenderer:
tokenizer,
top_limit=top_limit,
)
if choice.logprobs is not None
if choice.logprobs is not None and not hide_metadata
else None
)
@@ -508,13 +516,6 @@ class OnlineDerenderer:
spaces_between_special_tokens=spaces_between,
)
# NOTE: parser-configured servers dispatch to
# _derender_chat_stream_parsed above and never reach this plain
# path. That parsed path does not resolve logprobs yet; when it
# does, mirror the generate chat streaming path, which suppresses
# logprobs entirely when a parser is configured and reasoning is
# hidden, because decoded logprob token text would leak hidden
# reasoning.
resolved_logprobs = None
if choice.logprobs is not None:
resolved_logprobs = _resolve_logprobs(
@@ -595,8 +596,17 @@ class OnlineDerenderer:
Text comes from a fresh incremental detokenizer with special tokens
preserved (``skip_special_tokens=False``), seeded from the prompt
tail on every call since replay starts from scratch.
Logprobs need no replay: a generate chunk's ``logprobs`` cover exactly
that chunk's tokens, so they are resolved once and attached to
whatever delta the parser emits for the chunk, as the coupled chat
endpoint does. Only the byte-fallback context crosses chunks, via
``state.logprob_context_token_ids``. With ``include_reasoning=False``
logprobs are dropped on every chunk, again like the coupled endpoint.
"""
tokenizer = self.renderer.get_tokenizer()
hide_metadata = not chat_request.include_reasoning
top_limit = _chat_top_logprobs_limit(chat_request)
parser = parser_cls(
tokenizer,
@@ -651,10 +661,9 @@ class OnlineDerenderer:
role_sent = state.role_sent
tools_streamed = state.tools_streamed
last_tool_call_ids = list(state.last_tool_call_ids)
logprob_context = list(state.logprob_context_token_ids)
return_token_ids = (
chat_request.return_token_ids and chat_request.include_reasoning
)
return_token_ids = chat_request.return_token_ids and not hide_metadata
# At most one choice: the caller (derender_chat_stream) already
# rejects >1 before dispatching here. role_sent/tools_streamed/
@@ -707,6 +716,18 @@ class OnlineDerenderer:
output_token_ids.extend(delta_tids)
output_chunk_lens.append(len(delta_tids))
resolved_logprobs = None
if choice.logprobs is not None and not hide_metadata:
resolved_logprobs = _resolve_logprobs(
choice.logprobs,
tokenizer,
initial_context_token_ids=state.logprob_context_token_ids,
top_limit=top_limit,
)
logprob_context = _logprob_context_tail(
state.logprob_context_token_ids, delta_tids
)
if delta_message is None:
delta_message = DeltaMessage()
@@ -735,6 +756,7 @@ class OnlineDerenderer:
stream_choice = ChatCompletionResponseStreamChoice(
index=choice.index,
delta=delta_message,
logprobs=resolved_logprobs,
finish_reason=finish_reason,
stop_reason=choice.stop_reason,
token_ids=choice.token_ids if return_token_ids else None,
@@ -750,6 +772,7 @@ class OnlineDerenderer:
"role_sent": role_sent,
"tools_streamed": tools_streamed,
"last_tool_call_ids": last_tool_call_ids,
"logprob_context_token_ids": logprob_context,
}
)