[Frontend] Add stream reasoning and tool calls from the derender endpoint (#50550)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
This commit is contained in:
@@ -6,7 +6,7 @@ This closes the loop for a token-in / token-out engine in disaggregated serving:
|
|||||||
|
|
||||||
- **GPU less post processing**: Detokenization, reasoning parsing, and tool call parsing run on the same GPU less frontend that hosts `/render`
|
- **GPU less post processing**: Detokenization, reasoning parsing, and tool call parsing run on the same GPU less frontend that hosts `/render`
|
||||||
- **Parser parity**: The derenderer reuses vLLM's tool and reasoning parsers, so a disaggregated deployment produces the same `content`/`reasoning`/ `tool_calls` split as a standard `vllm serve` server
|
- **Parser parity**: The derenderer reuses vLLM's tool and reasoning parsers, so a disaggregated deployment produces the same `content`/`reasoning`/ `tool_calls` split as a standard `vllm serve` server
|
||||||
- **Non-streaming**: The endpoints expect a complete `GenerateResponse` with all token IDs present and perform one-shot parsing. Streaming derender would require a separate endpoint design and is not currently supported but is in the pipeline
|
- **Streaming and one-shot parsing**: Non-streaming calls send a complete `GenerateResponse` with all token IDs present and get one-shot parsing. Both endpoints also accept `stream: true`, taking one `GenerateStreamResponse` delta plus a client carried `stream_state` and returning `{chunk, stream_state}`. The chat endpoint's streaming path supports reasoning and tool call parsing while emitting the same `reasoning`/`content`/`tool_calls` deltas the generate streaming path would for the same token IDs
|
||||||
|
|
||||||
Both endpoints are hosted by the GPU less rendering server started with [`vllm launch render`](../../cli/launch/render.md), alongside the `/render`
|
Both endpoints are hosted by the GPU less rendering server started with [`vllm launch render`](../../cli/launch/render.md), alongside the `/render`
|
||||||
endpoints.
|
endpoints.
|
||||||
@@ -21,7 +21,7 @@ endpoints.
|
|||||||
└─────────────── request + prompt_tokens ──┘
|
└─────────────── request + prompt_tokens ──┘
|
||||||
```
|
```
|
||||||
|
|
||||||
The derender step needs more than the engine's `token_ids`. It also consumes the original `chat_request`/`completion_request` and `prompt_tokens` carried over from the render step (see [Request format](#request-format)) so the tool and reasoning parsers have the context they need.
|
The derender step needs more than the engine's `token_ids`. It also consumes the original `chat_request`/`completion_request` and `prompt_tokens` carried over from the render step (see [Request format](#request-format)) so the tool and reasoning parsers have the context they need. The chat streaming path additionally needs `prompt_token_ids` (see [Streaming cost](#streaming-cost)) when a parser is configured.
|
||||||
|
|
||||||
## API Reference
|
## API Reference
|
||||||
|
|
||||||
@@ -52,6 +52,21 @@ Each request wraps the engine's `GenerateResponse`(s) together with the caller m
|
|||||||
|
|
||||||
Oversized payloads are rejected with a `400` before any `tokenizer.decode()` or parser runs.
|
Oversized payloads are rejected with a `400` before any `tokenizer.decode()` or parser runs.
|
||||||
|
|
||||||
|
## Streaming cost
|
||||||
|
|
||||||
|
Streaming derender threads a client carried `stream_state` across per-chunk calls instead of keeping session state on the server.
|
||||||
|
|
||||||
|
Plain detokenization (no parser configured) carries only a small, bounded incremental decode window in `stream_state` independent of generation length.
|
||||||
|
|
||||||
|
When a tool or reasoning parser is configured, parser internal state (buffered markup, reasoning/tool phase) can't be serialized. `stream_state` therefore instead carries the full `output_token_ids` seen so far plus `output_chunk_lens`, the token count of each chunk they arrived in. Each chunk rebuilds a fresh parser and replays that history through `parse_delta`, one call per original chunk, before processing the new tokens for real. For the parser path only, this means:
|
||||||
|
|
||||||
|
- **Transport**: `output_token_ids` and `output_chunk_lens` round-trip in full in both directions on every call. `output_chunk_lens` has one entry per chunk, which is one per token without speculative decoding. This means O(n) bytes per chunk, O(n²) bytes over a full generation. Bounded by `max_model_len`. `prompt_token_ids` is sent in full on every call too and it isn't trimmed as `output_token_ids` grows. This means that for most of a stream it dominates the per chunk payload. A 100k token prompt with 1k tokens of output means `prompt_token_ids` is ~99% of the request body on every chunk.
|
||||||
|
- **Compute**: replay is O(n) `parse_delta` calls per chunk (O(n²) per generation). `parse_delta` itself is O(n) for parsers that re-scan accumulated text (e.g. Hermes tool-call JSON, DeepSeek-R1 reasoning). The per-generation cost is O(n³) character work, not O(n²). This is a deliberately minimal first implementation with no caching layer.
|
||||||
|
- The parser path also requires `prompt_token_ids` so `parse_delta` can settle whether the prompt left reasoning open or not. Since parser state can't be carried across calls, it re-scans the full prompt once per chunk.
|
||||||
|
- Replay runs off the event loop on the renderer's executor (`renderer_num_workers`, default `1`). Size it for the expected number of concurrent parser configured streams.
|
||||||
|
|
||||||
|
`output_token_ids` and `prompt_token_ids` are both bounded by `max_model_len` but callers streaming long reasoning traces through a parser configured model should expect materially more state transport and CPU cost than the plain detokenization path.
|
||||||
|
|
||||||
## Example
|
## Example
|
||||||
|
|
||||||
The example below drives the full `render → generate → derender` round trip for a chat request against a GPU less render server (`/render`, `/derender`) and a token-in / token-out engine (`/inference/v1/generate`).
|
The example below drives the full `render → generate → derender` round trip for a chat request against a GPU less render server (`/render`, `/derender`) and a token-in / token-out engine (`/inference/v1/generate`).
|
||||||
@@ -102,4 +117,4 @@ with httpx.Client(timeout=60.0) as client:
|
|||||||
print(response["choices"][0]["message"]["content"])
|
print(response["choices"][0]["message"]["content"])
|
||||||
```
|
```
|
||||||
|
|
||||||
Passing `chat_request` lets the derenderer run the configured tool and reasoning parsers. This means `response["choices"][0]["message"]` carries the same `content` / `reasoning` / `tool_calls` split a `vllm serve` server would produce. Omit `chat_request` for plain detokenization only.
|
Passing `chat_request` lets the derenderer run the configured tool and reasoning parsers. This means `response["choices"][0]["message"]` carries the same `content` / `reasoning` / `tool_calls` split a `vllm serve` server would produce. `chat_request` can only be omitted for a model with no tool or reasoning parser configured. A parser configured model rejects a missing `chat_request` with a 400 rather than silently falling back to plain detokenization.
|
||||||
|
|||||||
@@ -800,7 +800,12 @@ def _e2e_generate_response(
|
|||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_e2e_plain_roundtrip(parser_client, parser_tokenizer):
|
async def test_e2e_plain_roundtrip(parser_client, parser_tokenizer):
|
||||||
"""Plain text without reasoning markers roundtrips correctly."""
|
"""Plain text without reasoning markers roundtrips correctly.
|
||||||
|
|
||||||
|
Markerless output has no ``</think>``, which deepseek_r1 classifies
|
||||||
|
wholly as reasoning, so the text lands there rather than in content.
|
||||||
|
What this pins is detokenization fidelity through the parser path.
|
||||||
|
"""
|
||||||
messages = [{"role": "user", "content": "What is 2+2?"}]
|
messages = [{"role": "user", "content": "What is 2+2?"}]
|
||||||
gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages)
|
gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages)
|
||||||
|
|
||||||
@@ -814,16 +819,22 @@ async def test_e2e_plain_roundtrip(parser_client, parser_tokenizer):
|
|||||||
"model": PARSER_MODEL,
|
"model": PARSER_MODEL,
|
||||||
"generate_response": _e2e_generate_response(output_ids),
|
"generate_response": _e2e_generate_response(output_ids),
|
||||||
"prompt_tokens": len(gen_req["token_ids"]),
|
"prompt_tokens": len(gen_req["token_ids"]),
|
||||||
|
"chat_request": {"model": PARSER_MODEL, "messages": messages},
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
assert resp.status_code == 200, resp.text
|
assert resp.status_code == 200, resp.text
|
||||||
content = resp.json()["choices"][0]["message"]["content"]
|
message = resp.json()["choices"][0]["message"]
|
||||||
assert content == expected
|
assert message["reasoning"] == expected
|
||||||
|
assert message["content"] is None
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_e2e_token_identity(parser_client, parser_tokenizer):
|
async def test_e2e_token_identity(parser_client, parser_tokenizer):
|
||||||
"""encode(derender(token_ids)) == token_ids (RL invariant)."""
|
"""encode(derender(token_ids)) == token_ids (RL invariant).
|
||||||
|
|
||||||
|
Markerless output comes back as reasoning (see
|
||||||
|
``test_e2e_plain_roundtrip``), so re-encode that.
|
||||||
|
"""
|
||||||
messages = [{"role": "user", "content": "Hi"}]
|
messages = [{"role": "user", "content": "Hi"}]
|
||||||
gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages)
|
gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages)
|
||||||
|
|
||||||
@@ -836,17 +847,22 @@ async def test_e2e_token_identity(parser_client, parser_tokenizer):
|
|||||||
"model": PARSER_MODEL,
|
"model": PARSER_MODEL,
|
||||||
"generate_response": _e2e_generate_response(output_ids),
|
"generate_response": _e2e_generate_response(output_ids),
|
||||||
"prompt_tokens": len(gen_req["token_ids"]),
|
"prompt_tokens": len(gen_req["token_ids"]),
|
||||||
|
"chat_request": {"model": PARSER_MODEL, "messages": messages},
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
assert resp.status_code == 200
|
assert resp.status_code == 200
|
||||||
content = resp.json()["choices"][0]["message"]["content"]
|
reasoning = resp.json()["choices"][0]["message"]["reasoning"]
|
||||||
re_encoded = _encode(parser_tokenizer, content)
|
re_encoded = _encode(parser_tokenizer, reasoning)
|
||||||
assert output_ids == re_encoded
|
assert output_ids == re_encoded
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_e2e_non_ascii_roundtrip(parser_client, parser_tokenizer):
|
async def test_e2e_non_ascii_roundtrip(parser_client, parser_tokenizer):
|
||||||
"""CJK + emoji roundtrip without U+FFFD."""
|
"""CJK + emoji roundtrip without U+FFFD.
|
||||||
|
|
||||||
|
Markerless output comes back as reasoning (see
|
||||||
|
``test_e2e_plain_roundtrip``).
|
||||||
|
"""
|
||||||
messages = [{"role": "user", "content": "Reply in Chinese"}]
|
messages = [{"role": "user", "content": "Reply in Chinese"}]
|
||||||
gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages)
|
gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages)
|
||||||
|
|
||||||
@@ -859,11 +875,12 @@ async def test_e2e_non_ascii_roundtrip(parser_client, parser_tokenizer):
|
|||||||
"model": PARSER_MODEL,
|
"model": PARSER_MODEL,
|
||||||
"generate_response": _e2e_generate_response(output_ids),
|
"generate_response": _e2e_generate_response(output_ids),
|
||||||
"prompt_tokens": len(gen_req["token_ids"]),
|
"prompt_tokens": len(gen_req["token_ids"]),
|
||||||
|
"chat_request": {"model": PARSER_MODEL, "messages": messages},
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
assert resp.status_code == 200
|
assert resp.status_code == 200
|
||||||
content = resp.json()["choices"][0]["message"]["content"]
|
reasoning = resp.json()["choices"][0]["message"]["reasoning"]
|
||||||
assert "�" not in content
|
assert "�" not in reasoning
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
@@ -976,8 +993,10 @@ async def test_e2e_parsed_reasoning_and_tool_call(parser_client, parser_tokenize
|
|||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_e2e_no_chat_request_fallback(parser_client, parser_tokenizer):
|
async def test_e2e_no_chat_request_rejected(parser_client, parser_tokenizer):
|
||||||
"""Without chat_request, derender falls back to plain detokenization."""
|
"""Without chat_request a parser configured model rejects with 400
|
||||||
|
rather than silently falling back to plain detokenization. This is to
|
||||||
|
prevent the leak of raw reasoning/tool markup into content."""
|
||||||
messages = [{"role": "user", "content": "Hello"}]
|
messages = [{"role": "user", "content": "Hello"}]
|
||||||
gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages)
|
gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages)
|
||||||
|
|
||||||
@@ -992,9 +1011,8 @@ async def test_e2e_no_chat_request_fallback(parser_client, parser_tokenizer):
|
|||||||
"prompt_tokens": len(gen_req["token_ids"]),
|
"prompt_tokens": len(gen_req["token_ids"]),
|
||||||
},
|
},
|
||||||
)
|
)
|
||||||
assert resp.status_code == 200
|
assert resp.status_code == 400
|
||||||
content = resp.json()["choices"][0]["message"]["content"]
|
assert "chat_request" in resp.json()["error"]["message"]
|
||||||
assert "Hi" in content
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -138,6 +138,13 @@ class ServingDerender(BaseServing):
|
|||||||
if bounds_error is not None:
|
if bounds_error is not None:
|
||||||
return bounds_error
|
return bounds_error
|
||||||
|
|
||||||
|
if self.online_derenderer.parser is not None and request.chat_request is None:
|
||||||
|
return self.create_error_response(
|
||||||
|
"chat_request is required when a tool or reasoning parser is "
|
||||||
|
"configured because plain detokenization would leak raw parser "
|
||||||
|
"markup into content."
|
||||||
|
)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
choices = await self.online_derenderer.derender_chat(
|
choices = await self.online_derenderer.derender_chat(
|
||||||
request.generate_response, request.chat_request
|
request.generate_response, request.chat_request
|
||||||
@@ -248,14 +255,67 @@ class ServingDerender(BaseServing):
|
|||||||
|
|
||||||
Processes one ``GenerateStreamResponse`` chunk and returns the
|
Processes one ``GenerateStreamResponse`` chunk and returns the
|
||||||
derendered chunk together with the updated client carried state.
|
derendered chunk together with the updated client carried state.
|
||||||
|
|
||||||
``parser is None`` or no ``chat_request`` until reasoning/tool call
|
|
||||||
functionality added in future PR.
|
|
||||||
"""
|
"""
|
||||||
error_check_ret = await self._check_model(request)
|
error_check_ret = await self._check_model(request)
|
||||||
if error_check_ret is not None:
|
if error_check_ret is not None:
|
||||||
return error_check_ret
|
return error_check_ret
|
||||||
|
|
||||||
|
if self.online_derenderer.parser is not None and request.chat_request is None:
|
||||||
|
return self.create_error_response(
|
||||||
|
"chat_request is required when a tool or reasoning parser is "
|
||||||
|
"configured because plain detokenization would leak raw parser "
|
||||||
|
"markup into content."
|
||||||
|
)
|
||||||
|
|
||||||
|
if (
|
||||||
|
self.online_derenderer.parser is not None
|
||||||
|
and request.prompt_token_ids is None
|
||||||
|
):
|
||||||
|
return self.create_error_response(
|
||||||
|
"prompt_token_ids is required when a tool or reasoning "
|
||||||
|
"parser is configured. Without it parse_delta cannot tell "
|
||||||
|
"whether the prompt left reasoning open (e.g. a chat "
|
||||||
|
"template that pre-opens <think>) and would misclassify "
|
||||||
|
"reasoning content as plain content."
|
||||||
|
)
|
||||||
|
|
||||||
|
# Each streamed chunk contains at most one choice per SSE event.
|
||||||
|
# A single DerenderStreamState is threaded
|
||||||
|
# through every choice in the chunk, so >1 would corrupt detok/parser
|
||||||
|
# state across choices. See the matching check in
|
||||||
|
# OnlineDerenderer.derender_chat_stream which this fails ahead of to
|
||||||
|
# avoid touching the tokenizer at all on a malformed chunk.
|
||||||
|
num_choices = len(request.generate_chunk.choices)
|
||||||
|
if num_choices > 1:
|
||||||
|
return self.create_error_response(
|
||||||
|
f"derender_chat_stream expects at most one choice per chunk "
|
||||||
|
f"(got {num_choices})."
|
||||||
|
)
|
||||||
|
|
||||||
|
# prompt_token_ids is folded in here too. It is caller supplied and
|
||||||
|
# otherwise unbounded. A parser configured deployment rescans it
|
||||||
|
# in full on every chunk (is_reasoning_end /
|
||||||
|
# adjust_initial_state_from_prompt, see abstract_parser.parse_delta)
|
||||||
|
# since parser state cannot be carried across calls.
|
||||||
|
prompt_len = (
|
||||||
|
len(request.prompt_token_ids) if request.prompt_token_ids is not None else 0
|
||||||
|
)
|
||||||
|
prior_len = (
|
||||||
|
len(request.stream_state.output_token_ids)
|
||||||
|
if request.stream_state is not None
|
||||||
|
else 0
|
||||||
|
)
|
||||||
|
delta_len = sum(
|
||||||
|
len(c.token_ids) for c in request.generate_chunk.choices if c.token_ids
|
||||||
|
)
|
||||||
|
max_model_len = self.model_config.max_model_len
|
||||||
|
if prompt_len + prior_len + delta_len > max_model_len:
|
||||||
|
return self.create_error_response(
|
||||||
|
f"prompt_token_ids length ({prompt_len}) plus "
|
||||||
|
f"output_token_ids length ({prior_len}) plus delta "
|
||||||
|
f"({delta_len}) exceeds max_model_len ({max_model_len})."
|
||||||
|
)
|
||||||
|
|
||||||
model_name = request.model or self.models.model_name()
|
model_name = request.model or self.models.model_name()
|
||||||
try:
|
try:
|
||||||
chunk, updated_state = await self.online_derenderer.derender_chat_stream(
|
chunk, updated_state = await self.online_derenderer.derender_chat_stream(
|
||||||
@@ -264,15 +324,21 @@ class ServingDerender(BaseServing):
|
|||||||
state=request.stream_state,
|
state=request.stream_state,
|
||||||
chat_request=request.chat_request,
|
chat_request=request.chat_request,
|
||||||
prompt_tokens=request.prompt_tokens,
|
prompt_tokens=request.prompt_tokens,
|
||||||
|
prompt_token_ids=request.prompt_token_ids,
|
||||||
)
|
)
|
||||||
except NotImplementedError as exc:
|
except Exception as exc:
|
||||||
return self.create_error_response(exc)
|
# The two ValueErrors derender_chat_stream can raise directly
|
||||||
except ValueError as exc:
|
# (missing chat_request, >1 choice per chunk) are already
|
||||||
return self.create_error_response(str(exc))
|
# pre-checked above, so anything reaching here comes from the
|
||||||
except (KeyError, IndexError) as exc:
|
# parser replaying entirely client controlled state. Hermes
|
||||||
return self.create_error_response(
|
# swallows its own streaming exceptions but finalize_generation
|
||||||
f"invalid stream_state: detokenization failed ({exc!r})"
|
# and engine based parsers don't, so an unexpected parser
|
||||||
|
# failure on malformed (but well typed) state must surface as
|
||||||
|
# a 400 here rather than an unhandled 500.
|
||||||
|
logger.debug(
|
||||||
|
"derender stream replay failed: %s: %s", type(exc).__name__, exc
|
||||||
)
|
)
|
||||||
|
return self.create_error_response("invalid stream_state: derender failed")
|
||||||
|
|
||||||
logger.debug(
|
logger.debug(
|
||||||
"derender_chat_stream request_id=%s model=%s delta_tokens=%d",
|
"derender_chat_stream request_id=%s model=%s delta_tokens=%d",
|
||||||
@@ -313,7 +379,7 @@ class ServingDerender(BaseServing):
|
|||||||
return self.create_error_response(str(exc))
|
return self.create_error_response(str(exc))
|
||||||
except (KeyError, IndexError) as exc:
|
except (KeyError, IndexError) as exc:
|
||||||
return self.create_error_response(
|
return self.create_error_response(
|
||||||
f"invalid stream_state: detokenization failed ({exc!r})"
|
f"invalid stream_state: detokenization failed ({exc})"
|
||||||
)
|
)
|
||||||
|
|
||||||
logger.debug(
|
logger.debug(
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# SPDX-License-Identifier: Apache-2.0
|
# SPDX-License-Identifier: Apache-2.0
|
||||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||||
from typing import Any, Literal, TypeAlias
|
from typing import Annotated, Any, Literal, TypeAlias
|
||||||
|
|
||||||
from pydantic import (
|
from pydantic import (
|
||||||
BaseModel,
|
BaseModel,
|
||||||
@@ -483,16 +483,32 @@ class DerenderStreamState(BaseModel):
|
|||||||
streaming derender endpoint. All fields are plain JSON serializable data.
|
streaming derender endpoint. All fields are plain JSON serializable data.
|
||||||
No opaque tokenizer or parser internals are stored here.
|
No opaque tokenizer or parser internals are stored here.
|
||||||
|
|
||||||
|
Two separate sets of fields support two different streaming modes:
|
||||||
|
|
||||||
|
- For plain streaming (no parser configured including the completions path):
|
||||||
|
`prev_tokens`, `prefix_offset` and `read_offset` maintain a bounded incremental
|
||||||
|
decoding window. This requires O(window) transport and O(delta) computation
|
||||||
|
per chunk.
|
||||||
|
- For parser enabled chat streaming: `output_token_ids`, `output_chunk_lens`,
|
||||||
|
`tools_streamed` and `last_tool_call_ids` are used to replay `parse_delta()`
|
||||||
|
from scratch on every chunk because parser state cannot be serialized. This
|
||||||
|
incurs O(n) transport per chunk (O(n²) per generation) and O(n²)
|
||||||
|
`parse_delta()` calls per generation. Since many parsers re-scan the
|
||||||
|
entire accumulated text on each invocation, `parse_delta()` itself is
|
||||||
|
O(n) yielding a true worst case compute cost of O(n³) per generation.
|
||||||
|
No caching is performed. Work is bounded by `max_model_len`.
|
||||||
|
See `OnlineDerenderer._derender_chat_stream_parsed`.
|
||||||
|
|
||||||
The detokenization strategy carries the incremental decode offsets
|
The detokenization strategy carries the incremental decode offsets
|
||||||
directly rather than re-sending the whole token history each chunk.
|
directly rather than re-sending the whole token history each chunk.
|
||||||
``detokenize_incrementally`` only ever reads the trailing token window
|
`detokenize_incrementally` only ever reads the trailing token window
|
||||||
``prev_tokens[prefix_offset:]``, so we carry just that tail plus the two
|
`prev_tokens[prefix_offset:]`, so we carry just that tail plus the two
|
||||||
offsets. Each chunk resumes exactly where the last one stopped, including
|
offsets. Each chunk resumes exactly where the last one stopped, including
|
||||||
any partially processed multi-byte character (tracked by ``read_offset``),
|
any partially processed multi-byte character (tracked by `read_offset`),
|
||||||
then trims and rebases the window so it never grows with generation length.
|
then trims and rebases the window so it never grows with generation length.
|
||||||
|
|
||||||
Performance:
|
Performance:
|
||||||
- Compute per chunk is O(delta). One ``detokenize_incrementally`` call per
|
- Compute per chunk is O(delta). One `detokenize_incrementally` call per
|
||||||
new token, independent of how many tokens preceded it.
|
new token, independent of how many tokens preceded it.
|
||||||
- Transport per chunk is O(window). The carried tail is bounded by the
|
- Transport per chunk is O(window). The carried tail is bounded by the
|
||||||
incremental detokenization offset, so cumulative bytes over the wire are
|
incremental detokenization offset, so cumulative bytes over the wire are
|
||||||
@@ -500,18 +516,18 @@ class DerenderStreamState(BaseModel):
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
prev_tokens: list[str] = Field(default_factory=list)
|
prev_tokens: list[str] = Field(default_factory=list)
|
||||||
"""Trailing decode window. Token strings from ``prefix_offset`` onward.
|
"""Trailing decode window. Token strings from `prefix_offset` onward.
|
||||||
|
|
||||||
Bounded, trimmed and rebased each chunk to the tail
|
Bounded, trimmed and rebased each chunk to the tail
|
||||||
``detokenize_incrementally`` still reads, so it does not grow with the
|
`detokenize_incrementally` still reads, so it does not grow with the
|
||||||
number of chunks.
|
number of chunks.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
prefix_offset: int = Field(default=0, ge=0)
|
prefix_offset: int = Field(default=0, ge=0)
|
||||||
"""Prefix offset into ``prev_tokens`` for incremental detokenization."""
|
"""Prefix offset into `prev_tokens` for incremental detokenization."""
|
||||||
|
|
||||||
read_offset: int = Field(default=0, ge=0)
|
read_offset: int = Field(default=0, ge=0)
|
||||||
"""Read offset into ``prev_tokens`` for incremental detokenization."""
|
"""Read offset into `prev_tokens` for incremental detokenization."""
|
||||||
|
|
||||||
@field_validator("prev_tokens")
|
@field_validator("prev_tokens")
|
||||||
@classmethod
|
@classmethod
|
||||||
@@ -525,23 +541,53 @@ class DerenderStreamState(BaseModel):
|
|||||||
return v
|
return v
|
||||||
|
|
||||||
role_sent: bool = False
|
role_sent: bool = False
|
||||||
"""True once the initial ``role: "assistant"`` delta has been emitted.
|
"""True once the initial `role: "assistant"` delta has been emitted.
|
||||||
|
|
||||||
Prevents re-emitting the role on subsequent chunks even when the detok
|
Prevents re-emitting the role on subsequent chunks even when the detok
|
||||||
window is transiently empty (e.g. usage only final chunk).
|
window is transiently empty (e.g. usage only final chunk).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
# TODO: Properties used in follow on PR for tool call parsing
|
output_token_ids: list[int] = Field(default_factory=list)
|
||||||
last_content: str | None = None
|
"""All output tokens seen so far. Parser path only.
|
||||||
"""Last emitted cumulative assistant content text."""
|
|
||||||
|
|
||||||
last_reasoning: str | None = None
|
Replay buffer: each chunk rebuilds a fresh parser and replays every
|
||||||
"""Last emitted cumulative reasoning text."""
|
token in here through `parse_delta` (discarding the result) before
|
||||||
|
processing the current chunk's tokens since parser internal state
|
||||||
|
cannot be serialized into this stateless model. Unavoidably O(n)
|
||||||
|
bounded by ``max_model_len`` (enforced server side, not by a field
|
||||||
|
validator here since the bound is model dependent).
|
||||||
|
"""
|
||||||
|
|
||||||
|
output_chunk_lens: list[Annotated[int, Field(gt=0)]] = Field(default_factory=list)
|
||||||
|
"""Token count of each chunk in `output_token_ids`. Parser path only.
|
||||||
|
|
||||||
|
Replay uses these to reproduce the original `parse_delta` call
|
||||||
|
boundaries. Must sum to `len(output_token_ids)`.
|
||||||
|
"""
|
||||||
|
|
||||||
|
@model_validator(mode="after")
|
||||||
|
def _validate_output_chunk_lens(self) -> "DerenderStreamState":
|
||||||
|
total = sum(self.output_chunk_lens)
|
||||||
|
if total != len(self.output_token_ids):
|
||||||
|
raise ValueError(
|
||||||
|
f"output_chunk_lens must sum to len(output_token_ids) "
|
||||||
|
f"(got sum={total}, len(output_token_ids)="
|
||||||
|
f"{len(self.output_token_ids)})"
|
||||||
|
)
|
||||||
|
return self
|
||||||
|
|
||||||
|
tools_streamed: bool = False
|
||||||
|
"""True once a tool call delta has been emitted. Parser path only.
|
||||||
|
|
||||||
|
Drives the `finish_reason` -> `"tool_calls"` rewrite on the final
|
||||||
|
chunk mirroring the generate streaming path.
|
||||||
|
"""
|
||||||
|
|
||||||
last_tool_call_ids: list[str] = Field(default_factory=list)
|
last_tool_call_ids: list[str] = Field(default_factory=list)
|
||||||
"""Stable tool-call IDs, assigned once when each call first appears.
|
"""Stable tool call IDs, assigned once when each call first appears.
|
||||||
|
|
||||||
Prevents ID regeneration across re-parsing.
|
Indexed by tool call index. Parser path only. Prevents ID regeneration
|
||||||
|
when replay reprocesses a tool call that already has a pinned ID.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
||||||
@@ -570,6 +616,19 @@ class DerenderChatStreamRequest(BaseModel):
|
|||||||
prompt_tokens: int | None = None
|
prompt_tokens: int | None = None
|
||||||
"""Prompt token count for usage. Forwarded from the render step."""
|
"""Prompt token count for usage. Forwarded from the render step."""
|
||||||
|
|
||||||
|
prompt_token_ids: list[int] | None = None
|
||||||
|
"""Prompt token IDs. Required by the parser path's `parse_delta` to
|
||||||
|
settle its initial reasoning state (e.g. chat templates that pre-open
|
||||||
|
``<think>``). `prompt_tokens` is a usage count and cannot serve this
|
||||||
|
purpose. Sourced from `GenerateRequest.token_ids` at the render step.
|
||||||
|
|
||||||
|
Rejected with a 400 (by `ServingDerender`) when a tool or reasoning
|
||||||
|
parser is configured and this is omitted. Without it, `parse_delta`
|
||||||
|
cannot tell whether the prompt left reasoning open and would silently
|
||||||
|
misclassify reasoning content as plain content. Unused on the plain
|
||||||
|
detokenization path.
|
||||||
|
"""
|
||||||
|
|
||||||
chat_request: ChatCompletionRequest | None = None
|
chat_request: ChatCompletionRequest | None = None
|
||||||
"""The original (post adjust_request) ChatCompletionRequest from /render."""
|
"""The original (post adjust_request) ChatCompletionRequest from /render."""
|
||||||
|
|
||||||
|
|||||||
@@ -4,7 +4,10 @@ from typing import Any
|
|||||||
|
|
||||||
from vllm.config import ModelConfig
|
from vllm.config import ModelConfig
|
||||||
from vllm.entrypoints.chat_utils import ChatTemplateContentFormatOption
|
from vllm.entrypoints.chat_utils import ChatTemplateContentFormatOption
|
||||||
from vllm.entrypoints.generate.base.protocol import DeltaMessage, ToolCall
|
from vllm.entrypoints.generate.base.protocol import (
|
||||||
|
DeltaMessage,
|
||||||
|
ToolCall,
|
||||||
|
)
|
||||||
from vllm.entrypoints.generate.base.serving import resolve_token_id_placeholder
|
from vllm.entrypoints.generate.base.serving import resolve_token_id_placeholder
|
||||||
from vllm.entrypoints.openai.chat_completion.protocol import (
|
from vllm.entrypoints.openai.chat_completion.protocol import (
|
||||||
ChatCompletionLogProbs,
|
ChatCompletionLogProbs,
|
||||||
@@ -29,6 +32,9 @@ from vllm.entrypoints.scale_out.token_in_token_out.protocol import (
|
|||||||
)
|
)
|
||||||
from vllm.entrypoints.serve.engine.protocol import UsageInfo
|
from vllm.entrypoints.serve.engine.protocol import UsageInfo
|
||||||
from vllm.entrypoints.serve.utils.request_logger import RequestLogger
|
from vllm.entrypoints.serve.utils.request_logger import RequestLogger
|
||||||
|
from vllm.entrypoints.serve.utils.tool_calls_utils import (
|
||||||
|
maybe_filter_parallel_tool_calls,
|
||||||
|
)
|
||||||
from vllm.logger import init_logger
|
from vllm.logger import init_logger
|
||||||
from vllm.parser import Parser, ParserManager
|
from vllm.parser import Parser, ParserManager
|
||||||
from vllm.renderers import BaseRenderer
|
from vllm.renderers import BaseRenderer
|
||||||
@@ -95,6 +101,11 @@ class OnlineDerenderer:
|
|||||||
self._derender_completion_async = make_async(
|
self._derender_completion_async = make_async(
|
||||||
self._derender_completion, executor=renderer._executor
|
self._derender_completion, executor=renderer._executor
|
||||||
)
|
)
|
||||||
|
# Replay is O(n) per chunk (unlike the O(delta) detok-only paths),
|
||||||
|
# so it must not run on the event loop either.
|
||||||
|
self._derender_chat_stream_parsed_async = make_async(
|
||||||
|
self._derender_chat_stream_parsed, executor=renderer._executor
|
||||||
|
)
|
||||||
|
|
||||||
async def derender_chat(
|
async def derender_chat(
|
||||||
self,
|
self,
|
||||||
@@ -144,6 +155,7 @@ class OnlineDerenderer:
|
|||||||
tokenizer,
|
tokenizer,
|
||||||
chat_request.tools,
|
chat_request.tools,
|
||||||
chat_template_kwargs=chat_template_kwargs,
|
chat_template_kwargs=chat_template_kwargs,
|
||||||
|
model_config=self.model_config,
|
||||||
)
|
)
|
||||||
reasoning, content, tool_calls = parser.parse(
|
reasoning, content, tool_calls = parser.parse(
|
||||||
decoded_text,
|
decoded_text,
|
||||||
@@ -278,23 +290,30 @@ class OnlineDerenderer:
|
|||||||
state: DerenderStreamState | None = None,
|
state: DerenderStreamState | None = None,
|
||||||
chat_request: ChatCompletionRequest | None = None,
|
chat_request: ChatCompletionRequest | None = None,
|
||||||
prompt_tokens: int | None = None,
|
prompt_tokens: int | None = None,
|
||||||
|
prompt_token_ids: list[int] | None = None,
|
||||||
) -> tuple[ChatCompletionStreamResponse, DerenderStreamState]:
|
) -> tuple[ChatCompletionStreamResponse, DerenderStreamState]:
|
||||||
"""Process one GenerateStreamResponse chunk for streaming chat derender.
|
"""Process one GenerateStreamResponse chunk for streaming chat derender.
|
||||||
|
|
||||||
TODO: parse path for reasoning and tool calls is implemented in future PR.
|
Unlike OpenAI's API, which always emits `role: "assistant"` on the
|
||||||
|
|
||||||
Unlike OpenAI's API, which always emits ``role: "assistant"`` on the
|
|
||||||
very first chunk, this emits it on the first chunk with a non empty
|
very first chunk, this emits it on the first chunk with a non empty
|
||||||
``choices`` list. A leading usage only chunk therefore defers the
|
`choices` list. A leading usage only chunk therefore defers the
|
||||||
role to the following content chunk instead of sending an empty
|
role to the following content chunk instead of sending an empty
|
||||||
role only delta.
|
role only delta.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
model: Model name for the response object.
|
model: Model name for the response object.
|
||||||
generate_chunk: One SSE chunk from ``/inference/v1/generate``.
|
generate_chunk: One SSE chunk from `/inference/v1/generate`.
|
||||||
state: Client carried detok state (``None`` for first call).
|
state: Client carried detok state (`None` for first call).
|
||||||
chat_request: Original ChatCompletionRequest from ``/render``.
|
chat_request: Original ChatCompletionRequest from `/render`.
|
||||||
|
Required when a reasoning or tool parser is configured
|
||||||
|
(validated by the caller — see `ServingDerender`) because
|
||||||
|
plain detokenization would leak raw parser markup into `content`.
|
||||||
prompt_tokens: Prompt token count for the usage chunk.
|
prompt_tokens: Prompt token count for the usage chunk.
|
||||||
|
prompt_token_ids: Prompt token IDs. Parser path only. Lets
|
||||||
|
`parse_delta` settle its initial reasoning state. Required
|
||||||
|
when a reasoning or tool parser is configured (validated by
|
||||||
|
the caller — see ``ServingDerender``). Without it reasoning
|
||||||
|
left open by the prompt would be misclassified as content.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
(chunk, updated_state) — the derendered SSE chunk and the state
|
(chunk, updated_state) — the derendered SSE chunk and the state
|
||||||
@@ -304,28 +323,36 @@ class OnlineDerenderer:
|
|||||||
if state is None:
|
if state is None:
|
||||||
state = DerenderStreamState()
|
state = DerenderStreamState()
|
||||||
|
|
||||||
if self.parser is not None:
|
|
||||||
# TODO: Follow on PR will implement the parse path. Check on the
|
|
||||||
# parser alone (fail closed). A parser configured model must never
|
|
||||||
# fall through to plain detok on the streaming path, even when
|
|
||||||
# ``chat_request`` is omitted or reasoning/tool markup would leak
|
|
||||||
# into ``delta.content``.
|
|
||||||
raise NotImplementedError(
|
|
||||||
"Streaming chat derender is not yet supported for models with "
|
|
||||||
"a reasoning or tool parser configured. Use the non-streaming "
|
|
||||||
"derender endpoint (stream=false) for parsed output."
|
|
||||||
)
|
|
||||||
|
|
||||||
# A single DerenderStreamState is threaded through every choice in
|
# A single DerenderStreamState is threaded through every choice in
|
||||||
# this chunk. Correct only when there is at most one choice per SSE
|
# this chunk. Correct only when there is at most one choice per SSE
|
||||||
# event (n=1, one call per index), as the streaming derender
|
# event (n=1, one call per index), as the streaming derender
|
||||||
# protocol assumes. Multiple choices sharing one chunk would corrupt
|
# protocol assumes. Multiple choices sharing one chunk would corrupt
|
||||||
# each other's detok window.
|
# each other's detok/parser state.
|
||||||
if len(generate_chunk.choices) > 1:
|
if len(generate_chunk.choices) > 1:
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
"derender_chat_stream expects at most one choice per chunk"
|
"derender_chat_stream expects at most one choice per chunk"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
parser_cls = self.parser
|
||||||
|
if parser_cls is not None:
|
||||||
|
# Fail (mirrors ServingDerender's pre-check) because a parser
|
||||||
|
# configured model must never fall through to plain detok or
|
||||||
|
# reasoning/tool markup would leak into `delta.content`.
|
||||||
|
if chat_request is None:
|
||||||
|
raise ValueError(
|
||||||
|
"chat_request is required for streaming chat derender "
|
||||||
|
"when a tool or reasoning parser is configured"
|
||||||
|
)
|
||||||
|
return await self._derender_chat_stream_parsed_async(
|
||||||
|
parser_cls,
|
||||||
|
model,
|
||||||
|
generate_chunk,
|
||||||
|
state,
|
||||||
|
chat_request,
|
||||||
|
prompt_tokens,
|
||||||
|
prompt_token_ids,
|
||||||
|
)
|
||||||
|
|
||||||
tokenizer = self.renderer.get_tokenizer()
|
tokenizer = self.renderer.get_tokenizer()
|
||||||
skip_special = (
|
skip_special = (
|
||||||
chat_request.skip_special_tokens if chat_request is not None else True
|
chat_request.skip_special_tokens if chat_request is not None else True
|
||||||
@@ -374,6 +401,203 @@ class OnlineDerenderer:
|
|||||||
)
|
)
|
||||||
return chunk, updated_state
|
return chunk, updated_state
|
||||||
|
|
||||||
|
def _derender_chat_stream_parsed(
|
||||||
|
self,
|
||||||
|
parser_cls: type[Parser],
|
||||||
|
model: str,
|
||||||
|
generate_chunk: GenerateStreamResponse,
|
||||||
|
state: DerenderStreamState,
|
||||||
|
chat_request: ChatCompletionRequest,
|
||||||
|
prompt_tokens: int | None,
|
||||||
|
prompt_token_ids: list[int] | None,
|
||||||
|
) -> tuple[ChatCompletionStreamResponse, DerenderStreamState]:
|
||||||
|
"""Parser path for streaming chat derender: replay + `parse_delta`.
|
||||||
|
|
||||||
|
Parser internal state (buffered markup, reasoning/tool phase, etc.)
|
||||||
|
cannot be serialized into `DerenderStreamState`, so each call
|
||||||
|
builds a fresh parser and replays every prior output token through
|
||||||
|
`parse_delta` (discarding the result) before processing this
|
||||||
|
chunk's tokens for real.
|
||||||
|
|
||||||
|
Every chunk, replayed or live, goes through one `parse_delta` call
|
||||||
|
with the tokens it arrived with (more than one under e.g.
|
||||||
|
speculative decoding). Replay recovers those boundaries from
|
||||||
|
`state.output_chunk_lens`, so the rebuilt parser state matches
|
||||||
|
standard serving, which calls `parse_delta` once per engine step.
|
||||||
|
Text comes from a fresh incremental detokenizer with special tokens
|
||||||
|
preserved (``skip_special_tokens=False``).
|
||||||
|
"""
|
||||||
|
tokenizer = self.renderer.get_tokenizer()
|
||||||
|
|
||||||
|
chat_template_kwargs: dict[str, Any] = {}
|
||||||
|
if not self.use_harmony:
|
||||||
|
chat_template_kwargs = (
|
||||||
|
chat_request.build_chat_params(
|
||||||
|
self.chat_template,
|
||||||
|
self.chat_template_content_format,
|
||||||
|
)
|
||||||
|
.with_defaults(self.default_chat_template_kwargs)
|
||||||
|
.chat_template_kwargs
|
||||||
|
)
|
||||||
|
|
||||||
|
parser = parser_cls(
|
||||||
|
tokenizer,
|
||||||
|
chat_request.tools,
|
||||||
|
chat_template_kwargs=chat_template_kwargs,
|
||||||
|
model_config=self.model_config,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Ephemeral incremental detok window, local to this call. Threaded
|
||||||
|
# across both the replay and current chunk phases (via `nonlocal`)
|
||||||
|
# so multi-byte characters split across that boundary still decode
|
||||||
|
# correctly. Discarded once the call returns.
|
||||||
|
detok_state = DerenderStreamState()
|
||||||
|
|
||||||
|
def _replay(token_ids: list[int], chunk_lens: list[int]) -> None:
|
||||||
|
"""Replay prior chunks through `parse_delta` to rebuild parser
|
||||||
|
state, discarding the result. Never `finished` since that only
|
||||||
|
applies to the current chunk.
|
||||||
|
"""
|
||||||
|
nonlocal detok_state
|
||||||
|
start = 0
|
||||||
|
for chunk_len in chunk_lens:
|
||||||
|
chunk = token_ids[start : start + chunk_len]
|
||||||
|
start += chunk_len
|
||||||
|
text, detok_state = self._detokenize_delta(
|
||||||
|
tokenizer, chunk, detok_state, skip_special_tokens=False
|
||||||
|
)
|
||||||
|
parser.parse_delta(
|
||||||
|
text,
|
||||||
|
chunk,
|
||||||
|
chat_request,
|
||||||
|
prompt_token_ids=prompt_token_ids,
|
||||||
|
finished=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Replay history to reconstruct parser state. The result is thrown
|
||||||
|
# away and only the current chunk's emission goes to the client.
|
||||||
|
_replay(state.output_token_ids, state.output_chunk_lens)
|
||||||
|
|
||||||
|
stream_choices: list[ChatCompletionResponseStreamChoice] = []
|
||||||
|
output_token_ids = list(state.output_token_ids)
|
||||||
|
output_chunk_lens = list(state.output_chunk_lens)
|
||||||
|
role_sent = state.role_sent
|
||||||
|
tools_streamed = state.tools_streamed
|
||||||
|
last_tool_call_ids = list(state.last_tool_call_ids)
|
||||||
|
|
||||||
|
# At most one choice: the caller (derender_chat_stream) already
|
||||||
|
# rejects >1 before dispatching here. role_sent/tools_streamed/
|
||||||
|
# output_token_ids below are updated for that single choice, not
|
||||||
|
# accumulated across choices. Looping over generate_chunk.choices
|
||||||
|
# could silently corrupt output if chunks were ever allowed to
|
||||||
|
# contain multiple choices.
|
||||||
|
if generate_chunk.choices:
|
||||||
|
choice = generate_chunk.choices[0]
|
||||||
|
delta_tids = choice.token_ids or []
|
||||||
|
is_finished = choice.finish_reason is not None
|
||||||
|
|
||||||
|
if delta_tids:
|
||||||
|
# One parse_delta call for the whole chunk (producer
|
||||||
|
# granularity), not one per token. See the granularity note
|
||||||
|
# in this method's docstring.
|
||||||
|
text, detok_state = self._detokenize_delta(
|
||||||
|
tokenizer, delta_tids, detok_state, skip_special_tokens=False
|
||||||
|
)
|
||||||
|
delta_message = parser.parse_delta(
|
||||||
|
text,
|
||||||
|
delta_tids,
|
||||||
|
chat_request,
|
||||||
|
prompt_token_ids=prompt_token_ids,
|
||||||
|
finished=is_finished,
|
||||||
|
)
|
||||||
|
elif is_finished:
|
||||||
|
# Finish only chunk (no new tokens). Still flush any
|
||||||
|
# buffered tool call arguments.
|
||||||
|
delta_message = parser.parse_delta(
|
||||||
|
"",
|
||||||
|
[],
|
||||||
|
chat_request,
|
||||||
|
prompt_token_ids=prompt_token_ids,
|
||||||
|
finished=True,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
delta_message = None
|
||||||
|
|
||||||
|
if delta_tids:
|
||||||
|
output_token_ids.extend(delta_tids)
|
||||||
|
output_chunk_lens.append(len(delta_tids))
|
||||||
|
|
||||||
|
if delta_message is None:
|
||||||
|
delta_message = DeltaMessage()
|
||||||
|
|
||||||
|
if delta_message.tool_calls:
|
||||||
|
tools_streamed = True
|
||||||
|
for tc in delta_message.tool_calls:
|
||||||
|
if tc.id is None:
|
||||||
|
continue
|
||||||
|
if tc.index < len(last_tool_call_ids):
|
||||||
|
# Pin: reuse the ID already recorded for this index
|
||||||
|
# rather than one a from scratch replay regenerated.
|
||||||
|
# Real trigger not just defensive with
|
||||||
|
# tool_choice="required",
|
||||||
|
# extract_required_tool_call_streaming resets
|
||||||
|
# function_name_returned to False whenever the
|
||||||
|
# partial JSON transiently fails to parse which
|
||||||
|
# re-emits id+name for the same index on replay.
|
||||||
|
tc.id = last_tool_call_ids[tc.index]
|
||||||
|
else:
|
||||||
|
last_tool_call_ids.append(tc.id)
|
||||||
|
|
||||||
|
if not role_sent:
|
||||||
|
delta_message.role = "assistant"
|
||||||
|
role_sent = True
|
||||||
|
|
||||||
|
finish_reason = choice.finish_reason
|
||||||
|
if finish_reason is not None:
|
||||||
|
is_named_tool_choice = (
|
||||||
|
type(chat_request.tool_choice) is ChatCompletionNamedToolChoiceParam
|
||||||
|
)
|
||||||
|
if tools_streamed and not is_named_tool_choice:
|
||||||
|
finish_reason = "tool_calls"
|
||||||
|
|
||||||
|
stream_choice = ChatCompletionResponseStreamChoice(
|
||||||
|
index=choice.index,
|
||||||
|
delta=delta_message,
|
||||||
|
finish_reason=finish_reason,
|
||||||
|
)
|
||||||
|
stream_choices.append(
|
||||||
|
maybe_filter_parallel_tool_calls(stream_choice, chat_request)
|
||||||
|
)
|
||||||
|
|
||||||
|
updated_state = state.model_copy(
|
||||||
|
update={
|
||||||
|
"output_token_ids": output_token_ids,
|
||||||
|
"output_chunk_lens": output_chunk_lens,
|
||||||
|
"role_sent": role_sent,
|
||||||
|
"tools_streamed": tools_streamed,
|
||||||
|
"last_tool_call_ids": last_tool_call_ids,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
usage: UsageInfo | None = None
|
||||||
|
if generate_chunk.usage is not None:
|
||||||
|
u = generate_chunk.usage
|
||||||
|
pt = prompt_tokens if prompt_tokens is not None else (u.prompt_tokens or 0)
|
||||||
|
ct = u.completion_tokens or 0
|
||||||
|
usage = UsageInfo(
|
||||||
|
prompt_tokens=pt,
|
||||||
|
completion_tokens=ct,
|
||||||
|
total_tokens=pt + ct,
|
||||||
|
)
|
||||||
|
|
||||||
|
chunk = ChatCompletionStreamResponse(
|
||||||
|
id=generate_chunk.request_id,
|
||||||
|
model=model,
|
||||||
|
choices=stream_choices,
|
||||||
|
usage=usage,
|
||||||
|
)
|
||||||
|
return chunk, updated_state
|
||||||
|
|
||||||
async def derender_completion(
|
async def derender_completion(
|
||||||
self,
|
self,
|
||||||
generate_responses: list[GenerateResponse],
|
generate_responses: list[GenerateResponse],
|
||||||
|
|||||||
Reference in New Issue
Block a user