diff --git a/docs/serving/online_serving/derenderer.md b/docs/serving/online_serving/derenderer.md index 298510062a..773131d033 100644 --- a/docs/serving/online_serving/derenderer.md +++ b/docs/serving/online_serving/derenderer.md @@ -6,7 +6,7 @@ This closes the loop for a token-in / token-out engine in disaggregated serving: - **GPU less post processing**: Detokenization, reasoning parsing, and tool call parsing run on the same GPU less frontend that hosts `/render` - **Parser parity**: The derenderer reuses vLLM's tool and reasoning parsers, so a disaggregated deployment produces the same `content`/`reasoning`/ `tool_calls` split as a standard `vllm serve` server -- **Non-streaming**: The endpoints expect a complete `GenerateResponse` with all token IDs present and perform one-shot parsing. Streaming derender would require a separate endpoint design and is not currently supported but is in the pipeline +- **Streaming and one-shot parsing**: Non-streaming calls send a complete `GenerateResponse` with all token IDs present and get one-shot parsing. Both endpoints also accept `stream: true`, taking one `GenerateStreamResponse` delta plus a client carried `stream_state` and returning `{chunk, stream_state}`. The chat endpoint's streaming path supports reasoning and tool call parsing while emitting the same `reasoning`/`content`/`tool_calls` deltas the generate streaming path would for the same token IDs Both endpoints are hosted by the GPU less rendering server started with [`vllm launch render`](../../cli/launch/render.md), alongside the `/render` endpoints. @@ -21,7 +21,7 @@ endpoints. └─────────────── request + prompt_tokens ──┘ ``` -The derender step needs more than the engine's `token_ids`. It also consumes the original `chat_request`/`completion_request` and `prompt_tokens` carried over from the render step (see [Request format](#request-format)) so the tool and reasoning parsers have the context they need. +The derender step needs more than the engine's `token_ids`. It also consumes the original `chat_request`/`completion_request` and `prompt_tokens` carried over from the render step (see [Request format](#request-format)) so the tool and reasoning parsers have the context they need. The chat streaming path additionally needs `prompt_token_ids` (see [Streaming cost](#streaming-cost)) when a parser is configured. ## API Reference @@ -52,6 +52,21 @@ Each request wraps the engine's `GenerateResponse`(s) together with the caller m Oversized payloads are rejected with a `400` before any `tokenizer.decode()` or parser runs. +## Streaming cost + +Streaming derender threads a client carried `stream_state` across per-chunk calls instead of keeping session state on the server. + +Plain detokenization (no parser configured) carries only a small, bounded incremental decode window in `stream_state` independent of generation length. + +When a tool or reasoning parser is configured, parser internal state (buffered markup, reasoning/tool phase) can't be serialized. `stream_state` therefore instead carries the full `output_token_ids` seen so far plus `output_chunk_lens`, the token count of each chunk they arrived in. Each chunk rebuilds a fresh parser and replays that history through `parse_delta`, one call per original chunk, before processing the new tokens for real. For the parser path only, this means: + +- **Transport**: `output_token_ids` and `output_chunk_lens` round-trip in full in both directions on every call. `output_chunk_lens` has one entry per chunk, which is one per token without speculative decoding. This means O(n) bytes per chunk, O(n²) bytes over a full generation. Bounded by `max_model_len`. `prompt_token_ids` is sent in full on every call too and it isn't trimmed as `output_token_ids` grows. This means that for most of a stream it dominates the per chunk payload. A 100k token prompt with 1k tokens of output means `prompt_token_ids` is ~99% of the request body on every chunk. +- **Compute**: replay is O(n) `parse_delta` calls per chunk (O(n²) per generation). `parse_delta` itself is O(n) for parsers that re-scan accumulated text (e.g. Hermes tool-call JSON, DeepSeek-R1 reasoning). The per-generation cost is O(n³) character work, not O(n²). This is a deliberately minimal first implementation with no caching layer. +- The parser path also requires `prompt_token_ids` so `parse_delta` can settle whether the prompt left reasoning open or not. Since parser state can't be carried across calls, it re-scans the full prompt once per chunk. +- Replay runs off the event loop on the renderer's executor (`renderer_num_workers`, default `1`). Size it for the expected number of concurrent parser configured streams. + +`output_token_ids` and `prompt_token_ids` are both bounded by `max_model_len` but callers streaming long reasoning traces through a parser configured model should expect materially more state transport and CPU cost than the plain detokenization path. + ## Example The example below drives the full `render → generate → derender` round trip for a chat request against a GPU less render server (`/render`, `/derender`) and a token-in / token-out engine (`/inference/v1/generate`). @@ -102,4 +117,4 @@ with httpx.Client(timeout=60.0) as client: print(response["choices"][0]["message"]["content"]) ``` -Passing `chat_request` lets the derenderer run the configured tool and reasoning parsers. This means `response["choices"][0]["message"]` carries the same `content` / `reasoning` / `tool_calls` split a `vllm serve` server would produce. Omit `chat_request` for plain detokenization only. +Passing `chat_request` lets the derenderer run the configured tool and reasoning parsers. This means `response["choices"][0]["message"]` carries the same `content` / `reasoning` / `tool_calls` split a `vllm serve` server would produce. `chat_request` can only be omitted for a model with no tool or reasoning parser configured. A parser configured model rejects a missing `chat_request` with a 400 rather than silently falling back to plain detokenization. diff --git a/tests/entrypoints/scale_out/derender/test_derender.py b/tests/entrypoints/scale_out/derender/test_derender.py index d7e0298ac3..9cc6124393 100644 --- a/tests/entrypoints/scale_out/derender/test_derender.py +++ b/tests/entrypoints/scale_out/derender/test_derender.py @@ -800,7 +800,12 @@ def _e2e_generate_response( @pytest.mark.asyncio async def test_e2e_plain_roundtrip(parser_client, parser_tokenizer): - """Plain text without reasoning markers roundtrips correctly.""" + """Plain text without reasoning markers roundtrips correctly. + + Markerless output has no ````, which deepseek_r1 classifies + wholly as reasoning, so the text lands there rather than in content. + What this pins is detokenization fidelity through the parser path. + """ messages = [{"role": "user", "content": "What is 2+2?"}] gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages) @@ -814,16 +819,22 @@ async def test_e2e_plain_roundtrip(parser_client, parser_tokenizer): "model": PARSER_MODEL, "generate_response": _e2e_generate_response(output_ids), "prompt_tokens": len(gen_req["token_ids"]), + "chat_request": {"model": PARSER_MODEL, "messages": messages}, }, ) assert resp.status_code == 200, resp.text - content = resp.json()["choices"][0]["message"]["content"] - assert content == expected + message = resp.json()["choices"][0]["message"] + assert message["reasoning"] == expected + assert message["content"] is None @pytest.mark.asyncio async def test_e2e_token_identity(parser_client, parser_tokenizer): - """encode(derender(token_ids)) == token_ids (RL invariant).""" + """encode(derender(token_ids)) == token_ids (RL invariant). + + Markerless output comes back as reasoning (see + ``test_e2e_plain_roundtrip``), so re-encode that. + """ messages = [{"role": "user", "content": "Hi"}] gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages) @@ -836,17 +847,22 @@ async def test_e2e_token_identity(parser_client, parser_tokenizer): "model": PARSER_MODEL, "generate_response": _e2e_generate_response(output_ids), "prompt_tokens": len(gen_req["token_ids"]), + "chat_request": {"model": PARSER_MODEL, "messages": messages}, }, ) assert resp.status_code == 200 - content = resp.json()["choices"][0]["message"]["content"] - re_encoded = _encode(parser_tokenizer, content) + reasoning = resp.json()["choices"][0]["message"]["reasoning"] + re_encoded = _encode(parser_tokenizer, reasoning) assert output_ids == re_encoded @pytest.mark.asyncio async def test_e2e_non_ascii_roundtrip(parser_client, parser_tokenizer): - """CJK + emoji roundtrip without U+FFFD.""" + """CJK + emoji roundtrip without U+FFFD. + + Markerless output comes back as reasoning (see + ``test_e2e_plain_roundtrip``). + """ messages = [{"role": "user", "content": "Reply in Chinese"}] gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages) @@ -859,11 +875,12 @@ async def test_e2e_non_ascii_roundtrip(parser_client, parser_tokenizer): "model": PARSER_MODEL, "generate_response": _e2e_generate_response(output_ids), "prompt_tokens": len(gen_req["token_ids"]), + "chat_request": {"model": PARSER_MODEL, "messages": messages}, }, ) assert resp.status_code == 200 - content = resp.json()["choices"][0]["message"]["content"] - assert "�" not in content + reasoning = resp.json()["choices"][0]["message"]["reasoning"] + assert "�" not in reasoning @pytest.mark.asyncio @@ -976,8 +993,10 @@ async def test_e2e_parsed_reasoning_and_tool_call(parser_client, parser_tokenize @pytest.mark.asyncio -async def test_e2e_no_chat_request_fallback(parser_client, parser_tokenizer): - """Without chat_request, derender falls back to plain detokenization.""" +async def test_e2e_no_chat_request_rejected(parser_client, parser_tokenizer): + """Without chat_request a parser configured model rejects with 400 + rather than silently falling back to plain detokenization. This is to + prevent the leak of raw reasoning/tool markup into content.""" messages = [{"role": "user", "content": "Hello"}] gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages) @@ -992,9 +1011,8 @@ async def test_e2e_no_chat_request_fallback(parser_client, parser_tokenizer): "prompt_tokens": len(gen_req["token_ids"]), }, ) - assert resp.status_code == 200 - content = resp.json()["choices"][0]["message"]["content"] - assert "Hi" in content + assert resp.status_code == 400 + assert "chat_request" in resp.json()["error"]["message"] # --------------------------------------------------------------------------- diff --git a/tests/entrypoints/scale_out/derender/test_derender_stream.py b/tests/entrypoints/scale_out/derender/test_derender_stream.py index df10b31d28..4032d9cbf3 100644 --- a/tests/entrypoints/scale_out/derender/test_derender_stream.py +++ b/tests/entrypoints/scale_out/derender/test_derender_stream.py @@ -8,6 +8,8 @@ Tests are split into two layers: 1. Unit tests (no server): covers ``_detokenize_delta`` correctness (chunked == one-shot) and ``derender_completion_stream`` / ``derender_chat_stream`` logic via a real tokenizer on a tiny model. + The parser path is covered both with a deterministic stub parser and + with the real ``HarmonyParser`` (skipped without ``openai_harmony``). 2. Integration tests (require a running render server): covers the full HTTP round-trip through the streaming endpoint. Marked with @@ -15,19 +17,142 @@ Tests are split into two layers: fixtures from the sibling ``test_derender.py``. """ +import json +from collections.abc import Callable + import pytest import pytest_asyncio +from vllm.entrypoints.generate.base.protocol import ( + DeltaFunctionCall, + DeltaMessage, + DeltaToolCall, +) from vllm.entrypoints.scale_out.token_in_token_out.protocol import ( DerenderStreamState, + GenerateResponse, + GenerateResponseChoice, GenerateResponseStreamChoice, GenerateStreamResponse, ) from vllm.entrypoints.serve.engine.protocol import ErrorResponse, UsageInfo +from vllm.parser import Parser +from vllm.utils import random_uuid MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM" +def _merge_fake_deltas( + accumulated: DeltaMessage | None, new: DeltaMessage | None +) -> DeltaMessage | None: + """Fold one token's worth of `_FakeParser` output into a per call + accumulator, the same way a real parser merges multiple tokens' + effects into the single `DeltaMessage` it returns per `parse_delta` + call. + """ + if new is None: + return accumulated + if accumulated is None: + accumulated = DeltaMessage() + + if new.content: + accumulated.content = (accumulated.content or "") + new.content + if new.reasoning: + accumulated.reasoning = (accumulated.reasoning or "") + new.reasoning + + for tc in new.tool_calls: + starts_new_call = tc.id is not None or ( + tc.function is not None and tc.function.name is not None + ) + existing = ( + None + if starts_new_call + else next((t for t in accumulated.tool_calls if t.index == tc.index), None) + ) + if existing is None: + accumulated.tool_calls.append(tc) + continue + if existing.function is None: + existing.function = DeltaFunctionCall() + if tc.function is not None and tc.function.arguments: + existing.function.arguments = ( + existing.function.arguments or "" + ) + tc.function.arguments + + return accumulated + + +class _FakeParser(Parser): + """Deterministic `Parser` stub for unit testing the replay/merge/pin + mechanics of `OnlineDerenderer._derender_chat_stream_parsed` in + isolation from any real reasoning/tool parser's markup grammar. + + Processes every id in `delta_token_ids` in order and folds their + effects into one `DeltaMessage`, like a real parser handling a + multi-token delta in a single call. Ignores `delta_text`: + + - `TOOL_START` opens a new tool call at index 0 (id + name). + - `TOOL_ARG` appends one `"a"` to that tool call's arguments. + - `REASON` emits one `"r"` of reasoning (suppressed when the request + has `include_reasoning=False`, mirroring the real parsers). + - `CONTENT` emits one `"c"` of content. + - An empty `delta_token_ids` with `finished=True` (the finish only + flush call) emits a `"FLUSH"` content sentinel so tests can confirm + it happened. + - Anything else emits nothing. + """ + + TOOL_START = 9 + TOOL_ARG = 10 + REASON = 11 + CONTENT = 12 + + def parse_delta( + self, + delta_text, + delta_token_ids, + request, + prompt_token_ids=None, + *, + finished, + ): + if not delta_token_ids: + return DeltaMessage(content="FLUSH") if finished else None + + accumulated: DeltaMessage | None = None + for tok in delta_token_ids: + accumulated = _merge_fake_deltas( + accumulated, self._parse_one_token(tok, request) + ) + return accumulated + + def _parse_one_token(self, tok: int, request) -> DeltaMessage | None: + if tok == self.TOOL_START: + return DeltaMessage( + tool_calls=[ + DeltaToolCall( + id=f"call-{random_uuid()}", + type="function", + function=DeltaFunctionCall(name="get_weather", arguments=""), + index=0, + ) + ] + ) + if tok == self.TOOL_ARG: + return DeltaMessage( + tool_calls=[ + DeltaToolCall(index=0, function=DeltaFunctionCall(arguments="a")) + ] + ) + if tok == self.REASON: + if not request.include_reasoning: + return None + return DeltaMessage(reasoning="r") + if tok == self.CONTENT: + return DeltaMessage(content="c") + return None + + # --------------------------------------------------------------------------- # Helpers shared across tests # --------------------------------------------------------------------------- @@ -111,6 +236,40 @@ def derenderer(tokenizer): ) +@pytest.fixture(scope="module") +def parsed_derenderer(tokenizer): + """OnlineDerenderer with `_FakeParser` wired in as `self.parser`. + + The parser configured path runs `_derender_chat_stream_parsed` via + `make_async(..., executor=renderer._executor)`, so unlike + `derenderer` above (whose streaming paths never touch the executor), + `renderer._executor` must be a real `ThreadPoolExecutor` here. + `loop.run_in_executor` cannot submit work to a `MagicMock`. + """ + from concurrent.futures import ThreadPoolExecutor + from unittest.mock import MagicMock + + from vllm.renderers.online_derenderer import OnlineDerenderer + + renderer = MagicMock() + renderer.get_tokenizer.return_value = tokenizer + renderer._executor = ThreadPoolExecutor(max_workers=2) + + model_config = MagicMock() + model_config.hf_config.model_type = "llama" + model_config.model = MODEL_NAME + + dr = OnlineDerenderer( + model_config=model_config, + renderer=renderer, + request_logger=None, + chat_template=None, + chat_template_content_format="string", + ) + dr.parser = _FakeParser + return dr + + class TestDetokenizeDelta: """_detokenize_delta: chunked decode must equal one shot decode.""" @@ -378,49 +537,456 @@ class TestDerenderChatStream: assert streamed == one_shot @pytest.mark.asyncio - @pytest.mark.parametrize("with_chat_request", [True, False]) - async def test_parser_raises_not_implemented(self, tokenizer, with_chat_request): - """Stream chat with a parser active must fail closed (NotImplementedError). - - Checks that a parser configured model must 501 even when ``chat_request`` is - omitted, otherwise reasoning/tool markup would leak into ``delta.content` - via the plain detok fallback. - """ - from unittest.mock import MagicMock - - from vllm.renderers.online_derenderer import OnlineDerenderer - - renderer = MagicMock() - renderer.get_tokenizer.return_value = tokenizer - - model_config = MagicMock() - model_config.hf_config.model_type = "llama" - model_config.model = MODEL_NAME - - # Construct a derenderer WITH a tool/reasoning parser active. - # ParserManager.get_parser returns None when no parser is named, so - # we inject a mock parser class directly. - dr = OnlineDerenderer( - model_config=model_config, - renderer=renderer, - request_logger=None, - chat_template=None, - chat_template_content_format="string", - ) - dr.parser = MagicMock() # simulate a parser being active - - chat_request = MagicMock() if with_chat_request else None - token_ids = tokenizer.encode("hello")[:3] - - with pytest.raises(NotImplementedError): - await dr.derender_chat_stream( + async def test_parser_configured_missing_chat_request_raises( + self, parsed_derenderer + ): + """A parser configured model must never fall through to plain detok + even when `chat_request` is omitted. If allowed this would leak raw + reasoning/tool markup into `delta.content`. `ServingDerender` + pre-checks this too (400 before touching the tokenizer). This pins + the `OnlineDerenderer` level backstop.""" + with pytest.raises(ValueError, match="chat_request"): + await parsed_derenderer.derender_chat_stream( model=MODEL_NAME, - generate_chunk=_make_stream_chunk(token_ids), + generate_chunk=_make_stream_chunk([_FakeParser.CONTENT]), state=None, - chat_request=chat_request, + chat_request=None, ) +def _chat_request(**kwargs): + from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest + + kwargs.setdefault("messages", [{"role": "user", "content": "hi"}]) + kwargs.setdefault("model", MODEL_NAME) + return ChatCompletionRequest(**kwargs) + + +class TestDerenderChatStreamParsed: + """derender_chat_stream: parser branch (replay + parse_delta) exercised + against the deterministic `_FakeParser` so these pin OnlineDerenderer's + own replay/merge/pin/finish_reason logic independent of any real + parser's markup grammar (covered separately by the parser_server backed + integration tests below).""" + + @pytest.mark.asyncio + async def test_dispatches_and_emits_content(self, parsed_derenderer): + chunk, state = await parsed_derenderer.derender_chat_stream( + model=MODEL_NAME, + generate_chunk=_make_stream_chunk( + [_FakeParser.CONTENT], finish_reason="stop" + ), + chat_request=_chat_request(), + ) + assert chunk.choices[0].delta.content == "c" + assert chunk.choices[0].delta.role == "assistant" + assert chunk.choices[0].finish_reason == "stop" + assert state.output_token_ids == [_FakeParser.CONTENT] + + @pytest.mark.asyncio + async def test_role_sent_once(self, parsed_derenderer): + chat_request = _chat_request() + chunk1, state = await parsed_derenderer.derender_chat_stream( + model=MODEL_NAME, + generate_chunk=_make_stream_chunk([_FakeParser.CONTENT]), + chat_request=chat_request, + ) + chunk2, _ = await parsed_derenderer.derender_chat_stream( + model=MODEL_NAME, + generate_chunk=_make_stream_chunk( + [_FakeParser.CONTENT], finish_reason="stop" + ), + state=state, + chat_request=chat_request, + ) + assert chunk1.choices[0].delta.role == "assistant" + assert chunk2.choices[0].delta.role is None + + @pytest.mark.asyncio + async def test_finish_only_chunk_flushes(self, parsed_derenderer): + """A finish only chunk (no new tokens) still calls `parse_delta` + once with `finished=True` to flush buffered state.""" + chat_request = _chat_request() + _, state = await parsed_derenderer.derender_chat_stream( + model=MODEL_NAME, + generate_chunk=_make_stream_chunk([_FakeParser.CONTENT]), + chat_request=chat_request, + ) + chunk2, _ = await parsed_derenderer.derender_chat_stream( + model=MODEL_NAME, + generate_chunk=_make_stream_chunk([], finish_reason="stop"), + state=state, + chat_request=chat_request, + ) + assert chunk2.choices[0].delta.content == "FLUSH" + assert chunk2.choices[0].finish_reason == "stop" + + @pytest.mark.asyncio + async def test_chunking_invariance(self, parsed_derenderer): + """1 token per chunk vs a single whole chunk assemble identically. + + Every chunk (live or replayed) is fed to `parse_delta` at its own + producer granularity (see `_derender_chat_stream_parsed`'s + docstring). This proves that choice doesn't change the assembled + output for a well behaved incremental parser. Each call's effects + fold together the same way regardless of how many tokens land in + one call vs. many.""" + token_ids = [ + _FakeParser.REASON, + _FakeParser.REASON, + _FakeParser.TOOL_START, + _FakeParser.TOOL_ARG, + _FakeParser.TOOL_ARG, + _FakeParser.CONTENT, + ] + chat_request = _chat_request( + tools=[{"type": "function", "function": {"name": "get_weather"}}], + tool_choice="auto", + include_reasoning=True, + ) + + async def _assemble(chunks: list[list[int]]) -> dict: + state = None + content = "" + reasoning = "" + tool_args = "" + for i, tids in enumerate(chunks): + finish = "stop" if i == len(chunks) - 1 else None + chunk, state = await parsed_derenderer.derender_chat_stream( + model=MODEL_NAME, + generate_chunk=_make_stream_chunk(tids, finish_reason=finish), + state=state, + chat_request=chat_request, + ) + delta = chunk.choices[0].delta + content += delta.content or "" + reasoning += delta.reasoning or "" + for tc in delta.tool_calls: + if tc.function and tc.function.arguments: + tool_args += tc.function.arguments + return {"content": content, "reasoning": reasoning, "tool_args": tool_args} + + whole = await _assemble([token_ids]) + one_at_a_time = await _assemble([[t] for t in token_ids]) + + assert whole == one_at_a_time + assert whole == {"content": "c", "reasoning": "rr", "tool_args": "aa"} + + @pytest.mark.asyncio + async def test_replay_preserves_chunk_boundaries( + self, parsed_derenderer, monkeypatch + ): + """A multi-token chunk (e.g. a speculative decoding step) must be + replayed as one `parse_delta` call, not one per token. Parsers + with per-call state such as `history_tool_call_cnt` would + otherwise diverge from standard serving.""" + + class _CallCountingParser(_FakeParser): + """Emits how many `parse_delta` calls it has seen.""" + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + self.num_calls = 0 + + def parse_delta( + self, + delta_text, + delta_token_ids, + request, + prompt_token_ids=None, + *, + finished, + ): + self.num_calls += 1 + return DeltaMessage(content=str(self.num_calls)) + + monkeypatch.setattr(parsed_derenderer, "parser", _CallCountingParser) + chat_request = _chat_request() + + _, state = await parsed_derenderer.derender_chat_stream( + model=MODEL_NAME, + generate_chunk=_make_stream_chunk([_FakeParser.CONTENT] * 3), + chat_request=chat_request, + ) + assert state.output_chunk_lens == [3] + + chunk, _ = await parsed_derenderer.derender_chat_stream( + model=MODEL_NAME, + generate_chunk=_make_stream_chunk( + [_FakeParser.CONTENT], finish_reason="stop" + ), + state=state, + chat_request=chat_request, + ) + assert chunk.choices[0].delta.content == "2" + + @pytest.mark.asyncio + async def test_tool_call_id_pinned_across_chunks(self, parsed_derenderer): + """Once an index's ID is recorded in `last_tool_call_ids`, a later + id bearing delta for that same index is pinned to the recorded + value rather than a freshly (re-)generated one.""" + chat_request = _chat_request( + tools=[{"type": "function", "function": {"name": "get_weather"}}], + tool_choice="auto", + ) + chunk1, state = await parsed_derenderer.derender_chat_stream( + model=MODEL_NAME, + generate_chunk=_make_stream_chunk([_FakeParser.TOOL_START]), + chat_request=chat_request, + ) + first_id = chunk1.choices[0].delta.tool_calls[0].id + assert first_id is not None + assert state.last_tool_call_ids == [first_id] + + chunk2, state = await parsed_derenderer.derender_chat_stream( + model=MODEL_NAME, + generate_chunk=_make_stream_chunk( + [_FakeParser.TOOL_START], finish_reason="stop" + ), + state=state, + chat_request=chat_request, + ) + assert chunk2.choices[0].delta.tool_calls[0].id == first_id + assert state.last_tool_call_ids == [first_id] + + @pytest.mark.asyncio + async def test_finish_reason_rewritten_to_tool_calls(self, parsed_derenderer): + chat_request = _chat_request( + tools=[{"type": "function", "function": {"name": "get_weather"}}], + tool_choice="auto", + ) + chunk, _ = await parsed_derenderer.derender_chat_stream( + model=MODEL_NAME, + generate_chunk=_make_stream_chunk( + [_FakeParser.TOOL_START, _FakeParser.TOOL_ARG], finish_reason="stop" + ), + chat_request=chat_request, + ) + assert chunk.choices[0].finish_reason == "tool_calls" + + @pytest.mark.asyncio + async def test_finish_reason_stop_for_named_tool_choice(self, parsed_derenderer): + # ChatCompletionRequest's `check_tool_usage` is a mode="before" + # validator, so it sees the raw value and only accepts "auto", + # "required" or a dict, never an already built + # ChatCompletionNamedToolChoiceParam. + chat_request = _chat_request( + tools=[{"type": "function", "function": {"name": "get_weather"}}], + tool_choice={"type": "function", "function": {"name": "get_weather"}}, + ) + chunk, _ = await parsed_derenderer.derender_chat_stream( + model=MODEL_NAME, + generate_chunk=_make_stream_chunk( + [_FakeParser.TOOL_START, _FakeParser.TOOL_ARG], finish_reason="stop" + ), + chat_request=chat_request, + ) + assert chunk.choices[0].finish_reason == "stop" + + @pytest.mark.asyncio + async def test_include_reasoning_false_suppresses_reasoning( + self, parsed_derenderer + ): + chat_request = _chat_request(include_reasoning=False) + chunk, _ = await parsed_derenderer.derender_chat_stream( + model=MODEL_NAME, + generate_chunk=_make_stream_chunk( + [_FakeParser.REASON, _FakeParser.CONTENT], finish_reason="stop" + ), + chat_request=chat_request, + ) + delta = chunk.choices[0].delta + assert delta.reasoning is None + assert delta.content == "c" + + +# --------------------------------------------------------------------------- +# Harmony / GPT-OSS replay — unit, no server +# --------------------------------------------------------------------------- + +HARMONY_MODEL = "openai/gpt-oss-20b" +HARMONY_REASONING = "The user wants 2 plus 3." +HARMONY_ANSWER = "The answer is 5." +HARMONY_PROMPT = "<|start|>user<|message|>Add 2 and 3.<|end|><|start|>assistant" +HARMONY_OUTPUT = ( + f"<|channel|>analysis<|message|>{HARMONY_REASONING}<|end|>" + f"<|start|>assistant<|channel|>final<|message|>{HARMONY_ANSWER}<|return|>" +) + + +@pytest.fixture(scope="module") +def harmony_encode(): + """Encoder for canned GPT-OSS harmony token sequences.""" + pytest.importorskip("openai_harmony") + + # Pre-caches the o200k_base BPE file that openai-harmony's Rust backend + # downloads on first use, same as the sibling suite's E2E harmony tests. + from tests.entrypoints.scale_out.derender.test_derender import ( + _ensure_harmony_vocab, + ) + from vllm.entrypoints.openai.parser.harmony_utils import get_encoding + + _ensure_harmony_vocab() + encoding = get_encoding() + + def _encode(harmony_str: str) -> list[int]: + return encoding.encode(harmony_str, allowed_special="all") + + return _encode + + +@pytest.fixture(scope="module") +def harmony_tokenizer(): + pytest.importorskip("openai_harmony") + from vllm.tokenizers import get_tokenizer + + return get_tokenizer(HARMONY_MODEL, trust_remote_code=True) + + +@pytest.fixture(scope="module") +def harmony_derenderer(harmony_tokenizer): + """OnlineDerenderer whose parser resolves to the real `HarmonyParser`. + + Same shape as `parsed_derenderer` (mocked renderer, real executor) but + with a real tokenizer and parser, so the replay path is exercised + against Harmony's actual channel grammar rather than a stub. + """ + from concurrent.futures import ThreadPoolExecutor + from unittest.mock import MagicMock + + from vllm.parser.harmony import HarmonyParser + from vllm.renderers.online_derenderer import OnlineDerenderer + + renderer = MagicMock() + renderer.get_tokenizer.return_value = harmony_tokenizer + renderer._executor = ThreadPoolExecutor(max_workers=2) + + model_config = MagicMock() + model_config.model = HARMONY_MODEL + model_config.hf_config.model_type = "gpt_oss" + model_config.hf_text_config.model_type = "gpt_oss" + model_config.hf_overrides = None + + dr = OnlineDerenderer( + model_config=model_config, + renderer=renderer, + request_logger=None, + chat_template=None, + chat_template_content_format="string", + enable_auto_tools=True, + tool_parser="openai", + reasoning_parser="openai_gptoss", + ) + assert dr.use_harmony + assert dr.parser is HarmonyParser + return dr + + +async def _stream_harmony_deltas( + derenderer, + chat_request, + output_ids: list[int], + prompt_ids: list[int], + chunk_size: int, +) -> list[DeltaMessage]: + """Drive `output_ids` through the parser path in fixed size chunks. + + Returns the per chunk `DeltaMessage`s in order so callers can assert on + intermediate emissions, not just the assembled result. + """ + deltas: list[DeltaMessage] = [] + state = None + for start in range(0, len(output_ids), chunk_size): + tids = output_ids[start : start + chunk_size] + is_last = start + chunk_size >= len(output_ids) + chunk, state = await derenderer.derender_chat_stream( + model=HARMONY_MODEL, + generate_chunk=_make_stream_chunk( + tids, finish_reason="stop" if is_last else None + ), + state=state, + chat_request=chat_request, + prompt_token_ids=prompt_ids, + ) + deltas.append(chunk.choices[0].delta) + return deltas + + +class TestDerenderChatStreamHarmony: + """derender_chat_stream: replay + `parse_delta` against real HarmonyParser. + + Skipped where `openai_harmony` is not installed. `prompt_token_ids` is + threaded through for call shape fidelity only, since HarmonyParser + derives its state from the output tokens alone. + """ + + @pytest.mark.asyncio + async def test_reasoning_never_leaks_as_content_midstream( + self, harmony_derenderer, harmony_encode + ): + """Regression guard for the RFC's original replay + diff design. + + `HarmonyParser.parse()` always flushes to EOS. Mid-stream that + raises `HarmonyError` and the recovery branch re-emits the in + flight message on the `final` channel, so partial analysis surfaces + as content. Replay + `parse_delta` must never do that. Driven one + token per chunk, the finest granularity a client can produce. + """ + output_ids = harmony_encode(HARMONY_OUTPUT) + prompt_ids = harmony_encode(HARMONY_PROMPT) + chat_request = _chat_request(model=HARMONY_MODEL, include_reasoning=True) + + deltas = await _stream_harmony_deltas( + harmony_derenderer, chat_request, output_ids, prompt_ids, chunk_size=1 + ) + + reasoning = "" + content = "" + for i, delta in enumerate(deltas): + reasoning += delta.reasoning or "" + content += delta.content or "" + # Any analysis text emitted as content breaks the prefix + # property, since the two channels share no prefix here. + assert HARMONY_ANSWER.startswith(content), ( + f"content after chunk {i} is not a prefix of the final " + f"channel text: {content!r}" + ) + assert HARMONY_REASONING.startswith(reasoning) + assert not delta.tool_calls + + assert reasoning == HARMONY_REASONING + assert content == HARMONY_ANSWER + + @pytest.mark.asyncio + async def test_stream_matches_batch(self, harmony_derenderer, harmony_encode): + """Streamed assembly equals one shot `/derender` over the same IDs.""" + output_ids = harmony_encode(HARMONY_OUTPUT) + prompt_ids = harmony_encode(HARMONY_PROMPT) + chat_request = _chat_request(model=HARMONY_MODEL, include_reasoning=True) + + deltas = await _stream_harmony_deltas( + harmony_derenderer, chat_request, output_ids, prompt_ids, chunk_size=3 + ) + reasoning = "".join(d.reasoning or "" for d in deltas) + content = "".join(d.content or "" for d in deltas) + + batch_choices = await harmony_derenderer.derender_chat( + GenerateResponse( + request_id="test-harmony-batch", + choices=[ + GenerateResponseChoice( + index=0, token_ids=output_ids, finish_reason="stop" + ) + ], + ), + chat_request, + ) + message = batch_choices[0].message + + assert reasoning == message.reasoning + assert content == message.content + + class TestDerenderStreamStateValidation: """DerenderStreamState rejects malformed caller supplied offsets/lengths.""" @@ -446,6 +1012,24 @@ class TestDerenderStreamStateValidation: state = DerenderStreamState(prev_tokens=["a"] * 1024) assert len(state.prev_tokens) == 1024 + def test_output_chunk_lens_mismatch_rejected(self): + from pydantic import ValidationError + + with pytest.raises(ValidationError): + DerenderStreamState(output_token_ids=[1, 2, 3], output_chunk_lens=[1, 1]) + + def test_output_chunk_lens_zero_entry_rejected(self): + from pydantic import ValidationError + + with pytest.raises(ValidationError): + DerenderStreamState(output_token_ids=[1, 2], output_chunk_lens=[0, 2]) + + def test_output_chunk_lens_matching_accepted(self): + state = DerenderStreamState( + output_token_ids=[1, 2, 3], output_chunk_lens=[1, 2] + ) + assert state.output_chunk_lens == [1, 2] + class TestServingDerenderStreamErrorHandling: """Malformed stream_state must surface as 400 and not an unhandled 500.""" @@ -458,8 +1042,10 @@ class TestServingDerenderStreamErrorHandling: models = MagicMock() models.is_base_model.return_value = True models.model_config = MagicMock() + models.model_config.max_model_len = 100_000 online_derenderer = MagicMock() + online_derenderer.parser = None online_derenderer.derender_completion_stream = AsyncMock( side_effect=side_effect ) @@ -504,6 +1090,179 @@ class TestServingDerenderStreamErrorHandling: assert result.error.code == 400 +class TestServingDerenderStreamValidation: + """If checks cannot run in `derender_chat_stream_response`: all must + reject with 400 before ever calling into `online_derenderer` (i.e. + before touching the tokenizer).""" + + def _make_serving(self, *, parser_configured: bool, max_model_len: int = 100_000): + from unittest.mock import AsyncMock, MagicMock + + from vllm.entrypoints.openai.chat_completion.protocol import ( + ChatCompletionStreamResponse, + ) + from vllm.entrypoints.scale_out.derender.serving import ServingDerender + + models = MagicMock() + models.is_base_model.return_value = True + models.model_config = MagicMock() + models.model_config.max_model_len = max_model_len + + online_derenderer = MagicMock() + online_derenderer.parser = MagicMock() if parser_configured else None + online_derenderer.derender_chat_stream = AsyncMock( + return_value=( + ChatCompletionStreamResponse(id="t", model=MODEL_NAME, choices=[]), + DerenderStreamState(), + ) + ) + return ServingDerender(models=models, online_derenderer=online_derenderer) + + @pytest.mark.asyncio + async def test_missing_chat_request_with_parser_rejected(self): + from vllm.entrypoints.scale_out.token_in_token_out.protocol import ( + DerenderChatStreamRequest, + ) + + serving = self._make_serving(parser_configured=True) + request = DerenderChatStreamRequest( + stream=True, + model=MODEL_NAME, + generate_chunk=_make_stream_chunk([1, 2]), + ) + result = await serving.derender_chat_stream_response(request) + assert isinstance(result, ErrorResponse) + assert result.error.code == 400 + assert "chat_request" in result.error.message + serving.online_derenderer.derender_chat_stream.assert_not_called() + + @pytest.mark.asyncio + async def test_missing_chat_request_without_parser_ok(self): + from vllm.entrypoints.scale_out.token_in_token_out.protocol import ( + DerenderChatStreamRequest, + ) + + serving = self._make_serving(parser_configured=False) + request = DerenderChatStreamRequest( + stream=True, + model=MODEL_NAME, + generate_chunk=_make_stream_chunk([1, 2]), + ) + result = await serving.derender_chat_stream_response(request) + assert not isinstance(result, ErrorResponse) + + @pytest.mark.asyncio + async def test_missing_prompt_token_ids_with_parser_rejected(self): + """A parser configured model must reject a missing prompt_token_ids + the same way it rejects a missing chat_request. Without it, + parse_delta cannot tell whether the prompt left reasoning open and + would misclassify reasoning content as plain content.""" + from vllm.entrypoints.scale_out.token_in_token_out.protocol import ( + DerenderChatStreamRequest, + ) + + serving = self._make_serving(parser_configured=True) + request = DerenderChatStreamRequest( + stream=True, + model=MODEL_NAME, + generate_chunk=_make_stream_chunk([1, 2]), + chat_request=_chat_request(), + ) + result = await serving.derender_chat_stream_response(request) + assert isinstance(result, ErrorResponse) + assert result.error.code == 400 + assert "prompt_token_ids" in result.error.message + serving.online_derenderer.derender_chat_stream.assert_not_called() + + @pytest.mark.asyncio + async def test_prompt_token_ids_present_with_parser_ok(self): + from vllm.entrypoints.scale_out.token_in_token_out.protocol import ( + DerenderChatStreamRequest, + ) + + serving = self._make_serving(parser_configured=True) + request = DerenderChatStreamRequest( + stream=True, + model=MODEL_NAME, + generate_chunk=_make_stream_chunk([1, 2]), + chat_request=_chat_request(), + prompt_token_ids=[1, 2, 3], + ) + result = await serving.derender_chat_stream_response(request) + assert not isinstance(result, ErrorResponse) + + @pytest.mark.asyncio + async def test_oversized_output_token_ids_rejected(self): + from vllm.entrypoints.scale_out.token_in_token_out.protocol import ( + DerenderChatStreamRequest, + ) + + serving = self._make_serving(parser_configured=False, max_model_len=4) + request = DerenderChatStreamRequest( + stream=True, + model=MODEL_NAME, + generate_chunk=_make_stream_chunk([1, 2, 3]), + stream_state=DerenderStreamState( + output_token_ids=[1, 2], output_chunk_lens=[2] + ), + ) + result = await serving.derender_chat_stream_response(request) + assert isinstance(result, ErrorResponse) + assert result.error.code == 400 + assert "max_model_len" in result.error.message + serving.online_derenderer.derender_chat_stream.assert_not_called() + + @pytest.mark.asyncio + async def test_oversized_prompt_token_ids_rejected(self): + """prompt_token_ids is caller supplied and otherwise unbounded. A + parser configured deployment rescans it in full on every chunk + (is_reasoning_end / adjust_initial_state_from_prompt), so it must be + bounded the same way output_token_ids is.""" + from vllm.entrypoints.scale_out.token_in_token_out.protocol import ( + DerenderChatStreamRequest, + ) + + serving = self._make_serving(parser_configured=True, max_model_len=4) + request = DerenderChatStreamRequest( + stream=True, + model=MODEL_NAME, + generate_chunk=_make_stream_chunk([1]), + chat_request=_chat_request(), + prompt_token_ids=[1, 2, 3, 4, 5], + ) + result = await serving.derender_chat_stream_response(request) + assert isinstance(result, ErrorResponse) + assert result.error.code == 400 + assert "max_model_len" in result.error.message + serving.online_derenderer.derender_chat_stream.assert_not_called() + + @pytest.mark.asyncio + async def test_too_many_choices_rejected(self): + """Each streamed chunk contains at most one choice (a single + DerenderStreamState is threaded through every choice). The check + could never fire since derender_chat_stream itself rejects anything + above 1 first.""" + from vllm.entrypoints.scale_out.token_in_token_out.protocol import ( + DerenderChatStreamRequest, + ) + + serving = self._make_serving(parser_configured=False) + two_choices = GenerateStreamResponse( + request_id="t", + choices=[ + GenerateResponseStreamChoice(index=i, token_ids=[1]) for i in range(2) + ], + ) + request = DerenderChatStreamRequest( + stream=True, model=MODEL_NAME, generate_chunk=two_choices + ) + result = await serving.derender_chat_stream_response(request) + assert isinstance(result, ErrorResponse) + assert result.error.code == 400 + assert "at most one choice" in result.error.message + serving.online_derenderer.derender_chat_stream.assert_not_called() + + # --------------------------------------------------------------------------- # Integration tests — require a live render server # --------------------------------------------------------------------------- @@ -757,3 +1516,416 @@ async def test_streaming_usage_chunk(client): assert d2["chunk"]["choices"] == [] assert d2["chunk"]["usage"]["prompt_tokens"] == 10 assert d2["chunk"]["usage"]["completion_tokens"] == len(token_ids) + + +# --------------------------------------------------------------------------- +# Integration tests for parser configured (reasoning + tool calls) require a +# live render server. Mirrors the parser_server / parser_tokenizer pattern +# from test_derender.py. +# --------------------------------------------------------------------------- + +PARSER_MODEL = "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B" + +_PARSER_TOOLS = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get weather for a city", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}}, + }, + }, + } +] + + +@pytest.fixture(scope="module") +def parser_server(): + from tests.utils import RemoteLaunchRenderServer + + args = [ + "--enable-auto-tool-choice", + "--tool-call-parser", + "hermes", + "--reasoning-parser", + "deepseek_r1", + ] + with RemoteLaunchRenderServer(PARSER_MODEL, args) as remote_server: + yield remote_server + + +@pytest_asyncio.fixture +async def parser_client(parser_server): + import httpx + + async with httpx.AsyncClient( + base_url=parser_server.url_for(""), timeout=60.0 + ) as http_client: + yield http_client + + +@pytest.fixture(scope="module") +def parser_tokenizer(): + from vllm.tokenizers import get_tokenizer + + return get_tokenizer(PARSER_MODEL) + + +def _require_parser_markers(tokenizer, text: str, *markers: str) -> list[int]: + """Encode text and skip the test if any marker is lost in roundtrip.""" + ids = tokenizer.encode(text, add_special_tokens=False) + decoded = tokenizer.decode(ids, skip_special_tokens=False) + for m in markers: + if m not in decoded: + pytest.skip(f"Marker {m!r} lost in encode->decode roundtrip") + return ids + + +async def _render_parser_chat(client, messages: list[dict]) -> dict: + resp = await client.post( + "/v1/chat/completions/render", + json={"model": PARSER_MODEL, "messages": messages}, + ) + assert resp.status_code == 200, resp.text + return resp.json() + + +async def _stream_chat_derender( + client, + output_ids: list[int], + chunk_sizes: list[int], + chat_request: dict, + prompt_tokens: int, + prompt_token_ids: list[int], + on_chunk: Callable[[list[dict]], None] | None = None, +) -> dict: + """Feed `output_ids` through the streaming chat derender endpoint in + the given `chunk_sizes`, threading `stream_state` across calls and + return the assembled message. + + If `on_chunk` is given, it is called after every chunk with a + snapshot (deep copy) of the `tool_calls` accumulator so far, letting + callers assert properties of the intermediate deltas (e.g. monotonic + argument growth) rather than only the final assembled result. + """ + state = None + content = "" + reasoning = "" + tool_calls: list[dict] = [] + finish_reason = None + + pos = 0 + for i, size in enumerate(chunk_sizes): + tids = output_ids[pos : pos + size] + pos += size + is_last = i == len(chunk_sizes) - 1 + resp = await client.post( + "/v1/chat/completions/derender", + json={ + "stream": True, + "model": PARSER_MODEL, + "generate_chunk": { + "request_id": "stream-test", + "choices": [ + { + "index": 0, + "token_ids": tids, + "finish_reason": "stop" if is_last else None, + } + ], + }, + "stream_state": state, + "prompt_tokens": prompt_tokens, + "prompt_token_ids": prompt_token_ids, + "chat_request": chat_request, + }, + ) + assert resp.status_code == 200, resp.text + data = resp.json() + state = data["stream_state"] + delta = data["chunk"]["choices"][0]["delta"] + content += delta.get("content") or "" + reasoning += delta.get("reasoning") or "" + for tc in delta.get("tool_calls") or []: + idx = tc["index"] + while len(tool_calls) <= idx: + tool_calls.append({"id": None, "name": None, "arguments": ""}) + if tc.get("id"): + tool_calls[idx]["id"] = tc["id"] + fn = tc.get("function") or {} + if fn.get("name"): + tool_calls[idx]["name"] = fn["name"] + if fn.get("arguments"): + tool_calls[idx]["arguments"] += fn["arguments"] + if is_last: + finish_reason = data["chunk"]["choices"][0]["finish_reason"] + if on_chunk is not None: + on_chunk([dict(tc) for tc in tool_calls]) + + return { + "content": content or None, + "reasoning": reasoning or None, + "tool_calls": tool_calls, + "finish_reason": finish_reason, + } + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + "chunking", + [ + "whole", + "per_token", + pytest.param( + "triples", + marks=pytest.mark.xfail( + reason=" leaks into reasoning when it shares a " + "multi-token delta with reasoning text (vllm#55195)" + ), + ), + ], +) +async def test_stream_parsed_matches_batch_reasoning( + parser_client, parser_tokenizer, chunking +): + """Streamed == batch: assembled reasoning/content equal the non + streaming `/derender` result over the same token IDs, for every + chunking of the same output (chunking invariance).""" + messages = [{"role": "user", "content": "What is 2+3?"}] + gen_req = await _render_parser_chat(parser_client, messages) + + reasoning_text = "The user wants 2 plus 3. That is 5." + answer_text = "The answer is 5." + output_text = f"{reasoning_text}{answer_text}" + output_ids = _require_parser_markers(parser_tokenizer, output_text, "") + + chat_request = { + "model": PARSER_MODEL, + "messages": messages, + "include_reasoning": True, + } + + batch_resp = await parser_client.post( + "/v1/chat/completions/derender", + json={ + "model": PARSER_MODEL, + "generate_response": { + "request_id": "batch", + "choices": [ + {"index": 0, "token_ids": output_ids, "finish_reason": "stop"} + ], + }, + "prompt_tokens": len(gen_req["token_ids"]), + "chat_request": chat_request, + }, + ) + assert batch_resp.status_code == 200, batch_resp.text + batch_msg = batch_resp.json()["choices"][0]["message"] + + n = len(output_ids) + chunk_sizes = { + "whole": [n], + "per_token": [1] * n, + "triples": [3] * (n // 3) + [n % 3], + }[chunking] + chunk_sizes = [c for c in chunk_sizes if c > 0] + streamed = await _stream_chat_derender( + parser_client, + output_ids, + chunk_sizes, + chat_request, + len(gen_req["token_ids"]), + gen_req["token_ids"], + ) + assert streamed["content"] == batch_msg["content"] + assert streamed["reasoning"] == batch_msg.get("reasoning") + + +@pytest.mark.asyncio +async def test_stream_parsed_matches_batch_tool_call(parser_client, parser_tokenizer): + """Tool call name+id+arguments parity between streamed and batch, plus + the finish_reason -> "tool_calls" rewrite for auto tool choice.""" + messages = [{"role": "user", "content": "Weather in Paris?"}] + gen_req = await _render_parser_chat(parser_client, messages) + + output_text = ( + "Let me check." + '\n{"name": "get_weather", ' + '"arguments": {"city": "Paris"}}\n' + ) + output_ids = _require_parser_markers( + parser_tokenizer, output_text, "", "", "" + ) + chat_request = { + "model": PARSER_MODEL, + "messages": messages, + "tools": _PARSER_TOOLS, + "tool_choice": "auto", + } + + batch_resp = await parser_client.post( + "/v1/chat/completions/derender", + json={ + "model": PARSER_MODEL, + "generate_response": { + "request_id": "batch", + "choices": [ + {"index": 0, "token_ids": output_ids, "finish_reason": "stop"} + ], + }, + "prompt_tokens": len(gen_req["token_ids"]), + "chat_request": chat_request, + }, + ) + assert batch_resp.status_code == 200, batch_resp.text + batch_choice = batch_resp.json()["choices"][0] + if not batch_choice["message"]["tool_calls"]: + pytest.skip("Model did not emit a tool call") + + for chunk_sizes in ([len(output_ids)], [1] * len(output_ids)): + # Snapshot of tool_calls[0]["arguments"] after every chunk, to check + # it only ever grows by appending. It never retracts or duplicates + # already streamed text. That's the exact failure mode the + # replay + parse_delta design (over diff-based streaming) exists to + # avoid. Parity with the batch result alone wouldn't catch a + # transient mid-stream regression. + arg_snapshots: list[str] = [] + + def _record( + tool_calls: list[dict], _snapshots: list[str] = arg_snapshots + ) -> None: + if tool_calls and tool_calls[0]["arguments"]: + _snapshots.append(tool_calls[0]["arguments"]) + + streamed = await _stream_chat_derender( + parser_client, + output_ids, + chunk_sizes, + chat_request, + len(gen_req["token_ids"]), + gen_req["token_ids"], + on_chunk=_record, + ) + assert streamed["tool_calls"] + assert streamed["tool_calls"][0]["name"] == "get_weather" + assert streamed["tool_calls"][0]["id"] is not None + assert json.loads(streamed["tool_calls"][0]["arguments"]) == json.loads( + batch_choice["message"]["tool_calls"][0]["function"]["arguments"] + ) + assert streamed["finish_reason"] == "tool_calls" + + assert arg_snapshots, "expected at least one tool-call argument delta" + for prev, curr in zip(arg_snapshots, arg_snapshots[1:]): + assert curr.startswith(prev), ( + f"tool-call arguments regressed: {prev!r} -> {curr!r}" + ) + assert arg_snapshots[-1] == streamed["tool_calls"][0]["arguments"] + + +@pytest.mark.asyncio +async def test_stream_parsed_cjk_across_chunk_boundaries( + parser_client, parser_tokenizer +): + """CJK/emoji tokens split across chunk boundaries with a parser active + across the reasoning -> content transition (parser path regression + guard analogous to #46159 in the plain detok path).""" + messages = [{"role": "user", "content": "Reply in Chinese"}] + gen_req = await _render_parser_chat(parser_client, messages) + + reasoning_text = "思考中" + answer_text = "你好世界 😀" + output_text = f"{reasoning_text}{answer_text}" + output_ids = _require_parser_markers(parser_tokenizer, output_text, "") + + chat_request = { + "model": PARSER_MODEL, + "messages": messages, + "include_reasoning": True, + } + streamed = await _stream_chat_derender( + parser_client, + output_ids, + [1] * len(output_ids), + chat_request, + len(gen_req["token_ids"]), + gen_req["token_ids"], + ) + assert "�" not in (streamed["content"] or "") + assert "�" not in (streamed["reasoning"] or "") + assert answer_text in (streamed["content"] or "") + + +@pytest.mark.asyncio +async def test_stream_parsed_include_reasoning_false(parser_client, parser_tokenizer): + """include_reasoning=False emits no reasoning deltas on the streaming + parser path.""" + messages = [{"role": "user", "content": "What is 2+3?"}] + gen_req = await _render_parser_chat(parser_client, messages) + + output_text = "reasoning hereThe answer is 5." + output_ids = _require_parser_markers(parser_tokenizer, output_text, "") + + chat_request = { + "model": PARSER_MODEL, + "messages": messages, + "include_reasoning": False, + } + streamed = await _stream_chat_derender( + parser_client, + output_ids, + [1] * len(output_ids), + chat_request, + len(gen_req["token_ids"]), + gen_req["token_ids"], + ) + assert streamed["reasoning"] is None + + +@pytest.mark.asyncio +async def test_stream_parsed_missing_chat_request_rejected(parser_client): + """Parser configured + no chat_request on the streaming endpoint -> 400 + (the live-server counterpart to the mocked + `TestServingDerenderStreamValidation` checks above).""" + resp = await parser_client.post( + "/v1/chat/completions/derender", + json={ + "stream": True, + "model": PARSER_MODEL, + "generate_chunk": { + "request_id": "reject-test", + "choices": [{"index": 0, "token_ids": [1, 2], "finish_reason": None}], + }, + }, + ) + assert resp.status_code == 400 + assert "chat_request" in resp.json()["error"]["message"] + + +@pytest.mark.asyncio +async def test_stream_parsed_missing_prompt_token_ids_rejected(parser_client): + """Parser configured + chat_request given but no prompt_token_ids on the + streaming endpoint -> 400 (the live-server counterpart to the mocked + `test_missing_prompt_token_ids_with_parser_rejected` above). Without + prompt_token_ids, parse_delta cannot tell whether the prompt left + reasoning open and would silently misclassify reasoning as content.""" + messages = [{"role": "user", "content": "Hello"}] + resp = await parser_client.post( + "/v1/chat/completions/derender", + json={ + "stream": True, + "model": PARSER_MODEL, + "generate_chunk": { + "request_id": "reject-test", + "choices": [{"index": 0, "token_ids": [1, 2], "finish_reason": None}], + }, + "chat_request": { + "model": PARSER_MODEL, + "messages": messages, + "include_reasoning": True, + }, + }, + ) + assert resp.status_code == 400 + assert "prompt_token_ids" in resp.json()["error"]["message"] diff --git a/vllm/entrypoints/scale_out/derender/serving.py b/vllm/entrypoints/scale_out/derender/serving.py index 81e919d840..d3c5f3aac1 100644 --- a/vllm/entrypoints/scale_out/derender/serving.py +++ b/vllm/entrypoints/scale_out/derender/serving.py @@ -138,6 +138,13 @@ class ServingDerender(BaseServing): if bounds_error is not None: return bounds_error + if self.online_derenderer.parser is not None and request.chat_request is None: + return self.create_error_response( + "chat_request is required when a tool or reasoning parser is " + "configured because plain detokenization would leak raw parser " + "markup into content." + ) + try: choices = await self.online_derenderer.derender_chat( request.generate_response, request.chat_request @@ -248,14 +255,67 @@ class ServingDerender(BaseServing): Processes one ``GenerateStreamResponse`` chunk and returns the derendered chunk together with the updated client carried state. - - ``parser is None`` or no ``chat_request`` until reasoning/tool call - functionality added in future PR. """ error_check_ret = await self._check_model(request) if error_check_ret is not None: return error_check_ret + if self.online_derenderer.parser is not None and request.chat_request is None: + return self.create_error_response( + "chat_request is required when a tool or reasoning parser is " + "configured because plain detokenization would leak raw parser " + "markup into content." + ) + + if ( + self.online_derenderer.parser is not None + and request.prompt_token_ids is None + ): + return self.create_error_response( + "prompt_token_ids is required when a tool or reasoning " + "parser is configured. Without it parse_delta cannot tell " + "whether the prompt left reasoning open (e.g. a chat " + "template that pre-opens ) and would misclassify " + "reasoning content as plain content." + ) + + # Each streamed chunk contains at most one choice per SSE event. + # A single DerenderStreamState is threaded + # through every choice in the chunk, so >1 would corrupt detok/parser + # state across choices. See the matching check in + # OnlineDerenderer.derender_chat_stream which this fails ahead of to + # avoid touching the tokenizer at all on a malformed chunk. + num_choices = len(request.generate_chunk.choices) + if num_choices > 1: + return self.create_error_response( + f"derender_chat_stream expects at most one choice per chunk " + f"(got {num_choices})." + ) + + # prompt_token_ids is folded in here too. It is caller supplied and + # otherwise unbounded. A parser configured deployment rescans it + # in full on every chunk (is_reasoning_end / + # adjust_initial_state_from_prompt, see abstract_parser.parse_delta) + # since parser state cannot be carried across calls. + prompt_len = ( + len(request.prompt_token_ids) if request.prompt_token_ids is not None else 0 + ) + prior_len = ( + len(request.stream_state.output_token_ids) + if request.stream_state is not None + else 0 + ) + delta_len = sum( + len(c.token_ids) for c in request.generate_chunk.choices if c.token_ids + ) + max_model_len = self.model_config.max_model_len + if prompt_len + prior_len + delta_len > max_model_len: + return self.create_error_response( + f"prompt_token_ids length ({prompt_len}) plus " + f"output_token_ids length ({prior_len}) plus delta " + f"({delta_len}) exceeds max_model_len ({max_model_len})." + ) + model_name = request.model or self.models.model_name() try: chunk, updated_state = await self.online_derenderer.derender_chat_stream( @@ -264,15 +324,21 @@ class ServingDerender(BaseServing): state=request.stream_state, chat_request=request.chat_request, prompt_tokens=request.prompt_tokens, + prompt_token_ids=request.prompt_token_ids, ) - except NotImplementedError as exc: - return self.create_error_response(exc) - except ValueError as exc: - return self.create_error_response(str(exc)) - except (KeyError, IndexError) as exc: - return self.create_error_response( - f"invalid stream_state: detokenization failed ({exc!r})" + except Exception as exc: + # The two ValueErrors derender_chat_stream can raise directly + # (missing chat_request, >1 choice per chunk) are already + # pre-checked above, so anything reaching here comes from the + # parser replaying entirely client controlled state. Hermes + # swallows its own streaming exceptions but finalize_generation + # and engine based parsers don't, so an unexpected parser + # failure on malformed (but well typed) state must surface as + # a 400 here rather than an unhandled 500. + logger.debug( + "derender stream replay failed: %s: %s", type(exc).__name__, exc ) + return self.create_error_response("invalid stream_state: derender failed") logger.debug( "derender_chat_stream request_id=%s model=%s delta_tokens=%d", @@ -313,7 +379,7 @@ class ServingDerender(BaseServing): return self.create_error_response(str(exc)) except (KeyError, IndexError) as exc: return self.create_error_response( - f"invalid stream_state: detokenization failed ({exc!r})" + f"invalid stream_state: detokenization failed ({exc})" ) logger.debug( diff --git a/vllm/entrypoints/scale_out/token_in_token_out/protocol.py b/vllm/entrypoints/scale_out/token_in_token_out/protocol.py index be8130e314..2bd47b1059 100644 --- a/vllm/entrypoints/scale_out/token_in_token_out/protocol.py +++ b/vllm/entrypoints/scale_out/token_in_token_out/protocol.py @@ -1,6 +1,6 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project -from typing import Any, Literal, TypeAlias +from typing import Annotated, Any, Literal, TypeAlias from pydantic import ( BaseModel, @@ -483,16 +483,32 @@ class DerenderStreamState(BaseModel): streaming derender endpoint. All fields are plain JSON serializable data. No opaque tokenizer or parser internals are stored here. + Two separate sets of fields support two different streaming modes: + + - For plain streaming (no parser configured including the completions path): + `prev_tokens`, `prefix_offset` and `read_offset` maintain a bounded incremental + decoding window. This requires O(window) transport and O(delta) computation + per chunk. + - For parser enabled chat streaming: `output_token_ids`, `output_chunk_lens`, + `tools_streamed` and `last_tool_call_ids` are used to replay `parse_delta()` + from scratch on every chunk because parser state cannot be serialized. This + incurs O(n) transport per chunk (O(n²) per generation) and O(n²) + `parse_delta()` calls per generation. Since many parsers re-scan the + entire accumulated text on each invocation, `parse_delta()` itself is + O(n) yielding a true worst case compute cost of O(n³) per generation. + No caching is performed. Work is bounded by `max_model_len`. + See `OnlineDerenderer._derender_chat_stream_parsed`. + The detokenization strategy carries the incremental decode offsets directly rather than re-sending the whole token history each chunk. - ``detokenize_incrementally`` only ever reads the trailing token window - ``prev_tokens[prefix_offset:]``, so we carry just that tail plus the two + `detokenize_incrementally` only ever reads the trailing token window + `prev_tokens[prefix_offset:]`, so we carry just that tail plus the two offsets. Each chunk resumes exactly where the last one stopped, including - any partially processed multi-byte character (tracked by ``read_offset``), + any partially processed multi-byte character (tracked by `read_offset`), then trims and rebases the window so it never grows with generation length. Performance: - - Compute per chunk is O(delta). One ``detokenize_incrementally`` call per + - Compute per chunk is O(delta). One `detokenize_incrementally` call per new token, independent of how many tokens preceded it. - Transport per chunk is O(window). The carried tail is bounded by the incremental detokenization offset, so cumulative bytes over the wire are @@ -500,18 +516,18 @@ class DerenderStreamState(BaseModel): """ prev_tokens: list[str] = Field(default_factory=list) - """Trailing decode window. Token strings from ``prefix_offset`` onward. + """Trailing decode window. Token strings from `prefix_offset` onward. Bounded, trimmed and rebased each chunk to the tail - ``detokenize_incrementally`` still reads, so it does not grow with the + `detokenize_incrementally` still reads, so it does not grow with the number of chunks. """ prefix_offset: int = Field(default=0, ge=0) - """Prefix offset into ``prev_tokens`` for incremental detokenization.""" + """Prefix offset into `prev_tokens` for incremental detokenization.""" read_offset: int = Field(default=0, ge=0) - """Read offset into ``prev_tokens`` for incremental detokenization.""" + """Read offset into `prev_tokens` for incremental detokenization.""" @field_validator("prev_tokens") @classmethod @@ -525,23 +541,53 @@ class DerenderStreamState(BaseModel): return v role_sent: bool = False - """True once the initial ``role: "assistant"`` delta has been emitted. + """True once the initial `role: "assistant"` delta has been emitted. Prevents re-emitting the role on subsequent chunks even when the detok window is transiently empty (e.g. usage only final chunk). """ - # TODO: Properties used in follow on PR for tool call parsing - last_content: str | None = None - """Last emitted cumulative assistant content text.""" + output_token_ids: list[int] = Field(default_factory=list) + """All output tokens seen so far. Parser path only. - last_reasoning: str | None = None - """Last emitted cumulative reasoning text.""" + Replay buffer: each chunk rebuilds a fresh parser and replays every + token in here through `parse_delta` (discarding the result) before + processing the current chunk's tokens since parser internal state + cannot be serialized into this stateless model. Unavoidably O(n) + bounded by ``max_model_len`` (enforced server side, not by a field + validator here since the bound is model dependent). + """ + + output_chunk_lens: list[Annotated[int, Field(gt=0)]] = Field(default_factory=list) + """Token count of each chunk in `output_token_ids`. Parser path only. + + Replay uses these to reproduce the original `parse_delta` call + boundaries. Must sum to `len(output_token_ids)`. + """ + + @model_validator(mode="after") + def _validate_output_chunk_lens(self) -> "DerenderStreamState": + total = sum(self.output_chunk_lens) + if total != len(self.output_token_ids): + raise ValueError( + f"output_chunk_lens must sum to len(output_token_ids) " + f"(got sum={total}, len(output_token_ids)=" + f"{len(self.output_token_ids)})" + ) + return self + + tools_streamed: bool = False + """True once a tool call delta has been emitted. Parser path only. + + Drives the `finish_reason` -> `"tool_calls"` rewrite on the final + chunk mirroring the generate streaming path. + """ last_tool_call_ids: list[str] = Field(default_factory=list) - """Stable tool-call IDs, assigned once when each call first appears. + """Stable tool call IDs, assigned once when each call first appears. - Prevents ID regeneration across re-parsing. + Indexed by tool call index. Parser path only. Prevents ID regeneration + when replay reprocesses a tool call that already has a pinned ID. """ @@ -570,6 +616,19 @@ class DerenderChatStreamRequest(BaseModel): prompt_tokens: int | None = None """Prompt token count for usage. Forwarded from the render step.""" + prompt_token_ids: list[int] | None = None + """Prompt token IDs. Required by the parser path's `parse_delta` to + settle its initial reasoning state (e.g. chat templates that pre-open + ````). `prompt_tokens` is a usage count and cannot serve this + purpose. Sourced from `GenerateRequest.token_ids` at the render step. + + Rejected with a 400 (by `ServingDerender`) when a tool or reasoning + parser is configured and this is omitted. Without it, `parse_delta` + cannot tell whether the prompt left reasoning open and would silently + misclassify reasoning content as plain content. Unused on the plain + detokenization path. + """ + chat_request: ChatCompletionRequest | None = None """The original (post adjust_request) ChatCompletionRequest from /render.""" diff --git a/vllm/renderers/online_derenderer.py b/vllm/renderers/online_derenderer.py index 05f2710afb..a89bc843b8 100644 --- a/vllm/renderers/online_derenderer.py +++ b/vllm/renderers/online_derenderer.py @@ -4,7 +4,10 @@ from typing import Any from vllm.config import ModelConfig from vllm.entrypoints.chat_utils import ChatTemplateContentFormatOption -from vllm.entrypoints.generate.base.protocol import DeltaMessage, ToolCall +from vllm.entrypoints.generate.base.protocol import ( + DeltaMessage, + ToolCall, +) from vllm.entrypoints.generate.base.serving import resolve_token_id_placeholder from vllm.entrypoints.openai.chat_completion.protocol import ( ChatCompletionLogProbs, @@ -29,6 +32,9 @@ from vllm.entrypoints.scale_out.token_in_token_out.protocol import ( ) from vllm.entrypoints.serve.engine.protocol import UsageInfo from vllm.entrypoints.serve.utils.request_logger import RequestLogger +from vllm.entrypoints.serve.utils.tool_calls_utils import ( + maybe_filter_parallel_tool_calls, +) from vllm.logger import init_logger from vllm.parser import Parser, ParserManager from vllm.renderers import BaseRenderer @@ -95,6 +101,11 @@ class OnlineDerenderer: self._derender_completion_async = make_async( self._derender_completion, executor=renderer._executor ) + # Replay is O(n) per chunk (unlike the O(delta) detok-only paths), + # so it must not run on the event loop either. + self._derender_chat_stream_parsed_async = make_async( + self._derender_chat_stream_parsed, executor=renderer._executor + ) async def derender_chat( self, @@ -144,6 +155,7 @@ class OnlineDerenderer: tokenizer, chat_request.tools, chat_template_kwargs=chat_template_kwargs, + model_config=self.model_config, ) reasoning, content, tool_calls = parser.parse( decoded_text, @@ -278,23 +290,30 @@ class OnlineDerenderer: state: DerenderStreamState | None = None, chat_request: ChatCompletionRequest | None = None, prompt_tokens: int | None = None, + prompt_token_ids: list[int] | None = None, ) -> tuple[ChatCompletionStreamResponse, DerenderStreamState]: """Process one GenerateStreamResponse chunk for streaming chat derender. - TODO: parse path for reasoning and tool calls is implemented in future PR. - - Unlike OpenAI's API, which always emits ``role: "assistant"`` on the + Unlike OpenAI's API, which always emits `role: "assistant"` on the very first chunk, this emits it on the first chunk with a non empty - ``choices`` list. A leading usage only chunk therefore defers the + `choices` list. A leading usage only chunk therefore defers the role to the following content chunk instead of sending an empty role only delta. Args: model: Model name for the response object. - generate_chunk: One SSE chunk from ``/inference/v1/generate``. - state: Client carried detok state (``None`` for first call). - chat_request: Original ChatCompletionRequest from ``/render``. + generate_chunk: One SSE chunk from `/inference/v1/generate`. + state: Client carried detok state (`None` for first call). + chat_request: Original ChatCompletionRequest from `/render`. + Required when a reasoning or tool parser is configured + (validated by the caller — see `ServingDerender`) because + plain detokenization would leak raw parser markup into `content`. prompt_tokens: Prompt token count for the usage chunk. + prompt_token_ids: Prompt token IDs. Parser path only. Lets + `parse_delta` settle its initial reasoning state. Required + when a reasoning or tool parser is configured (validated by + the caller — see ``ServingDerender``). Without it reasoning + left open by the prompt would be misclassified as content. Returns: (chunk, updated_state) — the derendered SSE chunk and the state @@ -304,28 +323,36 @@ class OnlineDerenderer: if state is None: state = DerenderStreamState() - if self.parser is not None: - # TODO: Follow on PR will implement the parse path. Check on the - # parser alone (fail closed). A parser configured model must never - # fall through to plain detok on the streaming path, even when - # ``chat_request`` is omitted or reasoning/tool markup would leak - # into ``delta.content``. - raise NotImplementedError( - "Streaming chat derender is not yet supported for models with " - "a reasoning or tool parser configured. Use the non-streaming " - "derender endpoint (stream=false) for parsed output." - ) - # A single DerenderStreamState is threaded through every choice in # this chunk. Correct only when there is at most one choice per SSE # event (n=1, one call per index), as the streaming derender # protocol assumes. Multiple choices sharing one chunk would corrupt - # each other's detok window. + # each other's detok/parser state. if len(generate_chunk.choices) > 1: raise ValueError( "derender_chat_stream expects at most one choice per chunk" ) + parser_cls = self.parser + if parser_cls is not None: + # Fail (mirrors ServingDerender's pre-check) because a parser + # configured model must never fall through to plain detok or + # reasoning/tool markup would leak into `delta.content`. + if chat_request is None: + raise ValueError( + "chat_request is required for streaming chat derender " + "when a tool or reasoning parser is configured" + ) + return await self._derender_chat_stream_parsed_async( + parser_cls, + model, + generate_chunk, + state, + chat_request, + prompt_tokens, + prompt_token_ids, + ) + tokenizer = self.renderer.get_tokenizer() skip_special = ( chat_request.skip_special_tokens if chat_request is not None else True @@ -374,6 +401,203 @@ class OnlineDerenderer: ) return chunk, updated_state + def _derender_chat_stream_parsed( + self, + parser_cls: type[Parser], + model: str, + generate_chunk: GenerateStreamResponse, + state: DerenderStreamState, + chat_request: ChatCompletionRequest, + prompt_tokens: int | None, + prompt_token_ids: list[int] | None, + ) -> tuple[ChatCompletionStreamResponse, DerenderStreamState]: + """Parser path for streaming chat derender: replay + `parse_delta`. + + Parser internal state (buffered markup, reasoning/tool phase, etc.) + cannot be serialized into `DerenderStreamState`, so each call + builds a fresh parser and replays every prior output token through + `parse_delta` (discarding the result) before processing this + chunk's tokens for real. + + Every chunk, replayed or live, goes through one `parse_delta` call + with the tokens it arrived with (more than one under e.g. + speculative decoding). Replay recovers those boundaries from + `state.output_chunk_lens`, so the rebuilt parser state matches + standard serving, which calls `parse_delta` once per engine step. + Text comes from a fresh incremental detokenizer with special tokens + preserved (``skip_special_tokens=False``). + """ + tokenizer = self.renderer.get_tokenizer() + + chat_template_kwargs: dict[str, Any] = {} + if not self.use_harmony: + chat_template_kwargs = ( + chat_request.build_chat_params( + self.chat_template, + self.chat_template_content_format, + ) + .with_defaults(self.default_chat_template_kwargs) + .chat_template_kwargs + ) + + parser = parser_cls( + tokenizer, + chat_request.tools, + chat_template_kwargs=chat_template_kwargs, + model_config=self.model_config, + ) + + # Ephemeral incremental detok window, local to this call. Threaded + # across both the replay and current chunk phases (via `nonlocal`) + # so multi-byte characters split across that boundary still decode + # correctly. Discarded once the call returns. + detok_state = DerenderStreamState() + + def _replay(token_ids: list[int], chunk_lens: list[int]) -> None: + """Replay prior chunks through `parse_delta` to rebuild parser + state, discarding the result. Never `finished` since that only + applies to the current chunk. + """ + nonlocal detok_state + start = 0 + for chunk_len in chunk_lens: + chunk = token_ids[start : start + chunk_len] + start += chunk_len + text, detok_state = self._detokenize_delta( + tokenizer, chunk, detok_state, skip_special_tokens=False + ) + parser.parse_delta( + text, + chunk, + chat_request, + prompt_token_ids=prompt_token_ids, + finished=False, + ) + + # Replay history to reconstruct parser state. The result is thrown + # away and only the current chunk's emission goes to the client. + _replay(state.output_token_ids, state.output_chunk_lens) + + stream_choices: list[ChatCompletionResponseStreamChoice] = [] + output_token_ids = list(state.output_token_ids) + output_chunk_lens = list(state.output_chunk_lens) + role_sent = state.role_sent + tools_streamed = state.tools_streamed + last_tool_call_ids = list(state.last_tool_call_ids) + + # At most one choice: the caller (derender_chat_stream) already + # rejects >1 before dispatching here. role_sent/tools_streamed/ + # output_token_ids below are updated for that single choice, not + # accumulated across choices. Looping over generate_chunk.choices + # could silently corrupt output if chunks were ever allowed to + # contain multiple choices. + if generate_chunk.choices: + choice = generate_chunk.choices[0] + delta_tids = choice.token_ids or [] + is_finished = choice.finish_reason is not None + + if delta_tids: + # One parse_delta call for the whole chunk (producer + # granularity), not one per token. See the granularity note + # in this method's docstring. + text, detok_state = self._detokenize_delta( + tokenizer, delta_tids, detok_state, skip_special_tokens=False + ) + delta_message = parser.parse_delta( + text, + delta_tids, + chat_request, + prompt_token_ids=prompt_token_ids, + finished=is_finished, + ) + elif is_finished: + # Finish only chunk (no new tokens). Still flush any + # buffered tool call arguments. + delta_message = parser.parse_delta( + "", + [], + chat_request, + prompt_token_ids=prompt_token_ids, + finished=True, + ) + else: + delta_message = None + + if delta_tids: + output_token_ids.extend(delta_tids) + output_chunk_lens.append(len(delta_tids)) + + if delta_message is None: + delta_message = DeltaMessage() + + if delta_message.tool_calls: + tools_streamed = True + for tc in delta_message.tool_calls: + if tc.id is None: + continue + if tc.index < len(last_tool_call_ids): + # Pin: reuse the ID already recorded for this index + # rather than one a from scratch replay regenerated. + # Real trigger not just defensive with + # tool_choice="required", + # extract_required_tool_call_streaming resets + # function_name_returned to False whenever the + # partial JSON transiently fails to parse which + # re-emits id+name for the same index on replay. + tc.id = last_tool_call_ids[tc.index] + else: + last_tool_call_ids.append(tc.id) + + if not role_sent: + delta_message.role = "assistant" + role_sent = True + + finish_reason = choice.finish_reason + if finish_reason is not None: + is_named_tool_choice = ( + type(chat_request.tool_choice) is ChatCompletionNamedToolChoiceParam + ) + if tools_streamed and not is_named_tool_choice: + finish_reason = "tool_calls" + + stream_choice = ChatCompletionResponseStreamChoice( + index=choice.index, + delta=delta_message, + finish_reason=finish_reason, + ) + stream_choices.append( + maybe_filter_parallel_tool_calls(stream_choice, chat_request) + ) + + updated_state = state.model_copy( + update={ + "output_token_ids": output_token_ids, + "output_chunk_lens": output_chunk_lens, + "role_sent": role_sent, + "tools_streamed": tools_streamed, + "last_tool_call_ids": last_tool_call_ids, + } + ) + + usage: UsageInfo | None = None + if generate_chunk.usage is not None: + u = generate_chunk.usage + pt = prompt_tokens if prompt_tokens is not None else (u.prompt_tokens or 0) + ct = u.completion_tokens or 0 + usage = UsageInfo( + prompt_tokens=pt, + completion_tokens=ct, + total_tokens=pt + ct, + ) + + chunk = ChatCompletionStreamResponse( + id=generate_chunk.request_id, + model=model, + choices=stream_choices, + usage=usage, + ) + return chunk, updated_state + async def derender_completion( self, generate_responses: list[GenerateResponse],