diff --git a/docs/serving/online_serving/derenderer.md b/docs/serving/online_serving/derenderer.md
index 298510062a..773131d033 100644
--- a/docs/serving/online_serving/derenderer.md
+++ b/docs/serving/online_serving/derenderer.md
@@ -6,7 +6,7 @@ This closes the loop for a token-in / token-out engine in disaggregated serving:
- **GPU less post processing**: Detokenization, reasoning parsing, and tool call parsing run on the same GPU less frontend that hosts `/render`
- **Parser parity**: The derenderer reuses vLLM's tool and reasoning parsers, so a disaggregated deployment produces the same `content`/`reasoning`/ `tool_calls` split as a standard `vllm serve` server
-- **Non-streaming**: The endpoints expect a complete `GenerateResponse` with all token IDs present and perform one-shot parsing. Streaming derender would require a separate endpoint design and is not currently supported but is in the pipeline
+- **Streaming and one-shot parsing**: Non-streaming calls send a complete `GenerateResponse` with all token IDs present and get one-shot parsing. Both endpoints also accept `stream: true`, taking one `GenerateStreamResponse` delta plus a client carried `stream_state` and returning `{chunk, stream_state}`. The chat endpoint's streaming path supports reasoning and tool call parsing while emitting the same `reasoning`/`content`/`tool_calls` deltas the generate streaming path would for the same token IDs
Both endpoints are hosted by the GPU less rendering server started with [`vllm launch render`](../../cli/launch/render.md), alongside the `/render`
endpoints.
@@ -21,7 +21,7 @@ endpoints.
└─────────────── request + prompt_tokens ──┘
```
-The derender step needs more than the engine's `token_ids`. It also consumes the original `chat_request`/`completion_request` and `prompt_tokens` carried over from the render step (see [Request format](#request-format)) so the tool and reasoning parsers have the context they need.
+The derender step needs more than the engine's `token_ids`. It also consumes the original `chat_request`/`completion_request` and `prompt_tokens` carried over from the render step (see [Request format](#request-format)) so the tool and reasoning parsers have the context they need. The chat streaming path additionally needs `prompt_token_ids` (see [Streaming cost](#streaming-cost)) when a parser is configured.
## API Reference
@@ -52,6 +52,21 @@ Each request wraps the engine's `GenerateResponse`(s) together with the caller m
Oversized payloads are rejected with a `400` before any `tokenizer.decode()` or parser runs.
+## Streaming cost
+
+Streaming derender threads a client carried `stream_state` across per-chunk calls instead of keeping session state on the server.
+
+Plain detokenization (no parser configured) carries only a small, bounded incremental decode window in `stream_state` independent of generation length.
+
+When a tool or reasoning parser is configured, parser internal state (buffered markup, reasoning/tool phase) can't be serialized. `stream_state` therefore instead carries the full `output_token_ids` seen so far plus `output_chunk_lens`, the token count of each chunk they arrived in. Each chunk rebuilds a fresh parser and replays that history through `parse_delta`, one call per original chunk, before processing the new tokens for real. For the parser path only, this means:
+
+- **Transport**: `output_token_ids` and `output_chunk_lens` round-trip in full in both directions on every call. `output_chunk_lens` has one entry per chunk, which is one per token without speculative decoding. This means O(n) bytes per chunk, O(n²) bytes over a full generation. Bounded by `max_model_len`. `prompt_token_ids` is sent in full on every call too and it isn't trimmed as `output_token_ids` grows. This means that for most of a stream it dominates the per chunk payload. A 100k token prompt with 1k tokens of output means `prompt_token_ids` is ~99% of the request body on every chunk.
+- **Compute**: replay is O(n) `parse_delta` calls per chunk (O(n²) per generation). `parse_delta` itself is O(n) for parsers that re-scan accumulated text (e.g. Hermes tool-call JSON, DeepSeek-R1 reasoning). The per-generation cost is O(n³) character work, not O(n²). This is a deliberately minimal first implementation with no caching layer.
+- The parser path also requires `prompt_token_ids` so `parse_delta` can settle whether the prompt left reasoning open or not. Since parser state can't be carried across calls, it re-scans the full prompt once per chunk.
+- Replay runs off the event loop on the renderer's executor (`renderer_num_workers`, default `1`). Size it for the expected number of concurrent parser configured streams.
+
+`output_token_ids` and `prompt_token_ids` are both bounded by `max_model_len` but callers streaming long reasoning traces through a parser configured model should expect materially more state transport and CPU cost than the plain detokenization path.
+
## Example
The example below drives the full `render → generate → derender` round trip for a chat request against a GPU less render server (`/render`, `/derender`) and a token-in / token-out engine (`/inference/v1/generate`).
@@ -102,4 +117,4 @@ with httpx.Client(timeout=60.0) as client:
print(response["choices"][0]["message"]["content"])
```
-Passing `chat_request` lets the derenderer run the configured tool and reasoning parsers. This means `response["choices"][0]["message"]` carries the same `content` / `reasoning` / `tool_calls` split a `vllm serve` server would produce. Omit `chat_request` for plain detokenization only.
+Passing `chat_request` lets the derenderer run the configured tool and reasoning parsers. This means `response["choices"][0]["message"]` carries the same `content` / `reasoning` / `tool_calls` split a `vllm serve` server would produce. `chat_request` can only be omitted for a model with no tool or reasoning parser configured. A parser configured model rejects a missing `chat_request` with a 400 rather than silently falling back to plain detokenization.
diff --git a/tests/entrypoints/scale_out/derender/test_derender.py b/tests/entrypoints/scale_out/derender/test_derender.py
index d7e0298ac3..9cc6124393 100644
--- a/tests/entrypoints/scale_out/derender/test_derender.py
+++ b/tests/entrypoints/scale_out/derender/test_derender.py
@@ -800,7 +800,12 @@ def _e2e_generate_response(
@pytest.mark.asyncio
async def test_e2e_plain_roundtrip(parser_client, parser_tokenizer):
- """Plain text without reasoning markers roundtrips correctly."""
+ """Plain text without reasoning markers roundtrips correctly.
+
+ Markerless output has no ````, which deepseek_r1 classifies
+ wholly as reasoning, so the text lands there rather than in content.
+ What this pins is detokenization fidelity through the parser path.
+ """
messages = [{"role": "user", "content": "What is 2+2?"}]
gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages)
@@ -814,16 +819,22 @@ async def test_e2e_plain_roundtrip(parser_client, parser_tokenizer):
"model": PARSER_MODEL,
"generate_response": _e2e_generate_response(output_ids),
"prompt_tokens": len(gen_req["token_ids"]),
+ "chat_request": {"model": PARSER_MODEL, "messages": messages},
},
)
assert resp.status_code == 200, resp.text
- content = resp.json()["choices"][0]["message"]["content"]
- assert content == expected
+ message = resp.json()["choices"][0]["message"]
+ assert message["reasoning"] == expected
+ assert message["content"] is None
@pytest.mark.asyncio
async def test_e2e_token_identity(parser_client, parser_tokenizer):
- """encode(derender(token_ids)) == token_ids (RL invariant)."""
+ """encode(derender(token_ids)) == token_ids (RL invariant).
+
+ Markerless output comes back as reasoning (see
+ ``test_e2e_plain_roundtrip``), so re-encode that.
+ """
messages = [{"role": "user", "content": "Hi"}]
gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages)
@@ -836,17 +847,22 @@ async def test_e2e_token_identity(parser_client, parser_tokenizer):
"model": PARSER_MODEL,
"generate_response": _e2e_generate_response(output_ids),
"prompt_tokens": len(gen_req["token_ids"]),
+ "chat_request": {"model": PARSER_MODEL, "messages": messages},
},
)
assert resp.status_code == 200
- content = resp.json()["choices"][0]["message"]["content"]
- re_encoded = _encode(parser_tokenizer, content)
+ reasoning = resp.json()["choices"][0]["message"]["reasoning"]
+ re_encoded = _encode(parser_tokenizer, reasoning)
assert output_ids == re_encoded
@pytest.mark.asyncio
async def test_e2e_non_ascii_roundtrip(parser_client, parser_tokenizer):
- """CJK + emoji roundtrip without U+FFFD."""
+ """CJK + emoji roundtrip without U+FFFD.
+
+ Markerless output comes back as reasoning (see
+ ``test_e2e_plain_roundtrip``).
+ """
messages = [{"role": "user", "content": "Reply in Chinese"}]
gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages)
@@ -859,11 +875,12 @@ async def test_e2e_non_ascii_roundtrip(parser_client, parser_tokenizer):
"model": PARSER_MODEL,
"generate_response": _e2e_generate_response(output_ids),
"prompt_tokens": len(gen_req["token_ids"]),
+ "chat_request": {"model": PARSER_MODEL, "messages": messages},
},
)
assert resp.status_code == 200
- content = resp.json()["choices"][0]["message"]["content"]
- assert "�" not in content
+ reasoning = resp.json()["choices"][0]["message"]["reasoning"]
+ assert "�" not in reasoning
@pytest.mark.asyncio
@@ -976,8 +993,10 @@ async def test_e2e_parsed_reasoning_and_tool_call(parser_client, parser_tokenize
@pytest.mark.asyncio
-async def test_e2e_no_chat_request_fallback(parser_client, parser_tokenizer):
- """Without chat_request, derender falls back to plain detokenization."""
+async def test_e2e_no_chat_request_rejected(parser_client, parser_tokenizer):
+ """Without chat_request a parser configured model rejects with 400
+ rather than silently falling back to plain detokenization. This is to
+ prevent the leak of raw reasoning/tool markup into content."""
messages = [{"role": "user", "content": "Hello"}]
gen_req = await _e2e_render_chat(parser_client, PARSER_MODEL, messages)
@@ -992,9 +1011,8 @@ async def test_e2e_no_chat_request_fallback(parser_client, parser_tokenizer):
"prompt_tokens": len(gen_req["token_ids"]),
},
)
- assert resp.status_code == 200
- content = resp.json()["choices"][0]["message"]["content"]
- assert "Hi" in content
+ assert resp.status_code == 400
+ assert "chat_request" in resp.json()["error"]["message"]
# ---------------------------------------------------------------------------
diff --git a/tests/entrypoints/scale_out/derender/test_derender_stream.py b/tests/entrypoints/scale_out/derender/test_derender_stream.py
index df10b31d28..4032d9cbf3 100644
--- a/tests/entrypoints/scale_out/derender/test_derender_stream.py
+++ b/tests/entrypoints/scale_out/derender/test_derender_stream.py
@@ -8,6 +8,8 @@ Tests are split into two layers:
1. Unit tests (no server): covers ``_detokenize_delta`` correctness
(chunked == one-shot) and ``derender_completion_stream`` /
``derender_chat_stream`` logic via a real tokenizer on a tiny model.
+ The parser path is covered both with a deterministic stub parser and
+ with the real ``HarmonyParser`` (skipped without ``openai_harmony``).
2. Integration tests (require a running render server): covers the full
HTTP round-trip through the streaming endpoint. Marked with
@@ -15,19 +17,142 @@ Tests are split into two layers:
fixtures from the sibling ``test_derender.py``.
"""
+import json
+from collections.abc import Callable
+
import pytest
import pytest_asyncio
+from vllm.entrypoints.generate.base.protocol import (
+ DeltaFunctionCall,
+ DeltaMessage,
+ DeltaToolCall,
+)
from vllm.entrypoints.scale_out.token_in_token_out.protocol import (
DerenderStreamState,
+ GenerateResponse,
+ GenerateResponseChoice,
GenerateResponseStreamChoice,
GenerateStreamResponse,
)
from vllm.entrypoints.serve.engine.protocol import ErrorResponse, UsageInfo
+from vllm.parser import Parser
+from vllm.utils import random_uuid
MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM"
+def _merge_fake_deltas(
+ accumulated: DeltaMessage | None, new: DeltaMessage | None
+) -> DeltaMessage | None:
+ """Fold one token's worth of `_FakeParser` output into a per call
+ accumulator, the same way a real parser merges multiple tokens'
+ effects into the single `DeltaMessage` it returns per `parse_delta`
+ call.
+ """
+ if new is None:
+ return accumulated
+ if accumulated is None:
+ accumulated = DeltaMessage()
+
+ if new.content:
+ accumulated.content = (accumulated.content or "") + new.content
+ if new.reasoning:
+ accumulated.reasoning = (accumulated.reasoning or "") + new.reasoning
+
+ for tc in new.tool_calls:
+ starts_new_call = tc.id is not None or (
+ tc.function is not None and tc.function.name is not None
+ )
+ existing = (
+ None
+ if starts_new_call
+ else next((t for t in accumulated.tool_calls if t.index == tc.index), None)
+ )
+ if existing is None:
+ accumulated.tool_calls.append(tc)
+ continue
+ if existing.function is None:
+ existing.function = DeltaFunctionCall()
+ if tc.function is not None and tc.function.arguments:
+ existing.function.arguments = (
+ existing.function.arguments or ""
+ ) + tc.function.arguments
+
+ return accumulated
+
+
+class _FakeParser(Parser):
+ """Deterministic `Parser` stub for unit testing the replay/merge/pin
+ mechanics of `OnlineDerenderer._derender_chat_stream_parsed` in
+ isolation from any real reasoning/tool parser's markup grammar.
+
+ Processes every id in `delta_token_ids` in order and folds their
+ effects into one `DeltaMessage`, like a real parser handling a
+ multi-token delta in a single call. Ignores `delta_text`:
+
+ - `TOOL_START` opens a new tool call at index 0 (id + name).
+ - `TOOL_ARG` appends one `"a"` to that tool call's arguments.
+ - `REASON` emits one `"r"` of reasoning (suppressed when the request
+ has `include_reasoning=False`, mirroring the real parsers).
+ - `CONTENT` emits one `"c"` of content.
+ - An empty `delta_token_ids` with `finished=True` (the finish only
+ flush call) emits a `"FLUSH"` content sentinel so tests can confirm
+ it happened.
+ - Anything else emits nothing.
+ """
+
+ TOOL_START = 9
+ TOOL_ARG = 10
+ REASON = 11
+ CONTENT = 12
+
+ def parse_delta(
+ self,
+ delta_text,
+ delta_token_ids,
+ request,
+ prompt_token_ids=None,
+ *,
+ finished,
+ ):
+ if not delta_token_ids:
+ return DeltaMessage(content="FLUSH") if finished else None
+
+ accumulated: DeltaMessage | None = None
+ for tok in delta_token_ids:
+ accumulated = _merge_fake_deltas(
+ accumulated, self._parse_one_token(tok, request)
+ )
+ return accumulated
+
+ def _parse_one_token(self, tok: int, request) -> DeltaMessage | None:
+ if tok == self.TOOL_START:
+ return DeltaMessage(
+ tool_calls=[
+ DeltaToolCall(
+ id=f"call-{random_uuid()}",
+ type="function",
+ function=DeltaFunctionCall(name="get_weather", arguments=""),
+ index=0,
+ )
+ ]
+ )
+ if tok == self.TOOL_ARG:
+ return DeltaMessage(
+ tool_calls=[
+ DeltaToolCall(index=0, function=DeltaFunctionCall(arguments="a"))
+ ]
+ )
+ if tok == self.REASON:
+ if not request.include_reasoning:
+ return None
+ return DeltaMessage(reasoning="r")
+ if tok == self.CONTENT:
+ return DeltaMessage(content="c")
+ return None
+
+
# ---------------------------------------------------------------------------
# Helpers shared across tests
# ---------------------------------------------------------------------------
@@ -111,6 +236,40 @@ def derenderer(tokenizer):
)
+@pytest.fixture(scope="module")
+def parsed_derenderer(tokenizer):
+ """OnlineDerenderer with `_FakeParser` wired in as `self.parser`.
+
+ The parser configured path runs `_derender_chat_stream_parsed` via
+ `make_async(..., executor=renderer._executor)`, so unlike
+ `derenderer` above (whose streaming paths never touch the executor),
+ `renderer._executor` must be a real `ThreadPoolExecutor` here.
+ `loop.run_in_executor` cannot submit work to a `MagicMock`.
+ """
+ from concurrent.futures import ThreadPoolExecutor
+ from unittest.mock import MagicMock
+
+ from vllm.renderers.online_derenderer import OnlineDerenderer
+
+ renderer = MagicMock()
+ renderer.get_tokenizer.return_value = tokenizer
+ renderer._executor = ThreadPoolExecutor(max_workers=2)
+
+ model_config = MagicMock()
+ model_config.hf_config.model_type = "llama"
+ model_config.model = MODEL_NAME
+
+ dr = OnlineDerenderer(
+ model_config=model_config,
+ renderer=renderer,
+ request_logger=None,
+ chat_template=None,
+ chat_template_content_format="string",
+ )
+ dr.parser = _FakeParser
+ return dr
+
+
class TestDetokenizeDelta:
"""_detokenize_delta: chunked decode must equal one shot decode."""
@@ -378,49 +537,456 @@ class TestDerenderChatStream:
assert streamed == one_shot
@pytest.mark.asyncio
- @pytest.mark.parametrize("with_chat_request", [True, False])
- async def test_parser_raises_not_implemented(self, tokenizer, with_chat_request):
- """Stream chat with a parser active must fail closed (NotImplementedError).
-
- Checks that a parser configured model must 501 even when ``chat_request`` is
- omitted, otherwise reasoning/tool markup would leak into ``delta.content`
- via the plain detok fallback.
- """
- from unittest.mock import MagicMock
-
- from vllm.renderers.online_derenderer import OnlineDerenderer
-
- renderer = MagicMock()
- renderer.get_tokenizer.return_value = tokenizer
-
- model_config = MagicMock()
- model_config.hf_config.model_type = "llama"
- model_config.model = MODEL_NAME
-
- # Construct a derenderer WITH a tool/reasoning parser active.
- # ParserManager.get_parser returns None when no parser is named, so
- # we inject a mock parser class directly.
- dr = OnlineDerenderer(
- model_config=model_config,
- renderer=renderer,
- request_logger=None,
- chat_template=None,
- chat_template_content_format="string",
- )
- dr.parser = MagicMock() # simulate a parser being active
-
- chat_request = MagicMock() if with_chat_request else None
- token_ids = tokenizer.encode("hello")[:3]
-
- with pytest.raises(NotImplementedError):
- await dr.derender_chat_stream(
+ async def test_parser_configured_missing_chat_request_raises(
+ self, parsed_derenderer
+ ):
+ """A parser configured model must never fall through to plain detok
+ even when `chat_request` is omitted. If allowed this would leak raw
+ reasoning/tool markup into `delta.content`. `ServingDerender`
+ pre-checks this too (400 before touching the tokenizer). This pins
+ the `OnlineDerenderer` level backstop."""
+ with pytest.raises(ValueError, match="chat_request"):
+ await parsed_derenderer.derender_chat_stream(
model=MODEL_NAME,
- generate_chunk=_make_stream_chunk(token_ids),
+ generate_chunk=_make_stream_chunk([_FakeParser.CONTENT]),
state=None,
- chat_request=chat_request,
+ chat_request=None,
)
+def _chat_request(**kwargs):
+ from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest
+
+ kwargs.setdefault("messages", [{"role": "user", "content": "hi"}])
+ kwargs.setdefault("model", MODEL_NAME)
+ return ChatCompletionRequest(**kwargs)
+
+
+class TestDerenderChatStreamParsed:
+ """derender_chat_stream: parser branch (replay + parse_delta) exercised
+ against the deterministic `_FakeParser` so these pin OnlineDerenderer's
+ own replay/merge/pin/finish_reason logic independent of any real
+ parser's markup grammar (covered separately by the parser_server backed
+ integration tests below)."""
+
+ @pytest.mark.asyncio
+ async def test_dispatches_and_emits_content(self, parsed_derenderer):
+ chunk, state = await parsed_derenderer.derender_chat_stream(
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk(
+ [_FakeParser.CONTENT], finish_reason="stop"
+ ),
+ chat_request=_chat_request(),
+ )
+ assert chunk.choices[0].delta.content == "c"
+ assert chunk.choices[0].delta.role == "assistant"
+ assert chunk.choices[0].finish_reason == "stop"
+ assert state.output_token_ids == [_FakeParser.CONTENT]
+
+ @pytest.mark.asyncio
+ async def test_role_sent_once(self, parsed_derenderer):
+ chat_request = _chat_request()
+ chunk1, state = await parsed_derenderer.derender_chat_stream(
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk([_FakeParser.CONTENT]),
+ chat_request=chat_request,
+ )
+ chunk2, _ = await parsed_derenderer.derender_chat_stream(
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk(
+ [_FakeParser.CONTENT], finish_reason="stop"
+ ),
+ state=state,
+ chat_request=chat_request,
+ )
+ assert chunk1.choices[0].delta.role == "assistant"
+ assert chunk2.choices[0].delta.role is None
+
+ @pytest.mark.asyncio
+ async def test_finish_only_chunk_flushes(self, parsed_derenderer):
+ """A finish only chunk (no new tokens) still calls `parse_delta`
+ once with `finished=True` to flush buffered state."""
+ chat_request = _chat_request()
+ _, state = await parsed_derenderer.derender_chat_stream(
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk([_FakeParser.CONTENT]),
+ chat_request=chat_request,
+ )
+ chunk2, _ = await parsed_derenderer.derender_chat_stream(
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk([], finish_reason="stop"),
+ state=state,
+ chat_request=chat_request,
+ )
+ assert chunk2.choices[0].delta.content == "FLUSH"
+ assert chunk2.choices[0].finish_reason == "stop"
+
+ @pytest.mark.asyncio
+ async def test_chunking_invariance(self, parsed_derenderer):
+ """1 token per chunk vs a single whole chunk assemble identically.
+
+ Every chunk (live or replayed) is fed to `parse_delta` at its own
+ producer granularity (see `_derender_chat_stream_parsed`'s
+ docstring). This proves that choice doesn't change the assembled
+ output for a well behaved incremental parser. Each call's effects
+ fold together the same way regardless of how many tokens land in
+ one call vs. many."""
+ token_ids = [
+ _FakeParser.REASON,
+ _FakeParser.REASON,
+ _FakeParser.TOOL_START,
+ _FakeParser.TOOL_ARG,
+ _FakeParser.TOOL_ARG,
+ _FakeParser.CONTENT,
+ ]
+ chat_request = _chat_request(
+ tools=[{"type": "function", "function": {"name": "get_weather"}}],
+ tool_choice="auto",
+ include_reasoning=True,
+ )
+
+ async def _assemble(chunks: list[list[int]]) -> dict:
+ state = None
+ content = ""
+ reasoning = ""
+ tool_args = ""
+ for i, tids in enumerate(chunks):
+ finish = "stop" if i == len(chunks) - 1 else None
+ chunk, state = await parsed_derenderer.derender_chat_stream(
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk(tids, finish_reason=finish),
+ state=state,
+ chat_request=chat_request,
+ )
+ delta = chunk.choices[0].delta
+ content += delta.content or ""
+ reasoning += delta.reasoning or ""
+ for tc in delta.tool_calls:
+ if tc.function and tc.function.arguments:
+ tool_args += tc.function.arguments
+ return {"content": content, "reasoning": reasoning, "tool_args": tool_args}
+
+ whole = await _assemble([token_ids])
+ one_at_a_time = await _assemble([[t] for t in token_ids])
+
+ assert whole == one_at_a_time
+ assert whole == {"content": "c", "reasoning": "rr", "tool_args": "aa"}
+
+ @pytest.mark.asyncio
+ async def test_replay_preserves_chunk_boundaries(
+ self, parsed_derenderer, monkeypatch
+ ):
+ """A multi-token chunk (e.g. a speculative decoding step) must be
+ replayed as one `parse_delta` call, not one per token. Parsers
+ with per-call state such as `history_tool_call_cnt` would
+ otherwise diverge from standard serving."""
+
+ class _CallCountingParser(_FakeParser):
+ """Emits how many `parse_delta` calls it has seen."""
+
+ def __init__(self, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self.num_calls = 0
+
+ def parse_delta(
+ self,
+ delta_text,
+ delta_token_ids,
+ request,
+ prompt_token_ids=None,
+ *,
+ finished,
+ ):
+ self.num_calls += 1
+ return DeltaMessage(content=str(self.num_calls))
+
+ monkeypatch.setattr(parsed_derenderer, "parser", _CallCountingParser)
+ chat_request = _chat_request()
+
+ _, state = await parsed_derenderer.derender_chat_stream(
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk([_FakeParser.CONTENT] * 3),
+ chat_request=chat_request,
+ )
+ assert state.output_chunk_lens == [3]
+
+ chunk, _ = await parsed_derenderer.derender_chat_stream(
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk(
+ [_FakeParser.CONTENT], finish_reason="stop"
+ ),
+ state=state,
+ chat_request=chat_request,
+ )
+ assert chunk.choices[0].delta.content == "2"
+
+ @pytest.mark.asyncio
+ async def test_tool_call_id_pinned_across_chunks(self, parsed_derenderer):
+ """Once an index's ID is recorded in `last_tool_call_ids`, a later
+ id bearing delta for that same index is pinned to the recorded
+ value rather than a freshly (re-)generated one."""
+ chat_request = _chat_request(
+ tools=[{"type": "function", "function": {"name": "get_weather"}}],
+ tool_choice="auto",
+ )
+ chunk1, state = await parsed_derenderer.derender_chat_stream(
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk([_FakeParser.TOOL_START]),
+ chat_request=chat_request,
+ )
+ first_id = chunk1.choices[0].delta.tool_calls[0].id
+ assert first_id is not None
+ assert state.last_tool_call_ids == [first_id]
+
+ chunk2, state = await parsed_derenderer.derender_chat_stream(
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk(
+ [_FakeParser.TOOL_START], finish_reason="stop"
+ ),
+ state=state,
+ chat_request=chat_request,
+ )
+ assert chunk2.choices[0].delta.tool_calls[0].id == first_id
+ assert state.last_tool_call_ids == [first_id]
+
+ @pytest.mark.asyncio
+ async def test_finish_reason_rewritten_to_tool_calls(self, parsed_derenderer):
+ chat_request = _chat_request(
+ tools=[{"type": "function", "function": {"name": "get_weather"}}],
+ tool_choice="auto",
+ )
+ chunk, _ = await parsed_derenderer.derender_chat_stream(
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk(
+ [_FakeParser.TOOL_START, _FakeParser.TOOL_ARG], finish_reason="stop"
+ ),
+ chat_request=chat_request,
+ )
+ assert chunk.choices[0].finish_reason == "tool_calls"
+
+ @pytest.mark.asyncio
+ async def test_finish_reason_stop_for_named_tool_choice(self, parsed_derenderer):
+ # ChatCompletionRequest's `check_tool_usage` is a mode="before"
+ # validator, so it sees the raw value and only accepts "auto",
+ # "required" or a dict, never an already built
+ # ChatCompletionNamedToolChoiceParam.
+ chat_request = _chat_request(
+ tools=[{"type": "function", "function": {"name": "get_weather"}}],
+ tool_choice={"type": "function", "function": {"name": "get_weather"}},
+ )
+ chunk, _ = await parsed_derenderer.derender_chat_stream(
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk(
+ [_FakeParser.TOOL_START, _FakeParser.TOOL_ARG], finish_reason="stop"
+ ),
+ chat_request=chat_request,
+ )
+ assert chunk.choices[0].finish_reason == "stop"
+
+ @pytest.mark.asyncio
+ async def test_include_reasoning_false_suppresses_reasoning(
+ self, parsed_derenderer
+ ):
+ chat_request = _chat_request(include_reasoning=False)
+ chunk, _ = await parsed_derenderer.derender_chat_stream(
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk(
+ [_FakeParser.REASON, _FakeParser.CONTENT], finish_reason="stop"
+ ),
+ chat_request=chat_request,
+ )
+ delta = chunk.choices[0].delta
+ assert delta.reasoning is None
+ assert delta.content == "c"
+
+
+# ---------------------------------------------------------------------------
+# Harmony / GPT-OSS replay — unit, no server
+# ---------------------------------------------------------------------------
+
+HARMONY_MODEL = "openai/gpt-oss-20b"
+HARMONY_REASONING = "The user wants 2 plus 3."
+HARMONY_ANSWER = "The answer is 5."
+HARMONY_PROMPT = "<|start|>user<|message|>Add 2 and 3.<|end|><|start|>assistant"
+HARMONY_OUTPUT = (
+ f"<|channel|>analysis<|message|>{HARMONY_REASONING}<|end|>"
+ f"<|start|>assistant<|channel|>final<|message|>{HARMONY_ANSWER}<|return|>"
+)
+
+
+@pytest.fixture(scope="module")
+def harmony_encode():
+ """Encoder for canned GPT-OSS harmony token sequences."""
+ pytest.importorskip("openai_harmony")
+
+ # Pre-caches the o200k_base BPE file that openai-harmony's Rust backend
+ # downloads on first use, same as the sibling suite's E2E harmony tests.
+ from tests.entrypoints.scale_out.derender.test_derender import (
+ _ensure_harmony_vocab,
+ )
+ from vllm.entrypoints.openai.parser.harmony_utils import get_encoding
+
+ _ensure_harmony_vocab()
+ encoding = get_encoding()
+
+ def _encode(harmony_str: str) -> list[int]:
+ return encoding.encode(harmony_str, allowed_special="all")
+
+ return _encode
+
+
+@pytest.fixture(scope="module")
+def harmony_tokenizer():
+ pytest.importorskip("openai_harmony")
+ from vllm.tokenizers import get_tokenizer
+
+ return get_tokenizer(HARMONY_MODEL, trust_remote_code=True)
+
+
+@pytest.fixture(scope="module")
+def harmony_derenderer(harmony_tokenizer):
+ """OnlineDerenderer whose parser resolves to the real `HarmonyParser`.
+
+ Same shape as `parsed_derenderer` (mocked renderer, real executor) but
+ with a real tokenizer and parser, so the replay path is exercised
+ against Harmony's actual channel grammar rather than a stub.
+ """
+ from concurrent.futures import ThreadPoolExecutor
+ from unittest.mock import MagicMock
+
+ from vllm.parser.harmony import HarmonyParser
+ from vllm.renderers.online_derenderer import OnlineDerenderer
+
+ renderer = MagicMock()
+ renderer.get_tokenizer.return_value = harmony_tokenizer
+ renderer._executor = ThreadPoolExecutor(max_workers=2)
+
+ model_config = MagicMock()
+ model_config.model = HARMONY_MODEL
+ model_config.hf_config.model_type = "gpt_oss"
+ model_config.hf_text_config.model_type = "gpt_oss"
+ model_config.hf_overrides = None
+
+ dr = OnlineDerenderer(
+ model_config=model_config,
+ renderer=renderer,
+ request_logger=None,
+ chat_template=None,
+ chat_template_content_format="string",
+ enable_auto_tools=True,
+ tool_parser="openai",
+ reasoning_parser="openai_gptoss",
+ )
+ assert dr.use_harmony
+ assert dr.parser is HarmonyParser
+ return dr
+
+
+async def _stream_harmony_deltas(
+ derenderer,
+ chat_request,
+ output_ids: list[int],
+ prompt_ids: list[int],
+ chunk_size: int,
+) -> list[DeltaMessage]:
+ """Drive `output_ids` through the parser path in fixed size chunks.
+
+ Returns the per chunk `DeltaMessage`s in order so callers can assert on
+ intermediate emissions, not just the assembled result.
+ """
+ deltas: list[DeltaMessage] = []
+ state = None
+ for start in range(0, len(output_ids), chunk_size):
+ tids = output_ids[start : start + chunk_size]
+ is_last = start + chunk_size >= len(output_ids)
+ chunk, state = await derenderer.derender_chat_stream(
+ model=HARMONY_MODEL,
+ generate_chunk=_make_stream_chunk(
+ tids, finish_reason="stop" if is_last else None
+ ),
+ state=state,
+ chat_request=chat_request,
+ prompt_token_ids=prompt_ids,
+ )
+ deltas.append(chunk.choices[0].delta)
+ return deltas
+
+
+class TestDerenderChatStreamHarmony:
+ """derender_chat_stream: replay + `parse_delta` against real HarmonyParser.
+
+ Skipped where `openai_harmony` is not installed. `prompt_token_ids` is
+ threaded through for call shape fidelity only, since HarmonyParser
+ derives its state from the output tokens alone.
+ """
+
+ @pytest.mark.asyncio
+ async def test_reasoning_never_leaks_as_content_midstream(
+ self, harmony_derenderer, harmony_encode
+ ):
+ """Regression guard for the RFC's original replay + diff design.
+
+ `HarmonyParser.parse()` always flushes to EOS. Mid-stream that
+ raises `HarmonyError` and the recovery branch re-emits the in
+ flight message on the `final` channel, so partial analysis surfaces
+ as content. Replay + `parse_delta` must never do that. Driven one
+ token per chunk, the finest granularity a client can produce.
+ """
+ output_ids = harmony_encode(HARMONY_OUTPUT)
+ prompt_ids = harmony_encode(HARMONY_PROMPT)
+ chat_request = _chat_request(model=HARMONY_MODEL, include_reasoning=True)
+
+ deltas = await _stream_harmony_deltas(
+ harmony_derenderer, chat_request, output_ids, prompt_ids, chunk_size=1
+ )
+
+ reasoning = ""
+ content = ""
+ for i, delta in enumerate(deltas):
+ reasoning += delta.reasoning or ""
+ content += delta.content or ""
+ # Any analysis text emitted as content breaks the prefix
+ # property, since the two channels share no prefix here.
+ assert HARMONY_ANSWER.startswith(content), (
+ f"content after chunk {i} is not a prefix of the final "
+ f"channel text: {content!r}"
+ )
+ assert HARMONY_REASONING.startswith(reasoning)
+ assert not delta.tool_calls
+
+ assert reasoning == HARMONY_REASONING
+ assert content == HARMONY_ANSWER
+
+ @pytest.mark.asyncio
+ async def test_stream_matches_batch(self, harmony_derenderer, harmony_encode):
+ """Streamed assembly equals one shot `/derender` over the same IDs."""
+ output_ids = harmony_encode(HARMONY_OUTPUT)
+ prompt_ids = harmony_encode(HARMONY_PROMPT)
+ chat_request = _chat_request(model=HARMONY_MODEL, include_reasoning=True)
+
+ deltas = await _stream_harmony_deltas(
+ harmony_derenderer, chat_request, output_ids, prompt_ids, chunk_size=3
+ )
+ reasoning = "".join(d.reasoning or "" for d in deltas)
+ content = "".join(d.content or "" for d in deltas)
+
+ batch_choices = await harmony_derenderer.derender_chat(
+ GenerateResponse(
+ request_id="test-harmony-batch",
+ choices=[
+ GenerateResponseChoice(
+ index=0, token_ids=output_ids, finish_reason="stop"
+ )
+ ],
+ ),
+ chat_request,
+ )
+ message = batch_choices[0].message
+
+ assert reasoning == message.reasoning
+ assert content == message.content
+
+
class TestDerenderStreamStateValidation:
"""DerenderStreamState rejects malformed caller supplied offsets/lengths."""
@@ -446,6 +1012,24 @@ class TestDerenderStreamStateValidation:
state = DerenderStreamState(prev_tokens=["a"] * 1024)
assert len(state.prev_tokens) == 1024
+ def test_output_chunk_lens_mismatch_rejected(self):
+ from pydantic import ValidationError
+
+ with pytest.raises(ValidationError):
+ DerenderStreamState(output_token_ids=[1, 2, 3], output_chunk_lens=[1, 1])
+
+ def test_output_chunk_lens_zero_entry_rejected(self):
+ from pydantic import ValidationError
+
+ with pytest.raises(ValidationError):
+ DerenderStreamState(output_token_ids=[1, 2], output_chunk_lens=[0, 2])
+
+ def test_output_chunk_lens_matching_accepted(self):
+ state = DerenderStreamState(
+ output_token_ids=[1, 2, 3], output_chunk_lens=[1, 2]
+ )
+ assert state.output_chunk_lens == [1, 2]
+
class TestServingDerenderStreamErrorHandling:
"""Malformed stream_state must surface as 400 and not an unhandled 500."""
@@ -458,8 +1042,10 @@ class TestServingDerenderStreamErrorHandling:
models = MagicMock()
models.is_base_model.return_value = True
models.model_config = MagicMock()
+ models.model_config.max_model_len = 100_000
online_derenderer = MagicMock()
+ online_derenderer.parser = None
online_derenderer.derender_completion_stream = AsyncMock(
side_effect=side_effect
)
@@ -504,6 +1090,179 @@ class TestServingDerenderStreamErrorHandling:
assert result.error.code == 400
+class TestServingDerenderStreamValidation:
+ """If checks cannot run in `derender_chat_stream_response`: all must
+ reject with 400 before ever calling into `online_derenderer` (i.e.
+ before touching the tokenizer)."""
+
+ def _make_serving(self, *, parser_configured: bool, max_model_len: int = 100_000):
+ from unittest.mock import AsyncMock, MagicMock
+
+ from vllm.entrypoints.openai.chat_completion.protocol import (
+ ChatCompletionStreamResponse,
+ )
+ from vllm.entrypoints.scale_out.derender.serving import ServingDerender
+
+ models = MagicMock()
+ models.is_base_model.return_value = True
+ models.model_config = MagicMock()
+ models.model_config.max_model_len = max_model_len
+
+ online_derenderer = MagicMock()
+ online_derenderer.parser = MagicMock() if parser_configured else None
+ online_derenderer.derender_chat_stream = AsyncMock(
+ return_value=(
+ ChatCompletionStreamResponse(id="t", model=MODEL_NAME, choices=[]),
+ DerenderStreamState(),
+ )
+ )
+ return ServingDerender(models=models, online_derenderer=online_derenderer)
+
+ @pytest.mark.asyncio
+ async def test_missing_chat_request_with_parser_rejected(self):
+ from vllm.entrypoints.scale_out.token_in_token_out.protocol import (
+ DerenderChatStreamRequest,
+ )
+
+ serving = self._make_serving(parser_configured=True)
+ request = DerenderChatStreamRequest(
+ stream=True,
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk([1, 2]),
+ )
+ result = await serving.derender_chat_stream_response(request)
+ assert isinstance(result, ErrorResponse)
+ assert result.error.code == 400
+ assert "chat_request" in result.error.message
+ serving.online_derenderer.derender_chat_stream.assert_not_called()
+
+ @pytest.mark.asyncio
+ async def test_missing_chat_request_without_parser_ok(self):
+ from vllm.entrypoints.scale_out.token_in_token_out.protocol import (
+ DerenderChatStreamRequest,
+ )
+
+ serving = self._make_serving(parser_configured=False)
+ request = DerenderChatStreamRequest(
+ stream=True,
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk([1, 2]),
+ )
+ result = await serving.derender_chat_stream_response(request)
+ assert not isinstance(result, ErrorResponse)
+
+ @pytest.mark.asyncio
+ async def test_missing_prompt_token_ids_with_parser_rejected(self):
+ """A parser configured model must reject a missing prompt_token_ids
+ the same way it rejects a missing chat_request. Without it,
+ parse_delta cannot tell whether the prompt left reasoning open and
+ would misclassify reasoning content as plain content."""
+ from vllm.entrypoints.scale_out.token_in_token_out.protocol import (
+ DerenderChatStreamRequest,
+ )
+
+ serving = self._make_serving(parser_configured=True)
+ request = DerenderChatStreamRequest(
+ stream=True,
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk([1, 2]),
+ chat_request=_chat_request(),
+ )
+ result = await serving.derender_chat_stream_response(request)
+ assert isinstance(result, ErrorResponse)
+ assert result.error.code == 400
+ assert "prompt_token_ids" in result.error.message
+ serving.online_derenderer.derender_chat_stream.assert_not_called()
+
+ @pytest.mark.asyncio
+ async def test_prompt_token_ids_present_with_parser_ok(self):
+ from vllm.entrypoints.scale_out.token_in_token_out.protocol import (
+ DerenderChatStreamRequest,
+ )
+
+ serving = self._make_serving(parser_configured=True)
+ request = DerenderChatStreamRequest(
+ stream=True,
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk([1, 2]),
+ chat_request=_chat_request(),
+ prompt_token_ids=[1, 2, 3],
+ )
+ result = await serving.derender_chat_stream_response(request)
+ assert not isinstance(result, ErrorResponse)
+
+ @pytest.mark.asyncio
+ async def test_oversized_output_token_ids_rejected(self):
+ from vllm.entrypoints.scale_out.token_in_token_out.protocol import (
+ DerenderChatStreamRequest,
+ )
+
+ serving = self._make_serving(parser_configured=False, max_model_len=4)
+ request = DerenderChatStreamRequest(
+ stream=True,
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk([1, 2, 3]),
+ stream_state=DerenderStreamState(
+ output_token_ids=[1, 2], output_chunk_lens=[2]
+ ),
+ )
+ result = await serving.derender_chat_stream_response(request)
+ assert isinstance(result, ErrorResponse)
+ assert result.error.code == 400
+ assert "max_model_len" in result.error.message
+ serving.online_derenderer.derender_chat_stream.assert_not_called()
+
+ @pytest.mark.asyncio
+ async def test_oversized_prompt_token_ids_rejected(self):
+ """prompt_token_ids is caller supplied and otherwise unbounded. A
+ parser configured deployment rescans it in full on every chunk
+ (is_reasoning_end / adjust_initial_state_from_prompt), so it must be
+ bounded the same way output_token_ids is."""
+ from vllm.entrypoints.scale_out.token_in_token_out.protocol import (
+ DerenderChatStreamRequest,
+ )
+
+ serving = self._make_serving(parser_configured=True, max_model_len=4)
+ request = DerenderChatStreamRequest(
+ stream=True,
+ model=MODEL_NAME,
+ generate_chunk=_make_stream_chunk([1]),
+ chat_request=_chat_request(),
+ prompt_token_ids=[1, 2, 3, 4, 5],
+ )
+ result = await serving.derender_chat_stream_response(request)
+ assert isinstance(result, ErrorResponse)
+ assert result.error.code == 400
+ assert "max_model_len" in result.error.message
+ serving.online_derenderer.derender_chat_stream.assert_not_called()
+
+ @pytest.mark.asyncio
+ async def test_too_many_choices_rejected(self):
+ """Each streamed chunk contains at most one choice (a single
+ DerenderStreamState is threaded through every choice). The check
+ could never fire since derender_chat_stream itself rejects anything
+ above 1 first."""
+ from vllm.entrypoints.scale_out.token_in_token_out.protocol import (
+ DerenderChatStreamRequest,
+ )
+
+ serving = self._make_serving(parser_configured=False)
+ two_choices = GenerateStreamResponse(
+ request_id="t",
+ choices=[
+ GenerateResponseStreamChoice(index=i, token_ids=[1]) for i in range(2)
+ ],
+ )
+ request = DerenderChatStreamRequest(
+ stream=True, model=MODEL_NAME, generate_chunk=two_choices
+ )
+ result = await serving.derender_chat_stream_response(request)
+ assert isinstance(result, ErrorResponse)
+ assert result.error.code == 400
+ assert "at most one choice" in result.error.message
+ serving.online_derenderer.derender_chat_stream.assert_not_called()
+
+
# ---------------------------------------------------------------------------
# Integration tests — require a live render server
# ---------------------------------------------------------------------------
@@ -757,3 +1516,416 @@ async def test_streaming_usage_chunk(client):
assert d2["chunk"]["choices"] == []
assert d2["chunk"]["usage"]["prompt_tokens"] == 10
assert d2["chunk"]["usage"]["completion_tokens"] == len(token_ids)
+
+
+# ---------------------------------------------------------------------------
+# Integration tests for parser configured (reasoning + tool calls) require a
+# live render server. Mirrors the parser_server / parser_tokenizer pattern
+# from test_derender.py.
+# ---------------------------------------------------------------------------
+
+PARSER_MODEL = "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B"
+
+_PARSER_TOOLS = [
+ {
+ "type": "function",
+ "function": {
+ "name": "get_weather",
+ "description": "Get weather for a city",
+ "parameters": {
+ "type": "object",
+ "properties": {"city": {"type": "string"}},
+ },
+ },
+ }
+]
+
+
+@pytest.fixture(scope="module")
+def parser_server():
+ from tests.utils import RemoteLaunchRenderServer
+
+ args = [
+ "--enable-auto-tool-choice",
+ "--tool-call-parser",
+ "hermes",
+ "--reasoning-parser",
+ "deepseek_r1",
+ ]
+ with RemoteLaunchRenderServer(PARSER_MODEL, args) as remote_server:
+ yield remote_server
+
+
+@pytest_asyncio.fixture
+async def parser_client(parser_server):
+ import httpx
+
+ async with httpx.AsyncClient(
+ base_url=parser_server.url_for(""), timeout=60.0
+ ) as http_client:
+ yield http_client
+
+
+@pytest.fixture(scope="module")
+def parser_tokenizer():
+ from vllm.tokenizers import get_tokenizer
+
+ return get_tokenizer(PARSER_MODEL)
+
+
+def _require_parser_markers(tokenizer, text: str, *markers: str) -> list[int]:
+ """Encode text and skip the test if any marker is lost in roundtrip."""
+ ids = tokenizer.encode(text, add_special_tokens=False)
+ decoded = tokenizer.decode(ids, skip_special_tokens=False)
+ for m in markers:
+ if m not in decoded:
+ pytest.skip(f"Marker {m!r} lost in encode->decode roundtrip")
+ return ids
+
+
+async def _render_parser_chat(client, messages: list[dict]) -> dict:
+ resp = await client.post(
+ "/v1/chat/completions/render",
+ json={"model": PARSER_MODEL, "messages": messages},
+ )
+ assert resp.status_code == 200, resp.text
+ return resp.json()
+
+
+async def _stream_chat_derender(
+ client,
+ output_ids: list[int],
+ chunk_sizes: list[int],
+ chat_request: dict,
+ prompt_tokens: int,
+ prompt_token_ids: list[int],
+ on_chunk: Callable[[list[dict]], None] | None = None,
+) -> dict:
+ """Feed `output_ids` through the streaming chat derender endpoint in
+ the given `chunk_sizes`, threading `stream_state` across calls and
+ return the assembled message.
+
+ If `on_chunk` is given, it is called after every chunk with a
+ snapshot (deep copy) of the `tool_calls` accumulator so far, letting
+ callers assert properties of the intermediate deltas (e.g. monotonic
+ argument growth) rather than only the final assembled result.
+ """
+ state = None
+ content = ""
+ reasoning = ""
+ tool_calls: list[dict] = []
+ finish_reason = None
+
+ pos = 0
+ for i, size in enumerate(chunk_sizes):
+ tids = output_ids[pos : pos + size]
+ pos += size
+ is_last = i == len(chunk_sizes) - 1
+ resp = await client.post(
+ "/v1/chat/completions/derender",
+ json={
+ "stream": True,
+ "model": PARSER_MODEL,
+ "generate_chunk": {
+ "request_id": "stream-test",
+ "choices": [
+ {
+ "index": 0,
+ "token_ids": tids,
+ "finish_reason": "stop" if is_last else None,
+ }
+ ],
+ },
+ "stream_state": state,
+ "prompt_tokens": prompt_tokens,
+ "prompt_token_ids": prompt_token_ids,
+ "chat_request": chat_request,
+ },
+ )
+ assert resp.status_code == 200, resp.text
+ data = resp.json()
+ state = data["stream_state"]
+ delta = data["chunk"]["choices"][0]["delta"]
+ content += delta.get("content") or ""
+ reasoning += delta.get("reasoning") or ""
+ for tc in delta.get("tool_calls") or []:
+ idx = tc["index"]
+ while len(tool_calls) <= idx:
+ tool_calls.append({"id": None, "name": None, "arguments": ""})
+ if tc.get("id"):
+ tool_calls[idx]["id"] = tc["id"]
+ fn = tc.get("function") or {}
+ if fn.get("name"):
+ tool_calls[idx]["name"] = fn["name"]
+ if fn.get("arguments"):
+ tool_calls[idx]["arguments"] += fn["arguments"]
+ if is_last:
+ finish_reason = data["chunk"]["choices"][0]["finish_reason"]
+ if on_chunk is not None:
+ on_chunk([dict(tc) for tc in tool_calls])
+
+ return {
+ "content": content or None,
+ "reasoning": reasoning or None,
+ "tool_calls": tool_calls,
+ "finish_reason": finish_reason,
+ }
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize(
+ "chunking",
+ [
+ "whole",
+ "per_token",
+ pytest.param(
+ "triples",
+ marks=pytest.mark.xfail(
+ reason=" leaks into reasoning when it shares a "
+ "multi-token delta with reasoning text (vllm#55195)"
+ ),
+ ),
+ ],
+)
+async def test_stream_parsed_matches_batch_reasoning(
+ parser_client, parser_tokenizer, chunking
+):
+ """Streamed == batch: assembled reasoning/content equal the non
+ streaming `/derender` result over the same token IDs, for every
+ chunking of the same output (chunking invariance)."""
+ messages = [{"role": "user", "content": "What is 2+3?"}]
+ gen_req = await _render_parser_chat(parser_client, messages)
+
+ reasoning_text = "The user wants 2 plus 3. That is 5."
+ answer_text = "The answer is 5."
+ output_text = f"{reasoning_text}{answer_text}"
+ output_ids = _require_parser_markers(parser_tokenizer, output_text, "")
+
+ chat_request = {
+ "model": PARSER_MODEL,
+ "messages": messages,
+ "include_reasoning": True,
+ }
+
+ batch_resp = await parser_client.post(
+ "/v1/chat/completions/derender",
+ json={
+ "model": PARSER_MODEL,
+ "generate_response": {
+ "request_id": "batch",
+ "choices": [
+ {"index": 0, "token_ids": output_ids, "finish_reason": "stop"}
+ ],
+ },
+ "prompt_tokens": len(gen_req["token_ids"]),
+ "chat_request": chat_request,
+ },
+ )
+ assert batch_resp.status_code == 200, batch_resp.text
+ batch_msg = batch_resp.json()["choices"][0]["message"]
+
+ n = len(output_ids)
+ chunk_sizes = {
+ "whole": [n],
+ "per_token": [1] * n,
+ "triples": [3] * (n // 3) + [n % 3],
+ }[chunking]
+ chunk_sizes = [c for c in chunk_sizes if c > 0]
+ streamed = await _stream_chat_derender(
+ parser_client,
+ output_ids,
+ chunk_sizes,
+ chat_request,
+ len(gen_req["token_ids"]),
+ gen_req["token_ids"],
+ )
+ assert streamed["content"] == batch_msg["content"]
+ assert streamed["reasoning"] == batch_msg.get("reasoning")
+
+
+@pytest.mark.asyncio
+async def test_stream_parsed_matches_batch_tool_call(parser_client, parser_tokenizer):
+ """Tool call name+id+arguments parity between streamed and batch, plus
+ the finish_reason -> "tool_calls" rewrite for auto tool choice."""
+ messages = [{"role": "user", "content": "Weather in Paris?"}]
+ gen_req = await _render_parser_chat(parser_client, messages)
+
+ output_text = (
+ "Let me check."
+ '\n{"name": "get_weather", '
+ '"arguments": {"city": "Paris"}}\n'
+ )
+ output_ids = _require_parser_markers(
+ parser_tokenizer, output_text, "", "", ""
+ )
+ chat_request = {
+ "model": PARSER_MODEL,
+ "messages": messages,
+ "tools": _PARSER_TOOLS,
+ "tool_choice": "auto",
+ }
+
+ batch_resp = await parser_client.post(
+ "/v1/chat/completions/derender",
+ json={
+ "model": PARSER_MODEL,
+ "generate_response": {
+ "request_id": "batch",
+ "choices": [
+ {"index": 0, "token_ids": output_ids, "finish_reason": "stop"}
+ ],
+ },
+ "prompt_tokens": len(gen_req["token_ids"]),
+ "chat_request": chat_request,
+ },
+ )
+ assert batch_resp.status_code == 200, batch_resp.text
+ batch_choice = batch_resp.json()["choices"][0]
+ if not batch_choice["message"]["tool_calls"]:
+ pytest.skip("Model did not emit a tool call")
+
+ for chunk_sizes in ([len(output_ids)], [1] * len(output_ids)):
+ # Snapshot of tool_calls[0]["arguments"] after every chunk, to check
+ # it only ever grows by appending. It never retracts or duplicates
+ # already streamed text. That's the exact failure mode the
+ # replay + parse_delta design (over diff-based streaming) exists to
+ # avoid. Parity with the batch result alone wouldn't catch a
+ # transient mid-stream regression.
+ arg_snapshots: list[str] = []
+
+ def _record(
+ tool_calls: list[dict], _snapshots: list[str] = arg_snapshots
+ ) -> None:
+ if tool_calls and tool_calls[0]["arguments"]:
+ _snapshots.append(tool_calls[0]["arguments"])
+
+ streamed = await _stream_chat_derender(
+ parser_client,
+ output_ids,
+ chunk_sizes,
+ chat_request,
+ len(gen_req["token_ids"]),
+ gen_req["token_ids"],
+ on_chunk=_record,
+ )
+ assert streamed["tool_calls"]
+ assert streamed["tool_calls"][0]["name"] == "get_weather"
+ assert streamed["tool_calls"][0]["id"] is not None
+ assert json.loads(streamed["tool_calls"][0]["arguments"]) == json.loads(
+ batch_choice["message"]["tool_calls"][0]["function"]["arguments"]
+ )
+ assert streamed["finish_reason"] == "tool_calls"
+
+ assert arg_snapshots, "expected at least one tool-call argument delta"
+ for prev, curr in zip(arg_snapshots, arg_snapshots[1:]):
+ assert curr.startswith(prev), (
+ f"tool-call arguments regressed: {prev!r} -> {curr!r}"
+ )
+ assert arg_snapshots[-1] == streamed["tool_calls"][0]["arguments"]
+
+
+@pytest.mark.asyncio
+async def test_stream_parsed_cjk_across_chunk_boundaries(
+ parser_client, parser_tokenizer
+):
+ """CJK/emoji tokens split across chunk boundaries with a parser active
+ across the reasoning -> content transition (parser path regression
+ guard analogous to #46159 in the plain detok path)."""
+ messages = [{"role": "user", "content": "Reply in Chinese"}]
+ gen_req = await _render_parser_chat(parser_client, messages)
+
+ reasoning_text = "思考中"
+ answer_text = "你好世界 😀"
+ output_text = f"{reasoning_text}{answer_text}"
+ output_ids = _require_parser_markers(parser_tokenizer, output_text, "")
+
+ chat_request = {
+ "model": PARSER_MODEL,
+ "messages": messages,
+ "include_reasoning": True,
+ }
+ streamed = await _stream_chat_derender(
+ parser_client,
+ output_ids,
+ [1] * len(output_ids),
+ chat_request,
+ len(gen_req["token_ids"]),
+ gen_req["token_ids"],
+ )
+ assert "�" not in (streamed["content"] or "")
+ assert "�" not in (streamed["reasoning"] or "")
+ assert answer_text in (streamed["content"] or "")
+
+
+@pytest.mark.asyncio
+async def test_stream_parsed_include_reasoning_false(parser_client, parser_tokenizer):
+ """include_reasoning=False emits no reasoning deltas on the streaming
+ parser path."""
+ messages = [{"role": "user", "content": "What is 2+3?"}]
+ gen_req = await _render_parser_chat(parser_client, messages)
+
+ output_text = "reasoning hereThe answer is 5."
+ output_ids = _require_parser_markers(parser_tokenizer, output_text, "")
+
+ chat_request = {
+ "model": PARSER_MODEL,
+ "messages": messages,
+ "include_reasoning": False,
+ }
+ streamed = await _stream_chat_derender(
+ parser_client,
+ output_ids,
+ [1] * len(output_ids),
+ chat_request,
+ len(gen_req["token_ids"]),
+ gen_req["token_ids"],
+ )
+ assert streamed["reasoning"] is None
+
+
+@pytest.mark.asyncio
+async def test_stream_parsed_missing_chat_request_rejected(parser_client):
+ """Parser configured + no chat_request on the streaming endpoint -> 400
+ (the live-server counterpart to the mocked
+ `TestServingDerenderStreamValidation` checks above)."""
+ resp = await parser_client.post(
+ "/v1/chat/completions/derender",
+ json={
+ "stream": True,
+ "model": PARSER_MODEL,
+ "generate_chunk": {
+ "request_id": "reject-test",
+ "choices": [{"index": 0, "token_ids": [1, 2], "finish_reason": None}],
+ },
+ },
+ )
+ assert resp.status_code == 400
+ assert "chat_request" in resp.json()["error"]["message"]
+
+
+@pytest.mark.asyncio
+async def test_stream_parsed_missing_prompt_token_ids_rejected(parser_client):
+ """Parser configured + chat_request given but no prompt_token_ids on the
+ streaming endpoint -> 400 (the live-server counterpart to the mocked
+ `test_missing_prompt_token_ids_with_parser_rejected` above). Without
+ prompt_token_ids, parse_delta cannot tell whether the prompt left
+ reasoning open and would silently misclassify reasoning as content."""
+ messages = [{"role": "user", "content": "Hello"}]
+ resp = await parser_client.post(
+ "/v1/chat/completions/derender",
+ json={
+ "stream": True,
+ "model": PARSER_MODEL,
+ "generate_chunk": {
+ "request_id": "reject-test",
+ "choices": [{"index": 0, "token_ids": [1, 2], "finish_reason": None}],
+ },
+ "chat_request": {
+ "model": PARSER_MODEL,
+ "messages": messages,
+ "include_reasoning": True,
+ },
+ },
+ )
+ assert resp.status_code == 400
+ assert "prompt_token_ids" in resp.json()["error"]["message"]
diff --git a/vllm/entrypoints/scale_out/derender/serving.py b/vllm/entrypoints/scale_out/derender/serving.py
index 81e919d840..d3c5f3aac1 100644
--- a/vllm/entrypoints/scale_out/derender/serving.py
+++ b/vllm/entrypoints/scale_out/derender/serving.py
@@ -138,6 +138,13 @@ class ServingDerender(BaseServing):
if bounds_error is not None:
return bounds_error
+ if self.online_derenderer.parser is not None and request.chat_request is None:
+ return self.create_error_response(
+ "chat_request is required when a tool or reasoning parser is "
+ "configured because plain detokenization would leak raw parser "
+ "markup into content."
+ )
+
try:
choices = await self.online_derenderer.derender_chat(
request.generate_response, request.chat_request
@@ -248,14 +255,67 @@ class ServingDerender(BaseServing):
Processes one ``GenerateStreamResponse`` chunk and returns the
derendered chunk together with the updated client carried state.
-
- ``parser is None`` or no ``chat_request`` until reasoning/tool call
- functionality added in future PR.
"""
error_check_ret = await self._check_model(request)
if error_check_ret is not None:
return error_check_ret
+ if self.online_derenderer.parser is not None and request.chat_request is None:
+ return self.create_error_response(
+ "chat_request is required when a tool or reasoning parser is "
+ "configured because plain detokenization would leak raw parser "
+ "markup into content."
+ )
+
+ if (
+ self.online_derenderer.parser is not None
+ and request.prompt_token_ids is None
+ ):
+ return self.create_error_response(
+ "prompt_token_ids is required when a tool or reasoning "
+ "parser is configured. Without it parse_delta cannot tell "
+ "whether the prompt left reasoning open (e.g. a chat "
+ "template that pre-opens ) and would misclassify "
+ "reasoning content as plain content."
+ )
+
+ # Each streamed chunk contains at most one choice per SSE event.
+ # A single DerenderStreamState is threaded
+ # through every choice in the chunk, so >1 would corrupt detok/parser
+ # state across choices. See the matching check in
+ # OnlineDerenderer.derender_chat_stream which this fails ahead of to
+ # avoid touching the tokenizer at all on a malformed chunk.
+ num_choices = len(request.generate_chunk.choices)
+ if num_choices > 1:
+ return self.create_error_response(
+ f"derender_chat_stream expects at most one choice per chunk "
+ f"(got {num_choices})."
+ )
+
+ # prompt_token_ids is folded in here too. It is caller supplied and
+ # otherwise unbounded. A parser configured deployment rescans it
+ # in full on every chunk (is_reasoning_end /
+ # adjust_initial_state_from_prompt, see abstract_parser.parse_delta)
+ # since parser state cannot be carried across calls.
+ prompt_len = (
+ len(request.prompt_token_ids) if request.prompt_token_ids is not None else 0
+ )
+ prior_len = (
+ len(request.stream_state.output_token_ids)
+ if request.stream_state is not None
+ else 0
+ )
+ delta_len = sum(
+ len(c.token_ids) for c in request.generate_chunk.choices if c.token_ids
+ )
+ max_model_len = self.model_config.max_model_len
+ if prompt_len + prior_len + delta_len > max_model_len:
+ return self.create_error_response(
+ f"prompt_token_ids length ({prompt_len}) plus "
+ f"output_token_ids length ({prior_len}) plus delta "
+ f"({delta_len}) exceeds max_model_len ({max_model_len})."
+ )
+
model_name = request.model or self.models.model_name()
try:
chunk, updated_state = await self.online_derenderer.derender_chat_stream(
@@ -264,15 +324,21 @@ class ServingDerender(BaseServing):
state=request.stream_state,
chat_request=request.chat_request,
prompt_tokens=request.prompt_tokens,
+ prompt_token_ids=request.prompt_token_ids,
)
- except NotImplementedError as exc:
- return self.create_error_response(exc)
- except ValueError as exc:
- return self.create_error_response(str(exc))
- except (KeyError, IndexError) as exc:
- return self.create_error_response(
- f"invalid stream_state: detokenization failed ({exc!r})"
+ except Exception as exc:
+ # The two ValueErrors derender_chat_stream can raise directly
+ # (missing chat_request, >1 choice per chunk) are already
+ # pre-checked above, so anything reaching here comes from the
+ # parser replaying entirely client controlled state. Hermes
+ # swallows its own streaming exceptions but finalize_generation
+ # and engine based parsers don't, so an unexpected parser
+ # failure on malformed (but well typed) state must surface as
+ # a 400 here rather than an unhandled 500.
+ logger.debug(
+ "derender stream replay failed: %s: %s", type(exc).__name__, exc
)
+ return self.create_error_response("invalid stream_state: derender failed")
logger.debug(
"derender_chat_stream request_id=%s model=%s delta_tokens=%d",
@@ -313,7 +379,7 @@ class ServingDerender(BaseServing):
return self.create_error_response(str(exc))
except (KeyError, IndexError) as exc:
return self.create_error_response(
- f"invalid stream_state: detokenization failed ({exc!r})"
+ f"invalid stream_state: detokenization failed ({exc})"
)
logger.debug(
diff --git a/vllm/entrypoints/scale_out/token_in_token_out/protocol.py b/vllm/entrypoints/scale_out/token_in_token_out/protocol.py
index be8130e314..2bd47b1059 100644
--- a/vllm/entrypoints/scale_out/token_in_token_out/protocol.py
+++ b/vllm/entrypoints/scale_out/token_in_token_out/protocol.py
@@ -1,6 +1,6 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
-from typing import Any, Literal, TypeAlias
+from typing import Annotated, Any, Literal, TypeAlias
from pydantic import (
BaseModel,
@@ -483,16 +483,32 @@ class DerenderStreamState(BaseModel):
streaming derender endpoint. All fields are plain JSON serializable data.
No opaque tokenizer or parser internals are stored here.
+ Two separate sets of fields support two different streaming modes:
+
+ - For plain streaming (no parser configured including the completions path):
+ `prev_tokens`, `prefix_offset` and `read_offset` maintain a bounded incremental
+ decoding window. This requires O(window) transport and O(delta) computation
+ per chunk.
+ - For parser enabled chat streaming: `output_token_ids`, `output_chunk_lens`,
+ `tools_streamed` and `last_tool_call_ids` are used to replay `parse_delta()`
+ from scratch on every chunk because parser state cannot be serialized. This
+ incurs O(n) transport per chunk (O(n²) per generation) and O(n²)
+ `parse_delta()` calls per generation. Since many parsers re-scan the
+ entire accumulated text on each invocation, `parse_delta()` itself is
+ O(n) yielding a true worst case compute cost of O(n³) per generation.
+ No caching is performed. Work is bounded by `max_model_len`.
+ See `OnlineDerenderer._derender_chat_stream_parsed`.
+
The detokenization strategy carries the incremental decode offsets
directly rather than re-sending the whole token history each chunk.
- ``detokenize_incrementally`` only ever reads the trailing token window
- ``prev_tokens[prefix_offset:]``, so we carry just that tail plus the two
+ `detokenize_incrementally` only ever reads the trailing token window
+ `prev_tokens[prefix_offset:]`, so we carry just that tail plus the two
offsets. Each chunk resumes exactly where the last one stopped, including
- any partially processed multi-byte character (tracked by ``read_offset``),
+ any partially processed multi-byte character (tracked by `read_offset`),
then trims and rebases the window so it never grows with generation length.
Performance:
- - Compute per chunk is O(delta). One ``detokenize_incrementally`` call per
+ - Compute per chunk is O(delta). One `detokenize_incrementally` call per
new token, independent of how many tokens preceded it.
- Transport per chunk is O(window). The carried tail is bounded by the
incremental detokenization offset, so cumulative bytes over the wire are
@@ -500,18 +516,18 @@ class DerenderStreamState(BaseModel):
"""
prev_tokens: list[str] = Field(default_factory=list)
- """Trailing decode window. Token strings from ``prefix_offset`` onward.
+ """Trailing decode window. Token strings from `prefix_offset` onward.
Bounded, trimmed and rebased each chunk to the tail
- ``detokenize_incrementally`` still reads, so it does not grow with the
+ `detokenize_incrementally` still reads, so it does not grow with the
number of chunks.
"""
prefix_offset: int = Field(default=0, ge=0)
- """Prefix offset into ``prev_tokens`` for incremental detokenization."""
+ """Prefix offset into `prev_tokens` for incremental detokenization."""
read_offset: int = Field(default=0, ge=0)
- """Read offset into ``prev_tokens`` for incremental detokenization."""
+ """Read offset into `prev_tokens` for incremental detokenization."""
@field_validator("prev_tokens")
@classmethod
@@ -525,23 +541,53 @@ class DerenderStreamState(BaseModel):
return v
role_sent: bool = False
- """True once the initial ``role: "assistant"`` delta has been emitted.
+ """True once the initial `role: "assistant"` delta has been emitted.
Prevents re-emitting the role on subsequent chunks even when the detok
window is transiently empty (e.g. usage only final chunk).
"""
- # TODO: Properties used in follow on PR for tool call parsing
- last_content: str | None = None
- """Last emitted cumulative assistant content text."""
+ output_token_ids: list[int] = Field(default_factory=list)
+ """All output tokens seen so far. Parser path only.
- last_reasoning: str | None = None
- """Last emitted cumulative reasoning text."""
+ Replay buffer: each chunk rebuilds a fresh parser and replays every
+ token in here through `parse_delta` (discarding the result) before
+ processing the current chunk's tokens since parser internal state
+ cannot be serialized into this stateless model. Unavoidably O(n)
+ bounded by ``max_model_len`` (enforced server side, not by a field
+ validator here since the bound is model dependent).
+ """
+
+ output_chunk_lens: list[Annotated[int, Field(gt=0)]] = Field(default_factory=list)
+ """Token count of each chunk in `output_token_ids`. Parser path only.
+
+ Replay uses these to reproduce the original `parse_delta` call
+ boundaries. Must sum to `len(output_token_ids)`.
+ """
+
+ @model_validator(mode="after")
+ def _validate_output_chunk_lens(self) -> "DerenderStreamState":
+ total = sum(self.output_chunk_lens)
+ if total != len(self.output_token_ids):
+ raise ValueError(
+ f"output_chunk_lens must sum to len(output_token_ids) "
+ f"(got sum={total}, len(output_token_ids)="
+ f"{len(self.output_token_ids)})"
+ )
+ return self
+
+ tools_streamed: bool = False
+ """True once a tool call delta has been emitted. Parser path only.
+
+ Drives the `finish_reason` -> `"tool_calls"` rewrite on the final
+ chunk mirroring the generate streaming path.
+ """
last_tool_call_ids: list[str] = Field(default_factory=list)
- """Stable tool-call IDs, assigned once when each call first appears.
+ """Stable tool call IDs, assigned once when each call first appears.
- Prevents ID regeneration across re-parsing.
+ Indexed by tool call index. Parser path only. Prevents ID regeneration
+ when replay reprocesses a tool call that already has a pinned ID.
"""
@@ -570,6 +616,19 @@ class DerenderChatStreamRequest(BaseModel):
prompt_tokens: int | None = None
"""Prompt token count for usage. Forwarded from the render step."""
+ prompt_token_ids: list[int] | None = None
+ """Prompt token IDs. Required by the parser path's `parse_delta` to
+ settle its initial reasoning state (e.g. chat templates that pre-open
+ ````). `prompt_tokens` is a usage count and cannot serve this
+ purpose. Sourced from `GenerateRequest.token_ids` at the render step.
+
+ Rejected with a 400 (by `ServingDerender`) when a tool or reasoning
+ parser is configured and this is omitted. Without it, `parse_delta`
+ cannot tell whether the prompt left reasoning open and would silently
+ misclassify reasoning content as plain content. Unused on the plain
+ detokenization path.
+ """
+
chat_request: ChatCompletionRequest | None = None
"""The original (post adjust_request) ChatCompletionRequest from /render."""
diff --git a/vllm/renderers/online_derenderer.py b/vllm/renderers/online_derenderer.py
index 05f2710afb..a89bc843b8 100644
--- a/vllm/renderers/online_derenderer.py
+++ b/vllm/renderers/online_derenderer.py
@@ -4,7 +4,10 @@ from typing import Any
from vllm.config import ModelConfig
from vllm.entrypoints.chat_utils import ChatTemplateContentFormatOption
-from vllm.entrypoints.generate.base.protocol import DeltaMessage, ToolCall
+from vllm.entrypoints.generate.base.protocol import (
+ DeltaMessage,
+ ToolCall,
+)
from vllm.entrypoints.generate.base.serving import resolve_token_id_placeholder
from vllm.entrypoints.openai.chat_completion.protocol import (
ChatCompletionLogProbs,
@@ -29,6 +32,9 @@ from vllm.entrypoints.scale_out.token_in_token_out.protocol import (
)
from vllm.entrypoints.serve.engine.protocol import UsageInfo
from vllm.entrypoints.serve.utils.request_logger import RequestLogger
+from vllm.entrypoints.serve.utils.tool_calls_utils import (
+ maybe_filter_parallel_tool_calls,
+)
from vllm.logger import init_logger
from vllm.parser import Parser, ParserManager
from vllm.renderers import BaseRenderer
@@ -95,6 +101,11 @@ class OnlineDerenderer:
self._derender_completion_async = make_async(
self._derender_completion, executor=renderer._executor
)
+ # Replay is O(n) per chunk (unlike the O(delta) detok-only paths),
+ # so it must not run on the event loop either.
+ self._derender_chat_stream_parsed_async = make_async(
+ self._derender_chat_stream_parsed, executor=renderer._executor
+ )
async def derender_chat(
self,
@@ -144,6 +155,7 @@ class OnlineDerenderer:
tokenizer,
chat_request.tools,
chat_template_kwargs=chat_template_kwargs,
+ model_config=self.model_config,
)
reasoning, content, tool_calls = parser.parse(
decoded_text,
@@ -278,23 +290,30 @@ class OnlineDerenderer:
state: DerenderStreamState | None = None,
chat_request: ChatCompletionRequest | None = None,
prompt_tokens: int | None = None,
+ prompt_token_ids: list[int] | None = None,
) -> tuple[ChatCompletionStreamResponse, DerenderStreamState]:
"""Process one GenerateStreamResponse chunk for streaming chat derender.
- TODO: parse path for reasoning and tool calls is implemented in future PR.
-
- Unlike OpenAI's API, which always emits ``role: "assistant"`` on the
+ Unlike OpenAI's API, which always emits `role: "assistant"` on the
very first chunk, this emits it on the first chunk with a non empty
- ``choices`` list. A leading usage only chunk therefore defers the
+ `choices` list. A leading usage only chunk therefore defers the
role to the following content chunk instead of sending an empty
role only delta.
Args:
model: Model name for the response object.
- generate_chunk: One SSE chunk from ``/inference/v1/generate``.
- state: Client carried detok state (``None`` for first call).
- chat_request: Original ChatCompletionRequest from ``/render``.
+ generate_chunk: One SSE chunk from `/inference/v1/generate`.
+ state: Client carried detok state (`None` for first call).
+ chat_request: Original ChatCompletionRequest from `/render`.
+ Required when a reasoning or tool parser is configured
+ (validated by the caller — see `ServingDerender`) because
+ plain detokenization would leak raw parser markup into `content`.
prompt_tokens: Prompt token count for the usage chunk.
+ prompt_token_ids: Prompt token IDs. Parser path only. Lets
+ `parse_delta` settle its initial reasoning state. Required
+ when a reasoning or tool parser is configured (validated by
+ the caller — see ``ServingDerender``). Without it reasoning
+ left open by the prompt would be misclassified as content.
Returns:
(chunk, updated_state) — the derendered SSE chunk and the state
@@ -304,28 +323,36 @@ class OnlineDerenderer:
if state is None:
state = DerenderStreamState()
- if self.parser is not None:
- # TODO: Follow on PR will implement the parse path. Check on the
- # parser alone (fail closed). A parser configured model must never
- # fall through to plain detok on the streaming path, even when
- # ``chat_request`` is omitted or reasoning/tool markup would leak
- # into ``delta.content``.
- raise NotImplementedError(
- "Streaming chat derender is not yet supported for models with "
- "a reasoning or tool parser configured. Use the non-streaming "
- "derender endpoint (stream=false) for parsed output."
- )
-
# A single DerenderStreamState is threaded through every choice in
# this chunk. Correct only when there is at most one choice per SSE
# event (n=1, one call per index), as the streaming derender
# protocol assumes. Multiple choices sharing one chunk would corrupt
- # each other's detok window.
+ # each other's detok/parser state.
if len(generate_chunk.choices) > 1:
raise ValueError(
"derender_chat_stream expects at most one choice per chunk"
)
+ parser_cls = self.parser
+ if parser_cls is not None:
+ # Fail (mirrors ServingDerender's pre-check) because a parser
+ # configured model must never fall through to plain detok or
+ # reasoning/tool markup would leak into `delta.content`.
+ if chat_request is None:
+ raise ValueError(
+ "chat_request is required for streaming chat derender "
+ "when a tool or reasoning parser is configured"
+ )
+ return await self._derender_chat_stream_parsed_async(
+ parser_cls,
+ model,
+ generate_chunk,
+ state,
+ chat_request,
+ prompt_tokens,
+ prompt_token_ids,
+ )
+
tokenizer = self.renderer.get_tokenizer()
skip_special = (
chat_request.skip_special_tokens if chat_request is not None else True
@@ -374,6 +401,203 @@ class OnlineDerenderer:
)
return chunk, updated_state
+ def _derender_chat_stream_parsed(
+ self,
+ parser_cls: type[Parser],
+ model: str,
+ generate_chunk: GenerateStreamResponse,
+ state: DerenderStreamState,
+ chat_request: ChatCompletionRequest,
+ prompt_tokens: int | None,
+ prompt_token_ids: list[int] | None,
+ ) -> tuple[ChatCompletionStreamResponse, DerenderStreamState]:
+ """Parser path for streaming chat derender: replay + `parse_delta`.
+
+ Parser internal state (buffered markup, reasoning/tool phase, etc.)
+ cannot be serialized into `DerenderStreamState`, so each call
+ builds a fresh parser and replays every prior output token through
+ `parse_delta` (discarding the result) before processing this
+ chunk's tokens for real.
+
+ Every chunk, replayed or live, goes through one `parse_delta` call
+ with the tokens it arrived with (more than one under e.g.
+ speculative decoding). Replay recovers those boundaries from
+ `state.output_chunk_lens`, so the rebuilt parser state matches
+ standard serving, which calls `parse_delta` once per engine step.
+ Text comes from a fresh incremental detokenizer with special tokens
+ preserved (``skip_special_tokens=False``).
+ """
+ tokenizer = self.renderer.get_tokenizer()
+
+ chat_template_kwargs: dict[str, Any] = {}
+ if not self.use_harmony:
+ chat_template_kwargs = (
+ chat_request.build_chat_params(
+ self.chat_template,
+ self.chat_template_content_format,
+ )
+ .with_defaults(self.default_chat_template_kwargs)
+ .chat_template_kwargs
+ )
+
+ parser = parser_cls(
+ tokenizer,
+ chat_request.tools,
+ chat_template_kwargs=chat_template_kwargs,
+ model_config=self.model_config,
+ )
+
+ # Ephemeral incremental detok window, local to this call. Threaded
+ # across both the replay and current chunk phases (via `nonlocal`)
+ # so multi-byte characters split across that boundary still decode
+ # correctly. Discarded once the call returns.
+ detok_state = DerenderStreamState()
+
+ def _replay(token_ids: list[int], chunk_lens: list[int]) -> None:
+ """Replay prior chunks through `parse_delta` to rebuild parser
+ state, discarding the result. Never `finished` since that only
+ applies to the current chunk.
+ """
+ nonlocal detok_state
+ start = 0
+ for chunk_len in chunk_lens:
+ chunk = token_ids[start : start + chunk_len]
+ start += chunk_len
+ text, detok_state = self._detokenize_delta(
+ tokenizer, chunk, detok_state, skip_special_tokens=False
+ )
+ parser.parse_delta(
+ text,
+ chunk,
+ chat_request,
+ prompt_token_ids=prompt_token_ids,
+ finished=False,
+ )
+
+ # Replay history to reconstruct parser state. The result is thrown
+ # away and only the current chunk's emission goes to the client.
+ _replay(state.output_token_ids, state.output_chunk_lens)
+
+ stream_choices: list[ChatCompletionResponseStreamChoice] = []
+ output_token_ids = list(state.output_token_ids)
+ output_chunk_lens = list(state.output_chunk_lens)
+ role_sent = state.role_sent
+ tools_streamed = state.tools_streamed
+ last_tool_call_ids = list(state.last_tool_call_ids)
+
+ # At most one choice: the caller (derender_chat_stream) already
+ # rejects >1 before dispatching here. role_sent/tools_streamed/
+ # output_token_ids below are updated for that single choice, not
+ # accumulated across choices. Looping over generate_chunk.choices
+ # could silently corrupt output if chunks were ever allowed to
+ # contain multiple choices.
+ if generate_chunk.choices:
+ choice = generate_chunk.choices[0]
+ delta_tids = choice.token_ids or []
+ is_finished = choice.finish_reason is not None
+
+ if delta_tids:
+ # One parse_delta call for the whole chunk (producer
+ # granularity), not one per token. See the granularity note
+ # in this method's docstring.
+ text, detok_state = self._detokenize_delta(
+ tokenizer, delta_tids, detok_state, skip_special_tokens=False
+ )
+ delta_message = parser.parse_delta(
+ text,
+ delta_tids,
+ chat_request,
+ prompt_token_ids=prompt_token_ids,
+ finished=is_finished,
+ )
+ elif is_finished:
+ # Finish only chunk (no new tokens). Still flush any
+ # buffered tool call arguments.
+ delta_message = parser.parse_delta(
+ "",
+ [],
+ chat_request,
+ prompt_token_ids=prompt_token_ids,
+ finished=True,
+ )
+ else:
+ delta_message = None
+
+ if delta_tids:
+ output_token_ids.extend(delta_tids)
+ output_chunk_lens.append(len(delta_tids))
+
+ if delta_message is None:
+ delta_message = DeltaMessage()
+
+ if delta_message.tool_calls:
+ tools_streamed = True
+ for tc in delta_message.tool_calls:
+ if tc.id is None:
+ continue
+ if tc.index < len(last_tool_call_ids):
+ # Pin: reuse the ID already recorded for this index
+ # rather than one a from scratch replay regenerated.
+ # Real trigger not just defensive with
+ # tool_choice="required",
+ # extract_required_tool_call_streaming resets
+ # function_name_returned to False whenever the
+ # partial JSON transiently fails to parse which
+ # re-emits id+name for the same index on replay.
+ tc.id = last_tool_call_ids[tc.index]
+ else:
+ last_tool_call_ids.append(tc.id)
+
+ if not role_sent:
+ delta_message.role = "assistant"
+ role_sent = True
+
+ finish_reason = choice.finish_reason
+ if finish_reason is not None:
+ is_named_tool_choice = (
+ type(chat_request.tool_choice) is ChatCompletionNamedToolChoiceParam
+ )
+ if tools_streamed and not is_named_tool_choice:
+ finish_reason = "tool_calls"
+
+ stream_choice = ChatCompletionResponseStreamChoice(
+ index=choice.index,
+ delta=delta_message,
+ finish_reason=finish_reason,
+ )
+ stream_choices.append(
+ maybe_filter_parallel_tool_calls(stream_choice, chat_request)
+ )
+
+ updated_state = state.model_copy(
+ update={
+ "output_token_ids": output_token_ids,
+ "output_chunk_lens": output_chunk_lens,
+ "role_sent": role_sent,
+ "tools_streamed": tools_streamed,
+ "last_tool_call_ids": last_tool_call_ids,
+ }
+ )
+
+ usage: UsageInfo | None = None
+ if generate_chunk.usage is not None:
+ u = generate_chunk.usage
+ pt = prompt_tokens if prompt_tokens is not None else (u.prompt_tokens or 0)
+ ct = u.completion_tokens or 0
+ usage = UsageInfo(
+ prompt_tokens=pt,
+ completion_tokens=ct,
+ total_tokens=pt + ct,
+ )
+
+ chunk = ChatCompletionStreamResponse(
+ id=generate_chunk.request_id,
+ model=model,
+ choices=stream_choices,
+ usage=usage,
+ )
+ return chunk, updated_state
+
async def derender_completion(
self,
generate_responses: list[GenerateResponse],