[Render] Add round trip parity test and docs for derender (#48617)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
This commit is contained in:
@@ -137,8 +137,12 @@ For further details on renderer APIs, please refer to [this page](renderer.md).
|
||||
|
||||
### Derenderer APIs
|
||||
|
||||
- `/v1/completions/derender` - Derenderer completion requests
|
||||
- `/v1/chat/completions/derender` - Derenderer chat completion requests
|
||||
For further details on derenderer APIs, please refer to [this page](derenderer.md).
|
||||
|
||||
- [Chat Completions Derender API](derenderer.md) (`/v1/chat/completions/derender`)
|
||||
- Derender chat completion requests
|
||||
- [Completions Derender API](derenderer.md) (`/v1/completions/derender`)
|
||||
- Derender completion requests
|
||||
|
||||
## Tokenize APIs
|
||||
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
# Derenderer APIs
|
||||
|
||||
The derenderer API is the post processing counterpart to the [Renderer APIs](renderer.md). Where `/render` turns a request into token ID (preprocessing), `/derender` turns generated token IDs back into a fully formed OpenAI compatible response (detokenization, reasoning parsing, tool call parsing), all without a GPU.
|
||||
|
||||
This closes the loop for a token-in / token-out engine in disaggregated serving:
|
||||
|
||||
- **GPU less post processing**: Detokenization, reasoning parsing, and tool call parsing run on the same GPU less frontend that hosts `/render`
|
||||
- **Parser parity**: The derenderer reuses vLLM's tool and reasoning parsers, so a disaggregated deployment produces the same `content`/`reasoning`/ `tool_calls` split as a standard `vllm serve` server
|
||||
- **Non-streaming**: The endpoints expect a complete `GenerateResponse` with all token IDs present and perform one-shot parsing. Streaming derender would require a separate endpoint design and is not currently supported but is in the pipeline
|
||||
|
||||
Both endpoints are hosted by the GPU less rendering server started with [`vllm launch render`](../../cli/launch/render.md), alongside the `/render`
|
||||
endpoints.
|
||||
|
||||
## Pipeline
|
||||
|
||||
```text
|
||||
render generate derender
|
||||
request ───────────────▶ token_ids ─────────▶ token_ids ──────────▶ response
|
||||
(chat / (GPU less) (token-in / (GPU less) (OpenAI
|
||||
completion) │ token-out engine) ▲ compatible)
|
||||
└─────────────── request + prompt_tokens ──┘
|
||||
```
|
||||
|
||||
The derender step needs more than the engine's `token_ids`. It also consumes the original `chat_request`/`completion_request` and `prompt_tokens` carried over from the render step (see [Request format](#request-format)) so the tool and reasoning parsers have the context they need.
|
||||
|
||||
## API Reference
|
||||
|
||||
- Chat Completions Derender API (`/v1/chat/completions/derender`)
|
||||
- Post process a single `GenerateResponse` into a `ChatCompletionResponse`
|
||||
- Completions Derender API (`/v1/completions/derender`)
|
||||
- Post process a list of `GenerateResponse` objects (one per prompt) into a `CompletionResponse`
|
||||
|
||||
## Request format
|
||||
|
||||
Each request wraps the engine's `GenerateResponse`(s) together with the caller metadata needed to reconstruct the final response without a GPU.
|
||||
|
||||
`/v1/chat/completions/derender`:
|
||||
|
||||
??? code
|
||||
|
||||
```python
|
||||
--8<-- "vllm/entrypoints/scale_out/token_in_token_out/protocol.py:derender-chat-request"
|
||||
```
|
||||
|
||||
`/v1/completions/derender`:
|
||||
|
||||
??? code
|
||||
|
||||
```python
|
||||
--8<-- "vllm/entrypoints/scale_out/token_in_token_out/protocol.py:derender-completion-request"
|
||||
```
|
||||
|
||||
Oversized payloads are rejected with a `400` before any `tokenizer.decode()` or parser runs.
|
||||
|
||||
## Example
|
||||
|
||||
The example below drives the full `render → generate → derender` round trip for a chat request against a GPU less render server (`/render`, `/derender`) and a token-in / token-out engine (`/inference/v1/generate`).
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
MODEL = "meta-llama/Llama-3.2-1B-Instruct"
|
||||
RENDER = "http://localhost:8100" # vllm launch render ...
|
||||
ENGINE = "http://localhost:8200" # token-in / token-out engine
|
||||
|
||||
chat_request = {
|
||||
"model": MODEL,
|
||||
"messages": [{"role": "user", "content": "What is 2+2?"}],
|
||||
"max_tokens": 32,
|
||||
}
|
||||
|
||||
with httpx.Client(timeout=60.0) as client:
|
||||
# 1. Render: request -> token IDs (GPU less)
|
||||
generate_request = client.post(
|
||||
f"{RENDER}/v1/chat/completions/render", json=chat_request
|
||||
).json()
|
||||
prompt_tokens = len(generate_request["token_ids"])
|
||||
|
||||
# 2. Generate: token IDs -> token IDs (token-in / token-out engine)
|
||||
generate_response = client.post(
|
||||
f"{ENGINE}/inference/v1/generate", json=generate_request
|
||||
).json()
|
||||
|
||||
# 3. Derender: token IDs -> ChatCompletionResponse (GPU less)
|
||||
response = client.post(
|
||||
f"{RENDER}/v1/chat/completions/derender",
|
||||
json={
|
||||
"model": MODEL,
|
||||
"generate_response": generate_response,
|
||||
"prompt_tokens": prompt_tokens,
|
||||
"chat_request": chat_request,
|
||||
},
|
||||
).json()
|
||||
|
||||
print(response["choices"][0]["message"]["content"])
|
||||
```
|
||||
|
||||
Passing `chat_request` lets the derenderer run the configured tool and reasoning parsers. This means `response["choices"][0]["message"]` carries the same `content` / `reasoning` / `tool_calls` split a `vllm serve` server would produce. Omit `chat_request` for plain detokenization only.
|
||||
@@ -12,3 +12,5 @@ Our renderer API is designed to disaggregate the render phase(preprocessing) and
|
||||
- Render completion requests
|
||||
- [Chat Completions Render API](renderer.md) (`/v1/chat/completions/render`)
|
||||
- Render chat completions
|
||||
|
||||
For the post processing counterpart that turns generated token IDs back into OpenAI compatible responses, see the [Derenderer APIs](derenderer.md).
|
||||
|
||||
Reference in New Issue
Block a user