[Frontend] Apply each endpoint's top_logprobs cut in derender (#59518)

Signed-off-by: Yin Li <kevinli.ai.work@gmail.com>
Co-authored-by: Yin Li <kevinli.ai.work@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
This commit is contained in:
Yin Li
2026-10-15 15:04:04 +00:00
committed by GitHub
co-authored by Yin Li Cyrus Leung
parent 011fa68ca9
commit 3dcfaeb4e8
12 changed files with 409 additions and 112 deletions
+2 -2
View File
@@ -89,7 +89,7 @@ The derenderer builds its tool and reasoning parsers from its own server flags p
A mismatch doesn't fail. The parser just splits `reasoning`, `content` and `tool_calls` differently from what `vllm serve` would return.
`/v1/completions/derender` only detokenizes. It never runs tool or reasoning parsers, the same as `/v1/completions` on `vllm serve`. It only reads `skip_special_tokens` from `completion_request`.
`/v1/completions/derender` only detokenizes. It never runs tool or reasoning parsers, the same as `/v1/completions` on `vllm serve`. From `completion_request` it reads `skip_special_tokens`, and `logprobs` and `logprob_token_ids` to cut `top_logprobs` the way `/v1/completions` does.
## Streaming
@@ -105,7 +105,7 @@ The server keeps no state between calls. Everything the next call needs is in `s
- **Send `prompt_token_ids` on the first chunk, even without a parser.** It seeds detokenization so the first token's leading space matches the coupled endpoint (see [Request format](#request-format)). Later chunks already carry that context in `stream_state`.
- **Don't forward `[DONE]`.** It marks the end of the generate stream and isn't a chunk.
`/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Streaming derender does the same per chunk on the plain detokenization path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them.
`/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Generate returns every top-k candidate the engine produced; derender keeps the ones the coupled endpoint would, with the same helpers: the first `top_logprobs` on chat (all of them for `top_logprobs=-1`) and the first `logprobs + 1` on completions (all of them for `logprobs=-1`). It reads those values from `chat_request` / `completion_request`, so without the original request, or with one that does not ask for logprobs, it returns every candidate. Requests that use `logprob_token_ids` get no output logprobs from generate (their `sampling_params.logprobs` is unset), so derender has none to return for them either. Streaming derender does the same per chunk on the plain detokenization path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them.
## Streaming cost
+17 -4
View File
@@ -108,10 +108,13 @@ choice carries a `GenerateLogProbs`; `output_mode: "text"` returns decoded
- `content` has one entry per generated token, in generation order.
- `top_logprobs` is a list, not a dict: JSON turns dict keys into strings and
the ordering would be implicit. It follows the engine's order: the sampled
token first, then the remaining candidates in rank order. With non-greedy
sampling the sampled token can sit outside the top k (for example ranks
`[5, 1, 2]` at `logprobs=2`); it then takes one of the `logprobs` slots and
the rank-k candidate is left out, as on the OpenAI endpoints.
token first, then the remaining candidates in rank order. It holds every
candidate the engine returned and is not cut to `logprobs`: k entries when
the sampled token is in the top k, k + 1 when non-greedy sampling picked a
token outside it (for example ranks `[5, 1, 2]` at `logprobs=2`). The OpenAI
endpoints cut this differently (`/v1/chat/completions` keeps the first k,
`/v1/completions` the first k + 1), so [derender](derenderer.md) applies the
cut of the endpoint it renders for.
- `rank` is the token's rank in the vocabulary distribution (1 = most likely)
on every entry, the sampled one included; a top-k candidate's rank is its
top-k position. The list is not sorted by it, so sort by `rank` if you need
@@ -132,6 +135,16 @@ choice carries a `GenerateLogProbs`; `output_mode: "text"` returns decoded
`return_tokens_as_token_ids` on `/v1/chat/completions` and `/v1/completions`
is unchanged: it is a user-facing OpenAI option and still uses the
`token_id:N` format.
`top_logprobs` is no longer cut to `max(logprobs, 1)` entries, in either
`output_mode`: it now holds every candidate the engine returned, which is
`logprobs + 1` entries when the sampled token is outside the top
`logprobs`. Clients that assume `len(top_logprobs) == logprobs` should
take the first `logprobs` entries after sorting by `rank`.
Derender applies each endpoint's cut from the original request, so send
`chat_request` / `completion_request` to derender whenever logprobs are
requested: without it derender returns every candidate, which on chat is
`top_logprobs + 1` entries where `/v1/chat/completions` returns
`top_logprobs`.
## Multimodal Render Features