[Frontend] Apply each endpoint's top_logprobs cut in derender (#59518)
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com> Co-authored-by: Yin Li <kevinli.ai.work@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
This commit is contained in:
co-authored by
Yin Li
Cyrus Leung
parent
011fa68ca9
commit
3dcfaeb4e8
@@ -89,7 +89,7 @@ The derenderer builds its tool and reasoning parsers from its own server flags p
|
||||
|
||||
A mismatch doesn't fail. The parser just splits `reasoning`, `content` and `tool_calls` differently from what `vllm serve` would return.
|
||||
|
||||
`/v1/completions/derender` only detokenizes. It never runs tool or reasoning parsers, the same as `/v1/completions` on `vllm serve`. It only reads `skip_special_tokens` from `completion_request`.
|
||||
`/v1/completions/derender` only detokenizes. It never runs tool or reasoning parsers, the same as `/v1/completions` on `vllm serve`. From `completion_request` it reads `skip_special_tokens`, and `logprobs` and `logprob_token_ids` to cut `top_logprobs` the way `/v1/completions` does.
|
||||
|
||||
## Streaming
|
||||
|
||||
@@ -105,7 +105,7 @@ The server keeps no state between calls. Everything the next call needs is in `s
|
||||
- **Send `prompt_token_ids` on the first chunk, even without a parser.** It seeds detokenization so the first token's leading space matches the coupled endpoint (see [Request format](#request-format)). Later chunks already carry that context in `stream_state`.
|
||||
- **Don't forward `[DONE]`.** It marks the end of the generate stream and isn't a chunk.
|
||||
|
||||
`/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Streaming derender does the same per chunk on the plain detokenization path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them.
|
||||
`/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Generate returns every top-k candidate the engine produced; derender keeps the ones the coupled endpoint would, with the same helpers: the first `top_logprobs` on chat (all of them for `top_logprobs=-1`) and the first `logprobs + 1` on completions (all of them for `logprobs=-1`). It reads those values from `chat_request` / `completion_request`, so without the original request, or with one that does not ask for logprobs, it returns every candidate. Requests that use `logprob_token_ids` get no output logprobs from generate (their `sampling_params.logprobs` is unset), so derender has none to return for them either. Streaming derender does the same per chunk on the plain detokenization path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them.
|
||||
|
||||
## Streaming cost
|
||||
|
||||
|
||||
@@ -108,10 +108,13 @@ choice carries a `GenerateLogProbs`; `output_mode: "text"` returns decoded
|
||||
- `content` has one entry per generated token, in generation order.
|
||||
- `top_logprobs` is a list, not a dict: JSON turns dict keys into strings and
|
||||
the ordering would be implicit. It follows the engine's order: the sampled
|
||||
token first, then the remaining candidates in rank order. With non-greedy
|
||||
sampling the sampled token can sit outside the top k (for example ranks
|
||||
`[5, 1, 2]` at `logprobs=2`); it then takes one of the `logprobs` slots and
|
||||
the rank-k candidate is left out, as on the OpenAI endpoints.
|
||||
token first, then the remaining candidates in rank order. It holds every
|
||||
candidate the engine returned and is not cut to `logprobs`: k entries when
|
||||
the sampled token is in the top k, k + 1 when non-greedy sampling picked a
|
||||
token outside it (for example ranks `[5, 1, 2]` at `logprobs=2`). The OpenAI
|
||||
endpoints cut this differently (`/v1/chat/completions` keeps the first k,
|
||||
`/v1/completions` the first k + 1), so [derender](derenderer.md) applies the
|
||||
cut of the endpoint it renders for.
|
||||
- `rank` is the token's rank in the vocabulary distribution (1 = most likely)
|
||||
on every entry, the sampled one included; a top-k candidate's rank is its
|
||||
top-k position. The list is not sorted by it, so sort by `rank` if you need
|
||||
@@ -132,6 +135,16 @@ choice carries a `GenerateLogProbs`; `output_mode: "text"` returns decoded
|
||||
`return_tokens_as_token_ids` on `/v1/chat/completions` and `/v1/completions`
|
||||
is unchanged: it is a user-facing OpenAI option and still uses the
|
||||
`token_id:N` format.
|
||||
`top_logprobs` is no longer cut to `max(logprobs, 1)` entries, in either
|
||||
`output_mode`: it now holds every candidate the engine returned, which is
|
||||
`logprobs + 1` entries when the sampled token is outside the top
|
||||
`logprobs`. Clients that assume `len(top_logprobs) == logprobs` should
|
||||
take the first `logprobs` entries after sorting by `rank`.
|
||||
Derender applies each endpoint's cut from the original request, so send
|
||||
`chat_request` / `completion_request` to derender whenever logprobs are
|
||||
requested: without it derender returns every candidate, which on chat is
|
||||
`top_logprobs + 1` entries where `/v1/chat/completions` returns
|
||||
`top_logprobs`.
|
||||
|
||||
## Multimodal Render Features
|
||||
|
||||
|
||||
Reference in New Issue
Block a user