[Frontend] Integer token IDs for generate output logprobs (GenerateLogProbs) (#58181)
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com> Co-authored-by: Yin Li <kevinli.ai.work@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -105,7 +105,7 @@ The server keeps no state between calls. Everything the next call needs is in `s
|
||||
- **Send `prompt_token_ids` on the first chunk, even without a parser.** It seeds detokenization so the first token's leading space matches the coupled endpoint (see [Request format](#request-format)). Later chunks already carry that context in `stream_state`.
|
||||
- **Don't forward `[DONE]`.** It marks the end of the generate stream and isn't a chunk.
|
||||
|
||||
Streaming derender resolves `logprobs` on the plain detokenization path, including `token_id:N` placeholders, same as the non-streaming endpoints (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them.
|
||||
`/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Streaming derender does the same per chunk on the plain detokenization path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them.
|
||||
|
||||
## Streaming cost
|
||||
|
||||
@@ -129,7 +129,7 @@ Streaming derender is stateless on the server side: all mutable state lives in t
|
||||
- `logprob_context_token_ids`: the trailing sampled token IDs (at most 4) from previous chunks, used to seed byte-fallback (U+FFFD) correction so multi-byte characters whose tokens split across chunk boundaries still resolve to real strings
|
||||
- `logprob_text_offset`: the cumulative emitted text length, so `text_offset` in completion streaming logprobs stays absolute across chunks instead of restarting at 0
|
||||
|
||||
When a streamed `GenerateTokensStreamChoice` carries `logprobs`, the `token_id:N` placeholders are resolved per chunk and the resolved logprobs are attached to the corresponding streamed choice. For chat it is as `ChatCompletionLogProbs` and for completions it is converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`.
|
||||
When a streamed `GenerateTokensStreamChoice` carries `logprobs`, the integer `token_id`s are decoded per chunk and the resolved logprobs are attached to the corresponding streamed choice. For chat it is as `ChatCompletionLogProbs` and for completions it is converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`.
|
||||
|
||||
## Example
|
||||
|
||||
|
||||
@@ -79,6 +79,60 @@ for the existing authentication boundaries.
|
||||
|
||||
For the post processing counterpart that turns generated token IDs back into OpenAI compatible responses, see the [Derenderer APIs](derenderer.md).
|
||||
|
||||
## Generate Output Logprobs
|
||||
|
||||
`/inference/v1/generate` is a token in / token out API, so its output logprobs
|
||||
identify tokens by integer ID rather than by the OpenAI string token. With
|
||||
`sampling_params.logprobs` set and `output_mode: "tokens"` (the default), each
|
||||
choice carries a `GenerateLogProbs`; `output_mode: "text"` returns decoded
|
||||
`ChatCompletionLogProbs` instead (see [token in, token out](token_in_token_out.md)):
|
||||
|
||||
```json
|
||||
{
|
||||
"logprobs": {
|
||||
"content": [
|
||||
{
|
||||
"token_id": 262,
|
||||
"logprob": -0.10,
|
||||
"rank": 1,
|
||||
"top_logprobs": [
|
||||
{"token_id": 262, "logprob": -0.10, "rank": 1},
|
||||
{"token_id": 257, "logprob": -1.20, "rank": 2}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `content` has one entry per generated token, in generation order.
|
||||
- `top_logprobs` is a list, not a dict: JSON turns dict keys into strings and
|
||||
the ordering would be implicit. It follows the engine's order: the sampled
|
||||
token first, then the remaining candidates in rank order. With non-greedy
|
||||
sampling the sampled token can sit outside the top k (for example ranks
|
||||
`[5, 1, 2]` at `logprobs=2`); it then takes one of the `logprobs` slots and
|
||||
the rank-k candidate is left out, as on the OpenAI endpoints.
|
||||
- `rank` is the token's rank in the vocabulary distribution (1 = most likely)
|
||||
on every entry, the sampled one included; a top-k candidate's rank is its
|
||||
top-k position. The list is not sorted by it, so sort by `rank` if you need
|
||||
rank order. It is `null` when the engine could not rank the token (a NaN
|
||||
logprob, which is sent as `-9999.0`).
|
||||
- There is no `token` or `bytes` field. The generate server has no tokenizer;
|
||||
[derender](derenderer.md) fills those in when it converts the response to the
|
||||
OpenAI shapes.
|
||||
- `prompt_logprobs` on the same response is unchanged
|
||||
(`list[dict[int, Logprob] | None]`).
|
||||
|
||||
!!! warning "Changed in this release"
|
||||
Output logprobs used to be `ChatCompletionLogProbs` with every token written
|
||||
as a `"token_id:N"` placeholder string, and `bytes` set to the UTF-8 bytes of
|
||||
that placeholder by the Rust frontend but left unset by the Python one.
|
||||
Clients that read only `content[i].logprob` are unaffected. Clients that
|
||||
parsed the placeholder should read `content[i].token_id` instead.
|
||||
`return_tokens_as_token_ids` on `/v1/chat/completions` and `/v1/completions`
|
||||
is unchanged: it is a user-facing OpenAI option and still uses the
|
||||
`token_id:N` format.
|
||||
|
||||
## Multimodal Render Features
|
||||
|
||||
Multimodal render responses include a `features` object with per-modality
|
||||
|
||||
@@ -43,7 +43,7 @@ print(response["choices"][0]["token_ids"])
|
||||
|
||||
### Logprobs
|
||||
|
||||
With `output_mode: "tokens"`, logprob entries carry `token_id:N` placeholders. With `output_mode: "text"`, they carry the decoded token strings and their UTF-8 `bytes` for the sampled token and every entry in `top_logprobs`. A server started with `--return-tokens-as-token-ids` always returns the placeholders, as `/v1/completions` does.
|
||||
With `output_mode: "tokens"`, logprobs are `GenerateLogProbs` with an integer `token_id` and `rank` per entry and no `token` or `bytes` (see [Generate Output Logprobs](renderer.md#generate-output-logprobs)). With `output_mode: "text"`, they are `ChatCompletionLogProbs` carrying the decoded token strings and their UTF-8 `bytes` for the sampled token and every entry in `top_logprobs`; a server started with `--return-tokens-as-token-ids` returns `token_id:N` placeholders there instead, as `/v1/completions` does.
|
||||
|
||||
### Errors
|
||||
|
||||
|
||||
Reference in New Issue
Block a user