[Frontend] Integer token IDs for generate output logprobs (GenerateLogProbs) (#58181)
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com> Co-authored-by: Yin Li <kevinli.ai.work@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -79,6 +79,60 @@ for the existing authentication boundaries.
|
||||
|
||||
For the post processing counterpart that turns generated token IDs back into OpenAI compatible responses, see the [Derenderer APIs](derenderer.md).
|
||||
|
||||
## Generate Output Logprobs
|
||||
|
||||
`/inference/v1/generate` is a token in / token out API, so its output logprobs
|
||||
identify tokens by integer ID rather than by the OpenAI string token. With
|
||||
`sampling_params.logprobs` set and `output_mode: "tokens"` (the default), each
|
||||
choice carries a `GenerateLogProbs`; `output_mode: "text"` returns decoded
|
||||
`ChatCompletionLogProbs` instead (see [token in, token out](token_in_token_out.md)):
|
||||
|
||||
```json
|
||||
{
|
||||
"logprobs": {
|
||||
"content": [
|
||||
{
|
||||
"token_id": 262,
|
||||
"logprob": -0.10,
|
||||
"rank": 1,
|
||||
"top_logprobs": [
|
||||
{"token_id": 262, "logprob": -0.10, "rank": 1},
|
||||
{"token_id": 257, "logprob": -1.20, "rank": 2}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `content` has one entry per generated token, in generation order.
|
||||
- `top_logprobs` is a list, not a dict: JSON turns dict keys into strings and
|
||||
the ordering would be implicit. It follows the engine's order: the sampled
|
||||
token first, then the remaining candidates in rank order. With non-greedy
|
||||
sampling the sampled token can sit outside the top k (for example ranks
|
||||
`[5, 1, 2]` at `logprobs=2`); it then takes one of the `logprobs` slots and
|
||||
the rank-k candidate is left out, as on the OpenAI endpoints.
|
||||
- `rank` is the token's rank in the vocabulary distribution (1 = most likely)
|
||||
on every entry, the sampled one included; a top-k candidate's rank is its
|
||||
top-k position. The list is not sorted by it, so sort by `rank` if you need
|
||||
rank order. It is `null` when the engine could not rank the token (a NaN
|
||||
logprob, which is sent as `-9999.0`).
|
||||
- There is no `token` or `bytes` field. The generate server has no tokenizer;
|
||||
[derender](derenderer.md) fills those in when it converts the response to the
|
||||
OpenAI shapes.
|
||||
- `prompt_logprobs` on the same response is unchanged
|
||||
(`list[dict[int, Logprob] | None]`).
|
||||
|
||||
!!! warning "Changed in this release"
|
||||
Output logprobs used to be `ChatCompletionLogProbs` with every token written
|
||||
as a `"token_id:N"` placeholder string, and `bytes` set to the UTF-8 bytes of
|
||||
that placeholder by the Rust frontend but left unset by the Python one.
|
||||
Clients that read only `content[i].logprob` are unaffected. Clients that
|
||||
parsed the placeholder should read `content[i].token_id` instead.
|
||||
`return_tokens_as_token_ids` on `/v1/chat/completions` and `/v1/completions`
|
||||
is unchanged: it is a user-facing OpenAI option and still uses the
|
||||
`token_id:N` format.
|
||||
|
||||
## Multimodal Render Features
|
||||
|
||||
Multimodal render responses include a `features` object with per-modality
|
||||
|
||||
Reference in New Issue
Block a user