[Frontend] Apply each endpoint's top_logprobs cut in derender (#59518)
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com> Co-authored-by: Yin Li <kevinli.ai.work@gmail.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
This commit is contained in:
co-authored by
Yin Li
Cyrus Leung
parent
011fa68ca9
commit
3dcfaeb4e8
@@ -89,7 +89,7 @@ The derenderer builds its tool and reasoning parsers from its own server flags p
|
||||
|
||||
A mismatch doesn't fail. The parser just splits `reasoning`, `content` and `tool_calls` differently from what `vllm serve` would return.
|
||||
|
||||
`/v1/completions/derender` only detokenizes. It never runs tool or reasoning parsers, the same as `/v1/completions` on `vllm serve`. It only reads `skip_special_tokens` from `completion_request`.
|
||||
`/v1/completions/derender` only detokenizes. It never runs tool or reasoning parsers, the same as `/v1/completions` on `vllm serve`. From `completion_request` it reads `skip_special_tokens`, and `logprobs` and `logprob_token_ids` to cut `top_logprobs` the way `/v1/completions` does.
|
||||
|
||||
## Streaming
|
||||
|
||||
@@ -105,7 +105,7 @@ The server keeps no state between calls. Everything the next call needs is in `s
|
||||
- **Send `prompt_token_ids` on the first chunk, even without a parser.** It seeds detokenization so the first token's leading space matches the coupled endpoint (see [Request format](#request-format)). Later chunks already carry that context in `stream_state`.
|
||||
- **Don't forward `[DONE]`.** It marks the end of the generate stream and isn't a chunk.
|
||||
|
||||
`/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Streaming derender does the same per chunk on the plain detokenization path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them.
|
||||
`/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Generate returns every top-k candidate the engine produced; derender keeps the ones the coupled endpoint would, with the same helpers: the first `top_logprobs` on chat (all of them for `top_logprobs=-1`) and the first `logprobs + 1` on completions (all of them for `logprobs=-1`). It reads those values from `chat_request` / `completion_request`, so without the original request, or with one that does not ask for logprobs, it returns every candidate. Requests that use `logprob_token_ids` get no output logprobs from generate (their `sampling_params.logprobs` is unset), so derender has none to return for them either. Streaming derender does the same per chunk on the plain detokenization path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them.
|
||||
|
||||
## Streaming cost
|
||||
|
||||
|
||||
@@ -108,10 +108,13 @@ choice carries a `GenerateLogProbs`; `output_mode: "text"` returns decoded
|
||||
- `content` has one entry per generated token, in generation order.
|
||||
- `top_logprobs` is a list, not a dict: JSON turns dict keys into strings and
|
||||
the ordering would be implicit. It follows the engine's order: the sampled
|
||||
token first, then the remaining candidates in rank order. With non-greedy
|
||||
sampling the sampled token can sit outside the top k (for example ranks
|
||||
`[5, 1, 2]` at `logprobs=2`); it then takes one of the `logprobs` slots and
|
||||
the rank-k candidate is left out, as on the OpenAI endpoints.
|
||||
token first, then the remaining candidates in rank order. It holds every
|
||||
candidate the engine returned and is not cut to `logprobs`: k entries when
|
||||
the sampled token is in the top k, k + 1 when non-greedy sampling picked a
|
||||
token outside it (for example ranks `[5, 1, 2]` at `logprobs=2`). The OpenAI
|
||||
endpoints cut this differently (`/v1/chat/completions` keeps the first k,
|
||||
`/v1/completions` the first k + 1), so [derender](derenderer.md) applies the
|
||||
cut of the endpoint it renders for.
|
||||
- `rank` is the token's rank in the vocabulary distribution (1 = most likely)
|
||||
on every entry, the sampled one included; a top-k candidate's rank is its
|
||||
top-k position. The list is not sorted by it, so sort by `rank` if you need
|
||||
@@ -132,6 +135,16 @@ choice carries a `GenerateLogProbs`; `output_mode: "text"` returns decoded
|
||||
`return_tokens_as_token_ids` on `/v1/chat/completions` and `/v1/completions`
|
||||
is unchanged: it is a user-facing OpenAI option and still uses the
|
||||
`token_id:N` format.
|
||||
`top_logprobs` is no longer cut to `max(logprobs, 1)` entries, in either
|
||||
`output_mode`: it now holds every candidate the engine returned, which is
|
||||
`logprobs + 1` entries when the sampled token is outside the top
|
||||
`logprobs`. Clients that assume `len(top_logprobs) == logprobs` should
|
||||
take the first `logprobs` entries after sorting by `rank`.
|
||||
Derender applies each endpoint's cut from the original request, so send
|
||||
`chat_request` / `completion_request` to derender whenever logprobs are
|
||||
requested: without it derender returns every candidate, which on chat is
|
||||
`top_logprobs + 1` entries where `/v1/chat/completions` returns
|
||||
`top_logprobs`.
|
||||
|
||||
## Multimodal Render Features
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@ mod convert;
|
||||
mod types;
|
||||
mod validate;
|
||||
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::collections::HashMap;
|
||||
use std::convert::Infallible;
|
||||
use std::result::Result;
|
||||
use std::sync::Arc;
|
||||
@@ -158,7 +158,7 @@ async fn generate_chunk_stream(
|
||||
ResponseOptions {
|
||||
include_usage,
|
||||
include_continuous_usage,
|
||||
logprobs: output_logprobs,
|
||||
include_logprobs,
|
||||
// Ignored: raw generate streaming has no prompt-logprobs wire shape.
|
||||
include_prompt_logprobs: _,
|
||||
return_token_ids,
|
||||
@@ -213,15 +213,13 @@ async fn generate_chunk_stream(
|
||||
continue;
|
||||
}
|
||||
|
||||
let logprobs = if let Some(requested) = output_logprobs
|
||||
&& !token_ids.is_empty()
|
||||
{
|
||||
let logprobs = if include_logprobs && !token_ids.is_empty() {
|
||||
let logprobs = output.logprobs.as_ref().ok_or_else(|| {
|
||||
server_error!(
|
||||
"raw generate stream requested logprobs but generation returned none"
|
||||
)
|
||||
})?;
|
||||
Some(raw_logprobs_to_generate(logprobs, requested)?)
|
||||
Some(raw_logprobs_to_generate(logprobs)?)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
@@ -289,19 +287,19 @@ fn collect_generate(
|
||||
include_usage: _,
|
||||
// Ignored: continuous usage is a streaming-only option.
|
||||
include_continuous_usage: _,
|
||||
logprobs: output_logprobs,
|
||||
include_logprobs,
|
||||
include_prompt_logprobs,
|
||||
return_token_ids,
|
||||
}: ResponseOptions,
|
||||
mm_placeholders: Option<MultiModalPlaceholders>,
|
||||
) -> Result<GenerateResponse, ApiError> {
|
||||
let logprobs = if let Some(requested) = output_logprobs {
|
||||
let logprobs = if include_logprobs {
|
||||
let logprobs = collected.logprobs.as_ref().ok_or_else(|| {
|
||||
ApiError::server_error(
|
||||
"raw generate response requested logprobs but generation returned none".to_string(),
|
||||
)
|
||||
})?;
|
||||
Some(raw_logprobs_to_generate(logprobs, requested)?)
|
||||
Some(raw_logprobs_to_generate(logprobs)?)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
@@ -381,14 +379,11 @@ fn extract_mm_placeholders(features: Option<&[MmFeatureSpec]>) -> Option<MultiMo
|
||||
Some(placeholders)
|
||||
}
|
||||
|
||||
fn raw_logprobs_to_generate(
|
||||
logprobs: &Logprobs,
|
||||
requested: i32,
|
||||
) -> Result<GenerateLogProbs, ApiError> {
|
||||
fn raw_logprobs_to_generate(logprobs: &Logprobs) -> Result<GenerateLogProbs, ApiError> {
|
||||
let content = logprobs
|
||||
.positions
|
||||
.iter()
|
||||
.map(|position| position_to_generate_logprobs_content(position, requested))
|
||||
.map(position_to_generate_logprobs_content)
|
||||
.collect::<Result<Vec<_>, _>>()?;
|
||||
|
||||
Ok(GenerateLogProbs {
|
||||
@@ -409,12 +404,12 @@ fn raw_prompt_logprobs_to_maps(
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// `top_logprobs` matches the Python generate server: the engine's entries (the
|
||||
/// sampled token first, then ranks 1..k) with repeated token ids dropped, cut to
|
||||
/// `max(requested, 1)` entries, or all of them for `-1`.
|
||||
/// `top_logprobs` matches the Python generate server: every candidate the
|
||||
/// engine returned (the sampled token first, then ranks 1..k) with the repeated
|
||||
/// sampled entry dropped, and no cut. Derender applies each OpenAI endpoint's
|
||||
/// cut.
|
||||
fn position_to_generate_logprobs_content(
|
||||
position: &PositionLogprobs,
|
||||
requested: i32,
|
||||
) -> Result<GenerateLogProbsContent, ApiError> {
|
||||
let chosen = position.entries.first().ok_or_else(|| {
|
||||
ApiError::server_error(
|
||||
@@ -422,9 +417,9 @@ fn position_to_generate_logprobs_content(
|
||||
)
|
||||
})?;
|
||||
|
||||
// One pass: the first entry of each token id wins, as in Python's dict. The
|
||||
// sampled token is the only one the engine can list twice.
|
||||
let mut seen = HashSet::with_capacity(position.entries.len());
|
||||
// The engine row is the sampled token followed by the top k, which are
|
||||
// distinct; only the sampled token can repeat, at its own rank (anywhere in
|
||||
// 1..=k). Skip that repeat, as Python's dict does: linear, no allocation.
|
||||
Ok(GenerateLogProbsContent {
|
||||
token_id: chosen.token_id,
|
||||
logprob: clamp_logprob(chosen.logprob),
|
||||
@@ -432,9 +427,9 @@ fn position_to_generate_logprobs_content(
|
||||
top_logprobs: position
|
||||
.entries
|
||||
.iter()
|
||||
.filter(|entry| seen.insert(entry.token_id))
|
||||
.take(usize::try_from(requested).map_or(usize::MAX, |n| n.max(1)))
|
||||
.map(|entry| GenerateLogProb {
|
||||
.enumerate()
|
||||
.filter(|(i, entry)| *i == 0 || entry.token_id != chosen.token_id)
|
||||
.map(|(_, entry)| GenerateLogProb {
|
||||
token_id: entry.token_id,
|
||||
logprob: clamp_logprob(entry.logprob),
|
||||
rank: wire_rank(entry.rank),
|
||||
@@ -563,8 +558,8 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
fn top(position: &PositionLogprobs, requested: i32) -> Vec<(u32, Option<u32>)> {
|
||||
position_to_generate_logprobs_content(position, requested)
|
||||
fn top(position: &PositionLogprobs) -> Vec<(u32, Option<u32>)> {
|
||||
position_to_generate_logprobs_content(position)
|
||||
.unwrap()
|
||||
.top_logprobs
|
||||
.iter()
|
||||
@@ -576,24 +571,31 @@ mod tests {
|
||||
fn generate_top_logprobs_drop_the_repeated_sampled_token() {
|
||||
// Sampled token 7 is also top-1: the engine row is [7, 7, 8].
|
||||
let pos = position(&[(7, -0.1, 1), (7, -0.1, 1), (8, -2.0, 2)]);
|
||||
assert_eq!(top(&pos, 2), vec![(7, Some(1)), (8, Some(2))]);
|
||||
assert_eq!(top(&pos), vec![(7, Some(1)), (8, Some(2))]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generate_top_logprobs_keep_sampled_first_when_outside_top_k() {
|
||||
// Sampled token 50 has vocab rank 5; it takes a slot and rank 2 drops
|
||||
// out, as on the Python generate server and the OpenAI endpoints.
|
||||
fn generate_top_logprobs_keep_every_candidate_when_sampled_is_outside_top_k() {
|
||||
// Sampled token 50 has vocab rank 5 at logprobs=2: sampled first, then
|
||||
// ranks 1 and 2, all kept (derender cuts per endpoint).
|
||||
let pos = position(&[(50, -3.0, 5), (10, -0.2, 1), (20, -1.0, 2)]);
|
||||
assert_eq!(top(&pos, 2), vec![(50, Some(5)), (10, Some(1))]);
|
||||
assert_eq!(top(&pos), vec![(50, Some(5)), (10, Some(1)), (20, Some(2))]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generate_top_logprobs_respect_zero_and_all() {
|
||||
let pos = position(&[(7, -0.1, 1), (7, -0.1, 1), (8, -2.0, 2), (9, -3.0, 3)]);
|
||||
assert_eq!(top(&pos, 0), vec![(7, Some(1))]);
|
||||
fn generate_top_logprobs_drop_the_sampled_repeat_at_its_own_rank() {
|
||||
// Sampled token 30 is rank 3: the engine row is [30, 10, 20, 30, 40],
|
||||
// so the repeat is at entries[3], not entries[1].
|
||||
let pos = position(&[
|
||||
(30, -1.0, 3),
|
||||
(10, -0.2, 1),
|
||||
(20, -0.5, 2),
|
||||
(30, -1.0, 3),
|
||||
(40, -2.0, 4),
|
||||
]);
|
||||
assert_eq!(
|
||||
top(&pos, -1),
|
||||
vec![(7, Some(1)), (8, Some(2)), (9, Some(3))]
|
||||
top(&pos),
|
||||
vec![(30, Some(3)), (10, Some(1)), (20, Some(2)), (40, Some(4))]
|
||||
);
|
||||
}
|
||||
|
||||
@@ -606,7 +608,7 @@ mod tests {
|
||||
entries.extend((0..VOCAB).map(|t| (t, -1.0 - t as f32, t + 1)));
|
||||
let pos = position(&entries);
|
||||
let start = std::time::Instant::now();
|
||||
let got = top(&pos, -1);
|
||||
let got = top(&pos);
|
||||
assert!(start.elapsed() < std::time::Duration::from_secs(5));
|
||||
assert_eq!(got.len(), VOCAB as usize);
|
||||
assert_eq!(got[0], (5, Some(1)));
|
||||
@@ -616,10 +618,10 @@ mod tests {
|
||||
fn generate_rank_zero_from_a_nan_logprob_is_none() {
|
||||
// The engine reports rank 0 for a sampled token whose logprob is NaN.
|
||||
let pos = position(&[(7, f32::NAN, 0), (8, -0.2, 1)]);
|
||||
let content = position_to_generate_logprobs_content(&pos, 1).unwrap();
|
||||
let content = position_to_generate_logprobs_content(&pos).unwrap();
|
||||
assert_eq!(content.rank, None);
|
||||
assert_eq!(content.logprob, -9999.0);
|
||||
assert_eq!(top(&pos, 1), vec![(7, None)]);
|
||||
assert_eq!(top(&pos), vec![(7, None), (8, Some(1))]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -26,10 +26,8 @@ pub(super) struct ResponseOptions {
|
||||
pub include_usage: bool,
|
||||
/// Whether the caller asked for usage on every streamed chunk.
|
||||
pub include_continuous_usage: bool,
|
||||
/// Requested output logprobs (`sampling_params.logprobs`): `None` omits
|
||||
/// them, `Some(n)` returns `max(n, 1)` candidates per position, `Some(-1)`
|
||||
/// all of them.
|
||||
pub logprobs: Option<i32>,
|
||||
/// Whether the caller requested output logprobs on generate choices.
|
||||
pub include_logprobs: bool,
|
||||
/// Whether the caller requested top-level prompt logprobs.
|
||||
pub include_prompt_logprobs: bool,
|
||||
/// Whether the caller requested final prompt token metadata.
|
||||
@@ -58,7 +56,7 @@ pub(super) fn prepare_generate_request(
|
||||
.as_ref()
|
||||
.and_then(|options| options.continuous_usage_stats)
|
||||
.unwrap_or(false);
|
||||
let logprobs = request.sampling_params.inner.logprobs;
|
||||
let include_logprobs = request.sampling_params.inner.logprobs.is_some();
|
||||
let include_prompt_logprobs = request.sampling_params.inner.prompt_logprobs.is_some();
|
||||
let return_token_ids = request.return_token_ids.unwrap_or(false);
|
||||
let mut sampling_params = request.sampling_params.inner;
|
||||
@@ -98,7 +96,7 @@ pub(super) fn prepare_generate_request(
|
||||
options: ResponseOptions {
|
||||
include_usage,
|
||||
include_continuous_usage,
|
||||
logprobs,
|
||||
include_logprobs,
|
||||
include_prompt_logprobs,
|
||||
return_token_ids,
|
||||
},
|
||||
|
||||
@@ -139,7 +139,7 @@ def test_sentencepiece_leading_space_matches_engine():
|
||||
assert [t.token for t in entry.top_logprobs] == convert_ids_list_to_tokens(
|
||||
tokenizer, [spaced, bare]
|
||||
)
|
||||
completion = _convert_chat_logprobs_to_completion_logprobs(resolved)
|
||||
completion = _convert_chat_logprobs_to_completion_logprobs(resolved, logprobs)
|
||||
assert completion.top_logprobs == [{" true": -0.5, "true": -1.5}]
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,212 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""generate + derender must return the same `top_logprobs` as the coupled
|
||||
`/v1/chat/completions` and `/v1/completions` endpoints (#59513).
|
||||
|
||||
Both sides start from the same per-position engine dicts, built with the
|
||||
engine's own `append_logprobs_for_next_position` and detokenized the way the
|
||||
engine does, and go through the real coupled serving objects on one side and
|
||||
`/inference/v1/generate` + derender on the other. The cases include a sampled
|
||||
token outside the top k, where chat keeps k entries and completions k + 1; the
|
||||
greedy end-to-end parity tests never produce that case.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.entrypoints.openai.chat_completion.test_serving_chat import (
|
||||
_build_mock_engine,
|
||||
_build_serving_chat,
|
||||
)
|
||||
from tests.entrypoints.openai.completion.test_completion_error import (
|
||||
_build_serving_completion,
|
||||
)
|
||||
from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest
|
||||
from vllm.entrypoints.openai.completion.protocol import CompletionRequest
|
||||
from vllm.entrypoints.scale_out.token_in_token_out.protocol import GenerateLogProbs
|
||||
from vllm.entrypoints.scale_out.token_in_token_out.serving import ServingTokens
|
||||
from vllm.logprobs import append_logprobs_for_next_position
|
||||
from vllm.renderers.online_derenderer import (
|
||||
_chat_top_logprobs_limit,
|
||||
_completion_top_logprobs_limit,
|
||||
_convert_chat_logprobs_to_completion_logprobs,
|
||||
_resolve_logprobs,
|
||||
)
|
||||
from vllm.tokenizers import get_tokenizer
|
||||
from vllm.tokenizers.detokenizer_utils import convert_ids_list_to_tokens
|
||||
|
||||
MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM"
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def tokenizer():
|
||||
return get_tokenizer(MODEL_NAME)
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def serving_chat():
|
||||
return _build_serving_chat(_build_mock_engine())
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def serving_completion():
|
||||
return _build_serving_completion(_build_mock_engine())
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def vocab_ids(tokenizer):
|
||||
"""Ordinary word pieces (no byte-fallback tokens, which need the U+FFFD
|
||||
context repair that is compared elsewhere)."""
|
||||
ids = tokenizer.encode(
|
||||
"the quick brown fox jumps over a lazy dog while seven cats watch "
|
||||
"from an old wooden fence near the river bank",
|
||||
add_special_tokens=False,
|
||||
)
|
||||
pieces = tokenizer.convert_ids_to_tokens(ids)
|
||||
return list(dict.fromkeys(i for i, p in zip(ids, pieces) if "<0x" not in p))
|
||||
|
||||
|
||||
def _positions(tokenizer, vocab_ids, k, sampled_ranks):
|
||||
"""Engine output for one completion: per position the sampled id and its
|
||||
dict (sampled first, then ranks 1..k), with decoded tokens as the engine
|
||||
fills them. k = -1 returns every candidate, as for the whole vocabulary."""
|
||||
token_ids, dicts = [], []
|
||||
for pos, sampled_rank in enumerate(sampled_ranks):
|
||||
n_top = max(k, sampled_rank) if k != -1 else len(vocab_ids) - 1
|
||||
ranked = [vocab_ids[(pos + j) % len(vocab_ids)] for j in range(n_top + 1)]
|
||||
sampled = ranked[sampled_rank - 1]
|
||||
top = ranked[: len(ranked) - 1] if k == -1 else ranked[:k]
|
||||
ids = [sampled, *top]
|
||||
logprobs = [-0.25 * sampled_rank] + [-0.25 * (r + 1) for r in range(len(top))]
|
||||
per_position: list = []
|
||||
append_logprobs_for_next_position(
|
||||
per_position,
|
||||
ids,
|
||||
logprobs,
|
||||
convert_ids_list_to_tokens(tokenizer, ids),
|
||||
sampled_rank,
|
||||
len(top),
|
||||
)
|
||||
token_ids.append(sampled)
|
||||
dicts.append(per_position[0])
|
||||
return token_ids, dicts
|
||||
|
||||
|
||||
def _generate(token_ids, dicts) -> GenerateLogProbs:
|
||||
out = ServingTokens._create_tokens_logprobs(
|
||||
None, token_ids=token_ids, top_logprobs=dicts
|
||||
)
|
||||
# Over the wire, as a derender client would send it back.
|
||||
return GenerateLogProbs.model_validate_json(out.model_dump_json())
|
||||
|
||||
|
||||
# Sampled token rank per position: top-1, inside the top k, outside it.
|
||||
RANKS = [1, 2, 1, 7, 3, 9, 1]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("k", [0, 1, 2, 5, -1])
|
||||
def test_chat_derender_matches_coupled_chat(tokenizer, vocab_ids, serving_chat, k):
|
||||
token_ids, dicts = _positions(tokenizer, vocab_ids, k, RANKS)
|
||||
coupled = serving_chat._create_chat_logprobs(
|
||||
token_ids=token_ids,
|
||||
top_logprobs=dicts,
|
||||
tokenizer=tokenizer,
|
||||
num_output_top_logprobs=k,
|
||||
)
|
||||
request = ChatCompletionRequest(
|
||||
model=MODEL_NAME,
|
||||
messages=[{"role": "user", "content": "hi"}],
|
||||
logprobs=True,
|
||||
top_logprobs=k,
|
||||
)
|
||||
derendered = _resolve_logprobs(
|
||||
_generate(token_ids, dicts),
|
||||
tokenizer,
|
||||
top_limit=_chat_top_logprobs_limit(request),
|
||||
)
|
||||
assert derendered.model_dump() == coupled.model_dump()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("k", [0, 1, 2, 5, -1])
|
||||
def test_completion_derender_matches_coupled_completion(
|
||||
tokenizer, vocab_ids, serving_completion, k
|
||||
):
|
||||
token_ids, dicts = _positions(tokenizer, vocab_ids, k, RANKS)
|
||||
coupled = serving_completion._create_completion_logprobs(
|
||||
token_ids=token_ids,
|
||||
top_logprobs=dicts,
|
||||
num_output_top_logprobs=k,
|
||||
tokenizer=tokenizer,
|
||||
)
|
||||
request = CompletionRequest(model=MODEL_NAME, prompt="hi", logprobs=k)
|
||||
generated = _generate(token_ids, dicts)
|
||||
derendered = _convert_chat_logprobs_to_completion_logprobs(
|
||||
_resolve_logprobs(
|
||||
generated,
|
||||
tokenizer,
|
||||
top_limit=_completion_top_logprobs_limit(request),
|
||||
),
|
||||
generated,
|
||||
)
|
||||
assert derendered.model_dump() == coupled.model_dump()
|
||||
if k > 0:
|
||||
# The case #59513 is about: completions keeps k + 1 entries when the
|
||||
# sampled token is outside the top k.
|
||||
outside = [i for i, r in enumerate(RANKS) if r > k]
|
||||
assert all(len(derendered.top_logprobs[i]) == k + 1 for i in outside)
|
||||
if k == -1:
|
||||
# /v1/completions returns every candidate for logprobs=-1 (#55717).
|
||||
assert [len(top) for top in derendered.top_logprobs] == [len(d) for d in dicts]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"top_limit",
|
||||
[
|
||||
_chat_top_logprobs_limit(None),
|
||||
# A request that does not ask for logprobs says nothing about the cut.
|
||||
_chat_top_logprobs_limit(
|
||||
ChatCompletionRequest(
|
||||
model=MODEL_NAME, messages=[{"role": "user", "content": "hi"}]
|
||||
)
|
||||
),
|
||||
_completion_top_logprobs_limit(None),
|
||||
_completion_top_logprobs_limit(
|
||||
CompletionRequest(model=MODEL_NAME, prompt="hi")
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_without_a_logprobs_request_derender_keeps_every_candidate(
|
||||
tokenizer, vocab_ids, top_limit
|
||||
):
|
||||
token_ids, dicts = _positions(tokenizer, vocab_ids, 2, RANKS)
|
||||
derendered = _resolve_logprobs(
|
||||
_generate(token_ids, dicts), tokenizer, top_limit=top_limit
|
||||
)
|
||||
assert [len(e.top_logprobs) for e in derendered.content] == [len(d) for d in dicts]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("k", [-1, 0, 2])
|
||||
def test_completion_position_without_logprobs_stays_none(
|
||||
tokenizer, vocab_ids, serving_completion, k
|
||||
):
|
||||
"""A position the engine returned no logprobs for is `None` in
|
||||
`/v1/completions`, not `{}`; derender keeps it that way."""
|
||||
token_ids, dicts = _positions(tokenizer, vocab_ids, k, RANKS)
|
||||
dicts[3] = None
|
||||
coupled = serving_completion._create_completion_logprobs(
|
||||
token_ids=token_ids,
|
||||
top_logprobs=dicts,
|
||||
num_output_top_logprobs=k,
|
||||
tokenizer=tokenizer,
|
||||
)
|
||||
request = CompletionRequest(model=MODEL_NAME, prompt="hi", logprobs=k)
|
||||
generated = _generate(token_ids, dicts)
|
||||
derendered = _convert_chat_logprobs_to_completion_logprobs(
|
||||
_resolve_logprobs(
|
||||
generated,
|
||||
tokenizer,
|
||||
top_limit=_completion_top_logprobs_limit(request),
|
||||
),
|
||||
generated,
|
||||
)
|
||||
assert derendered.top_logprobs == coupled.top_logprobs
|
||||
assert derendered.top_logprobs[3] is None
|
||||
@@ -14,11 +14,11 @@ from vllm.logprobs import Logprob
|
||||
|
||||
def test_top_logprobs_alternatives_have_own_token_ids():
|
||||
"""Each top_logprobs alternative must carry its own integer token id."""
|
||||
# Engine dict at logprobs=2 with the sampled token 262 also top-1.
|
||||
result = ServingTokens._create_tokens_logprobs(
|
||||
None,
|
||||
token_ids=[262],
|
||||
top_logprobs=[{262: Logprob(-0.1), 257: Logprob(-1.2), 428: Logprob(-2.3)}],
|
||||
num_output_top_logprobs=2,
|
||||
top_logprobs=[{262: Logprob(-0.1), 257: Logprob(-1.2)}],
|
||||
)
|
||||
token_ids = {e.token_id for e in result.content[0].top_logprobs}
|
||||
assert token_ids == {262, 257}, f"got {token_ids}"
|
||||
@@ -32,7 +32,6 @@ def test_sampled_entry_carries_token_id_and_rank():
|
||||
top_logprobs=[
|
||||
{262: Logprob(-0.1, rank=1), 257: Logprob(-1.2, rank=2)},
|
||||
],
|
||||
num_output_top_logprobs=2,
|
||||
)
|
||||
entry = result.content[0]
|
||||
assert entry.token_id == 262
|
||||
@@ -50,7 +49,6 @@ def test_sampled_token_absent_from_topk_uses_sentinel():
|
||||
None,
|
||||
token_ids=[5],
|
||||
top_logprobs=[{7: Logprob(-0.9)}],
|
||||
num_output_top_logprobs=1,
|
||||
)
|
||||
entry = result.content[0]
|
||||
assert entry.token_id == 5
|
||||
@@ -60,14 +58,13 @@ def test_sampled_token_absent_from_topk_uses_sentinel():
|
||||
|
||||
|
||||
def test_logprobs_zero_emits_sampled_token():
|
||||
"""logprobs=0 must still emit 1 entry (the sampled token)."""
|
||||
"""logprobs=0: the engine returns only the sampled token, and it is kept."""
|
||||
result = ServingTokens._create_tokens_logprobs(
|
||||
None,
|
||||
token_ids=[7],
|
||||
top_logprobs=[{7: Logprob(-0.9), 8: Logprob(-1.1)}],
|
||||
num_output_top_logprobs=0,
|
||||
top_logprobs=[{7: Logprob(-0.9)}],
|
||||
)
|
||||
assert len(result.content[0].top_logprobs) == 1
|
||||
assert [t.token_id for t in result.content[0].top_logprobs] == [7]
|
||||
|
||||
|
||||
def test_logprobs_minus_one_emits_all_tokens():
|
||||
@@ -75,7 +72,6 @@ def test_logprobs_minus_one_emits_all_tokens():
|
||||
None,
|
||||
token_ids=[7],
|
||||
top_logprobs=[{7: Logprob(-0.9), 8: Logprob(-1.1)}],
|
||||
num_output_top_logprobs=-1,
|
||||
)
|
||||
assert len(result.content[0].top_logprobs) == 2
|
||||
|
||||
@@ -86,7 +82,6 @@ def test_text_logprobs_without_tokenizer_are_placeholders_without_bytes():
|
||||
None,
|
||||
token_ids=[7],
|
||||
top_logprobs=[{7: Logprob(-0.9), 8: Logprob(-1.1)}],
|
||||
num_output_top_logprobs=2,
|
||||
)
|
||||
entry = result.content[0]
|
||||
assert entry.token == "token_id:7"
|
||||
@@ -102,7 +97,6 @@ def test_resolved_logprobs_use_engine_decoded_tokens_and_bytes():
|
||||
top_logprobs=[
|
||||
{7: Logprob(-0.9, decoded_token="é"), 8: Logprob(-1.1, decoded_token="e")}
|
||||
],
|
||||
num_output_top_logprobs=2,
|
||||
tokenizer=tokenizer,
|
||||
)
|
||||
entry = result.content[0]
|
||||
@@ -121,7 +115,6 @@ def test_resolved_logprobs_decode_tokens_the_engine_did_not():
|
||||
None,
|
||||
token_ids=[7, 9],
|
||||
top_logprobs=[{7: Logprob(-0.9)}, None],
|
||||
num_output_top_logprobs=1,
|
||||
tokenizer=tokenizer,
|
||||
)
|
||||
assert [e.token for e in result.content] == ["<7>", "<9>"]
|
||||
@@ -130,9 +123,9 @@ def test_resolved_logprobs_decode_tokens_the_engine_did_not():
|
||||
|
||||
|
||||
def test_sampled_token_outside_topk_comes_first():
|
||||
"""The engine puts the sampled token first, then ranks 1..k. When the
|
||||
sampled token is outside the top k it takes one of the k slots, so rank k
|
||||
is left out (same as the OpenAI endpoints)."""
|
||||
"""The engine puts the sampled token first, then ranks 1..k. Generate keeps
|
||||
all k + 1 candidates when the sampled token is outside the top k; derender
|
||||
applies each endpoint's cut (#59513)."""
|
||||
result = ServingTokens._create_tokens_logprobs(
|
||||
None,
|
||||
token_ids=[50],
|
||||
@@ -143,9 +136,8 @@ def test_sampled_token_outside_topk_comes_first():
|
||||
20: Logprob(-1.0, rank=2),
|
||||
}
|
||||
],
|
||||
num_output_top_logprobs=2,
|
||||
)
|
||||
assert [t.rank for t in result.content[0].top_logprobs] == [5, 1]
|
||||
assert [t.rank for t in result.content[0].top_logprobs] == [5, 1, 2]
|
||||
|
||||
|
||||
def test_logprob_is_required_on_the_wire():
|
||||
@@ -160,7 +152,6 @@ def test_rank_zero_from_a_nan_logprob_is_none():
|
||||
None,
|
||||
token_ids=[7],
|
||||
top_logprobs=[{7: Logprob(float("nan"), rank=0), 8: Logprob(-0.2, rank=1)}],
|
||||
num_output_top_logprobs=1,
|
||||
)
|
||||
entry = result.content[0]
|
||||
assert entry.rank is None
|
||||
@@ -176,7 +167,6 @@ def test_text_logprobs_clamp_a_nan_logprob():
|
||||
None,
|
||||
token_ids=[7],
|
||||
top_logprobs=[{7: Logprob(float("nan"), rank=0, decoded_token="a")}],
|
||||
num_output_top_logprobs=1,
|
||||
tokenizer=MagicMock(),
|
||||
)
|
||||
entry = result.content[0]
|
||||
|
||||
@@ -356,6 +356,32 @@ def decode_token_ids(
|
||||
return out
|
||||
|
||||
|
||||
def chat_top_logprobs_limit(
|
||||
top_logprobs: int | None, return_all: bool = False
|
||||
) -> int | None:
|
||||
"""How many of a position's logprob candidates `/v1/chat/completions`
|
||||
returns, in the engine's order (sampled token first): the first
|
||||
``top_logprobs``, none when it is unset, and all of them (``None``) for
|
||||
``top_logprobs=-1`` or ``logprob_token_ids`` (``return_all``). Shared with
|
||||
chat derender so the two cannot drift."""
|
||||
if return_all or top_logprobs == -1:
|
||||
return None
|
||||
return top_logprobs or 0
|
||||
|
||||
|
||||
def completion_top_logprobs_limit(
|
||||
logprobs: int | None, return_all: bool = False
|
||||
) -> int | None:
|
||||
"""How many of a position's logprob candidates `/v1/completions` returns:
|
||||
the first ``logprobs + 1`` (the OpenAI completions convention), and all of
|
||||
them (``None``) for ``logprobs=-1``, ``logprob_token_ids`` (``return_all``)
|
||||
or when ``logprobs`` is unset. Shared with completions derender so the two
|
||||
cannot drift."""
|
||||
if return_all or logprobs is None or logprobs == -1:
|
||||
return None
|
||||
return max(logprobs + 1, 0)
|
||||
|
||||
|
||||
def clamp_prompt_logprobs(
|
||||
prompt_logprobs: PromptLogprobs | None,
|
||||
) -> PromptLogprobs | None:
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
import asyncio
|
||||
import itertools
|
||||
import time
|
||||
from collections.abc import AsyncGenerator, AsyncIterator
|
||||
from collections.abc import Sequence as GenericSequence
|
||||
@@ -22,6 +23,7 @@ from vllm.entrypoints.generate.base.serving import (
|
||||
GenerateBaseServing,
|
||||
build_per_request_timing_metrics,
|
||||
build_spec_decoding_metrics,
|
||||
chat_top_logprobs_limit,
|
||||
clamp_prompt_logprobs,
|
||||
format_token_id_placeholder,
|
||||
)
|
||||
@@ -1231,10 +1233,9 @@ class OpenAIServingChat(GenerateBaseServing):
|
||||
logprob=max(p[1].logprob, -9999.0),
|
||||
bytes=list(token.encode("utf-8", errors="replace")),
|
||||
)
|
||||
for i, p in enumerate(logprobs.items())
|
||||
if return_all
|
||||
or top_logprobs == -1
|
||||
or (top_logprobs is not None and i < top_logprobs)
|
||||
for p in itertools.islice(
|
||||
logprobs.items(), chat_top_logprobs_limit(top_logprobs, return_all)
|
||||
)
|
||||
]
|
||||
|
||||
def _create_chat_logprobs(
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
import asyncio
|
||||
import itertools
|
||||
import time
|
||||
from collections.abc import AsyncGenerator, AsyncIterator
|
||||
from collections.abc import Sequence as GenericSequence
|
||||
@@ -19,6 +20,7 @@ from vllm.entrypoints.generate.base.serving import (
|
||||
build_per_request_timing_metrics,
|
||||
build_spec_decoding_metrics,
|
||||
clamp_prompt_logprobs,
|
||||
completion_top_logprobs_limit,
|
||||
format_token_id_placeholder,
|
||||
)
|
||||
from vllm.entrypoints.openai.completion.protocol import (
|
||||
@@ -721,10 +723,12 @@ class OpenAIServingCompletion(GenerateBaseServing):
|
||||
tokenizer,
|
||||
return_as_token_id=should_return_as_token_id,
|
||||
): max(top_lp[1].logprob, -9999.0)
|
||||
for i, top_lp in enumerate(step_top_logprobs.items())
|
||||
if logprob_token_ids
|
||||
or num_output_top_logprobs == -1
|
||||
or num_output_top_logprobs >= i
|
||||
for top_lp in itertools.islice(
|
||||
step_top_logprobs.items(),
|
||||
completion_top_logprobs_limit(
|
||||
num_output_top_logprobs, bool(logprob_token_ids)
|
||||
),
|
||||
)
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@@ -424,14 +424,12 @@ class ServingTokens(GenerateBaseServing):
|
||||
logprobs = self._create_text_logprobs(
|
||||
token_ids=token_ids,
|
||||
top_logprobs=out_logprobs,
|
||||
num_output_top_logprobs=sampling_params.logprobs,
|
||||
tokenizer=tokenizer,
|
||||
)
|
||||
else:
|
||||
logprobs = self._create_tokens_logprobs(
|
||||
token_ids=token_ids,
|
||||
top_logprobs=out_logprobs,
|
||||
num_output_top_logprobs=sampling_params.logprobs,
|
||||
)
|
||||
else:
|
||||
logprobs = None
|
||||
@@ -599,14 +597,12 @@ class ServingTokens(GenerateBaseServing):
|
||||
logprobs = self._create_text_logprobs(
|
||||
token_ids=delta_token_ids,
|
||||
top_logprobs=out_logprobs,
|
||||
num_output_top_logprobs=sampling_params.logprobs,
|
||||
tokenizer=tokenizer,
|
||||
)
|
||||
else:
|
||||
logprobs = self._create_tokens_logprobs(
|
||||
token_ids=delta_token_ids,
|
||||
top_logprobs=out_logprobs,
|
||||
num_output_top_logprobs=sampling_params.logprobs,
|
||||
)
|
||||
else:
|
||||
logprobs = None
|
||||
@@ -728,14 +724,15 @@ class ServingTokens(GenerateBaseServing):
|
||||
self,
|
||||
token_ids: GenericSequence[int],
|
||||
top_logprobs: GenericSequence[dict[int, Logprob] | None],
|
||||
num_output_top_logprobs: int | None = None,
|
||||
tokenizer: TokenizerLike | None = None,
|
||||
) -> ChatCompletionLogProbs:
|
||||
"""Create OpenAI-style logprobs for ``output_mode="text"``.
|
||||
|
||||
With a ``tokenizer`` the tokens are decoded strings and carry ``bytes``;
|
||||
without one (``--return-tokens-as-token-ids``) they are ``token_id:N``
|
||||
placeholders.
|
||||
placeholders. Like tokens mode, ``top_logprobs`` carries every candidate
|
||||
the engine returned (the sampled token first, then the top k), not cut
|
||||
to the requested count.
|
||||
"""
|
||||
logprobs_content: list[ChatCompletionLogProbsContent] = []
|
||||
|
||||
@@ -757,14 +754,7 @@ class ServingTokens(GenerateBaseServing):
|
||||
bytes=token_bytes,
|
||||
top_logprobs=[
|
||||
_top_logprob(top_id, logprob, tokenizer)
|
||||
for i, (top_id, logprob) in enumerate(
|
||||
step_top_logprobs.items()
|
||||
)
|
||||
if num_output_top_logprobs is not None
|
||||
and (
|
||||
num_output_top_logprobs == -1
|
||||
or i < max(num_output_top_logprobs, 1)
|
||||
)
|
||||
for top_id, logprob in step_top_logprobs.items()
|
||||
],
|
||||
)
|
||||
)
|
||||
@@ -775,10 +765,16 @@ class ServingTokens(GenerateBaseServing):
|
||||
self,
|
||||
token_ids: GenericSequence[int],
|
||||
top_logprobs: GenericSequence[dict[int, Logprob] | None],
|
||||
num_output_top_logprobs: int | None = None,
|
||||
) -> GenerateLogProbs:
|
||||
"""Create generate-shaped logprobs (integer token ids, no tokenizer).
|
||||
|
||||
``top_logprobs`` carries every candidate the engine returned for the
|
||||
position: the sampled token first, then the top k, so k entries when the
|
||||
sampled token is in the top k and k + 1 when it is not. It is not cut to
|
||||
the requested count: `/v1/chat/completions` and
|
||||
`/v1/completions` cut differently, and derender applies each endpoint's
|
||||
cut the way the coupled server does.
|
||||
|
||||
The engine reports rank 0 for a sampled token whose logprob is NaN; that
|
||||
is not a rank, so it is sent as ``None`` (the logprob is clamped).
|
||||
"""
|
||||
@@ -806,14 +802,7 @@ class ServingTokens(GenerateBaseServing):
|
||||
logprob=_clamp_logprob(top_logprob.logprob),
|
||||
rank=top_logprob.rank or None,
|
||||
)
|
||||
for rank_index, (top_token_id, top_logprob) in enumerate(
|
||||
step_top_logprobs.items()
|
||||
)
|
||||
if num_output_top_logprobs is not None
|
||||
and (
|
||||
num_output_top_logprobs == -1
|
||||
or rank_index < max(num_output_top_logprobs, 1)
|
||||
)
|
||||
for top_token_id, top_logprob in step_top_logprobs.items()
|
||||
],
|
||||
)
|
||||
)
|
||||
|
||||
@@ -9,7 +9,11 @@ from vllm.entrypoints.generate.base.protocol import (
|
||||
DeltaMessage,
|
||||
ToolCall,
|
||||
)
|
||||
from vllm.entrypoints.generate.base.serving import decode_token_ids
|
||||
from vllm.entrypoints.generate.base.serving import (
|
||||
chat_top_logprobs_limit,
|
||||
completion_top_logprobs_limit,
|
||||
decode_token_ids,
|
||||
)
|
||||
from vllm.entrypoints.openai.chat_completion.protocol import (
|
||||
ChatCompletionLogProb,
|
||||
ChatCompletionLogProbs,
|
||||
@@ -204,12 +208,17 @@ class OnlineDerenderer:
|
||||
and chat_request.return_token_ids
|
||||
and (not has_parser or chat_request.include_reasoning)
|
||||
)
|
||||
top_limit = _chat_top_logprobs_limit(chat_request)
|
||||
for choice in generate_response.choices:
|
||||
if not choice.token_ids:
|
||||
raise ValueError(f"choice {choice.index} has empty or null token_ids")
|
||||
|
||||
resolved_logprobs = (
|
||||
_resolve_logprobs(choice.logprobs, tokenizer)
|
||||
_resolve_logprobs(
|
||||
choice.logprobs,
|
||||
tokenizer,
|
||||
top_limit=top_limit,
|
||||
)
|
||||
if choice.logprobs is not None
|
||||
else None
|
||||
)
|
||||
@@ -486,6 +495,7 @@ class OnlineDerenderer:
|
||||
updated_state = state
|
||||
|
||||
return_token_ids = chat_request is not None and chat_request.return_token_ids
|
||||
top_limit = _chat_top_logprobs_limit(chat_request)
|
||||
for choice in generate_chunk.choices:
|
||||
delta_tids = choice.token_ids or []
|
||||
new_text, updated_state = await self._detokenize_delta_async(
|
||||
@@ -511,6 +521,7 @@ class OnlineDerenderer:
|
||||
choice.logprobs,
|
||||
tokenizer,
|
||||
initial_context_token_ids=state.logprob_context_token_ids,
|
||||
top_limit=top_limit,
|
||||
)
|
||||
|
||||
include_role = not updated_state.role_sent
|
||||
@@ -794,6 +805,7 @@ class OnlineDerenderer:
|
||||
total_prompt_tokens = 0
|
||||
total_completion_tokens = 0
|
||||
index = 0
|
||||
top_limit = _completion_top_logprobs_limit(completion_request)
|
||||
|
||||
for gen, pt, seed_ids_override in zip(
|
||||
generate_responses, prompt_tokens_list, prompt_token_ids_list
|
||||
@@ -822,9 +834,13 @@ class OnlineDerenderer:
|
||||
)
|
||||
completion_logprobs = None
|
||||
if choice.logprobs is not None:
|
||||
resolved = _resolve_logprobs(choice.logprobs, tokenizer)
|
||||
resolved = _resolve_logprobs(
|
||||
choice.logprobs,
|
||||
tokenizer,
|
||||
top_limit=top_limit,
|
||||
)
|
||||
completion_logprobs = _convert_chat_logprobs_to_completion_logprobs(
|
||||
resolved
|
||||
resolved, choice.logprobs
|
||||
)
|
||||
choices.append(
|
||||
CompletionResponseChoice(
|
||||
@@ -897,6 +913,7 @@ class OnlineDerenderer:
|
||||
stream_choices: list[CompletionResponseStreamChoice] = []
|
||||
updated_state = state
|
||||
|
||||
top_limit = _completion_top_logprobs_limit(completion_request)
|
||||
for choice in generate_chunk.choices:
|
||||
delta_tids = choice.token_ids or []
|
||||
new_text, updated_state = await self._detokenize_delta_async(
|
||||
@@ -913,9 +930,12 @@ class OnlineDerenderer:
|
||||
choice.logprobs,
|
||||
tokenizer,
|
||||
initial_context_token_ids=state.logprob_context_token_ids,
|
||||
top_limit=top_limit,
|
||||
)
|
||||
completion_logprobs = _convert_chat_logprobs_to_completion_logprobs(
|
||||
resolved, initial_text_offset=state.logprob_text_offset
|
||||
resolved,
|
||||
choice.logprobs,
|
||||
initial_text_offset=state.logprob_text_offset,
|
||||
)
|
||||
|
||||
updated_state = updated_state.model_copy(
|
||||
@@ -1067,10 +1087,38 @@ def _correct_decoded_token(
|
||||
return ""
|
||||
|
||||
|
||||
def _chat_top_logprobs_limit(chat_request: ChatCompletionRequest | None) -> int | None:
|
||||
"""The cut `/v1/chat/completions` applies (`chat_top_logprobs_limit`, the
|
||||
same helper its `_get_top_logprobs` uses). None keeps every candidate, which
|
||||
is also what happens without a `chat_request` or with one that does not ask
|
||||
for logprobs: it then says nothing about the cut, so the candidates
|
||||
generate returned are passed through."""
|
||||
if chat_request is None or not chat_request.logprobs:
|
||||
return None
|
||||
return chat_top_logprobs_limit(
|
||||
chat_request.top_logprobs, bool(chat_request.logprob_token_ids)
|
||||
)
|
||||
|
||||
|
||||
def _completion_top_logprobs_limit(
|
||||
completion_request: CompletionRequest | None,
|
||||
) -> int | None:
|
||||
"""The cut `/v1/completions` applies (`completion_top_logprobs_limit`, the
|
||||
same helper `_create_completion_logprobs` uses). None keeps every candidate,
|
||||
which is also what happens without a `completion_request` or with one that
|
||||
does not ask for logprobs (`logprobs` unset)."""
|
||||
if completion_request is None:
|
||||
return None
|
||||
return completion_top_logprobs_limit(
|
||||
completion_request.logprobs, bool(completion_request.logprob_token_ids)
|
||||
)
|
||||
|
||||
|
||||
def _resolve_logprobs(
|
||||
logprobs: GenerateLogProbs,
|
||||
tokenizer: TokenizerLike,
|
||||
initial_context_token_ids: Sequence[int] = (),
|
||||
top_limit: int | None = None,
|
||||
) -> ChatCompletionLogProbs:
|
||||
"""Convert generate's integer-id logprobs to the OpenAI chat shape.
|
||||
|
||||
@@ -1080,6 +1128,11 @@ def _resolve_logprobs(
|
||||
ids as context). ``initial_context_token_ids`` seeds that context with
|
||||
sampled IDs from preceding chunks (streaming), so multi-byte characters
|
||||
split across chunk boundaries still resolve.
|
||||
|
||||
Generate returns every candidate the engine produced for a position (the
|
||||
sampled token first, then the top k). ``top_limit`` applies the target
|
||||
endpoint's cut before decoding, so each derender endpoint returns what its
|
||||
coupled counterpart would; None keeps all of them.
|
||||
"""
|
||||
if logprobs.content is None:
|
||||
return ChatCompletionLogProbs()
|
||||
@@ -1088,9 +1141,12 @@ def _resolve_logprobs(
|
||||
resolved_content = []
|
||||
|
||||
for entry in logprobs.content:
|
||||
# One batch per position: the sampled id and its top-k ids.
|
||||
top_entries = (
|
||||
entry.top_logprobs if top_limit is None else entry.top_logprobs[:top_limit]
|
||||
)
|
||||
# One batch per position: the sampled id and its kept top-k ids.
|
||||
(token_str, token_bytes), *top_decoded = decode_token_ids(
|
||||
[entry.token_id, *(top.token_id for top in entry.top_logprobs)],
|
||||
[entry.token_id, *(top.token_id for top in top_entries)],
|
||||
tokenizer,
|
||||
)
|
||||
|
||||
@@ -1101,7 +1157,7 @@ def _resolve_logprobs(
|
||||
token_bytes = list(token_str.encode("utf-8"))
|
||||
|
||||
resolved_top = []
|
||||
for top, (top_str, top_bytes) in zip(entry.top_logprobs, top_decoded):
|
||||
for top, (top_str, top_bytes) in zip(top_entries, top_decoded):
|
||||
if top_str.endswith("\ufffd"):
|
||||
top_str = _correct_decoded_token(
|
||||
top.token_id, context_token_ids, tokenizer
|
||||
@@ -1131,11 +1187,17 @@ def _resolve_logprobs(
|
||||
|
||||
def _convert_chat_logprobs_to_completion_logprobs(
|
||||
logprobs: ChatCompletionLogProbs,
|
||||
generate_logprobs: GenerateLogProbs,
|
||||
initial_text_offset: int = 0,
|
||||
) -> CompletionLogProbs:
|
||||
"""Convert ChatCompletionLogProbs (per-token objects) to CompletionLogProbs
|
||||
(parallel flat lists) as required by the /v1/completions response schema.
|
||||
|
||||
``generate_logprobs`` is what ``logprobs`` was resolved from. A position
|
||||
with no candidates there had no logprobs in the engine output, which
|
||||
/v1/completions reports as ``None``; every other position is a dict of
|
||||
the candidates the cut kept.
|
||||
|
||||
``initial_text_offset`` keeps ``text_offset`` absolute across streaming
|
||||
chunks, mirroring the generate streaming path.
|
||||
"""
|
||||
@@ -1148,13 +1210,13 @@ def _convert_chat_logprobs_to_completion_logprobs(
|
||||
text_offset: list[int] = []
|
||||
|
||||
offset = initial_text_offset
|
||||
for entry in logprobs.content:
|
||||
for entry, source in zip(logprobs.content, generate_logprobs.content or ()):
|
||||
text_offset.append(offset)
|
||||
tokens.append(entry.token)
|
||||
token_logprobs.append(entry.logprob)
|
||||
top_logprobs_list.append(
|
||||
{t.token: t.logprob for t in entry.top_logprobs}
|
||||
if entry.top_logprobs
|
||||
if source.top_logprobs
|
||||
else None
|
||||
)
|
||||
offset += len(entry.token)
|
||||
|
||||
Reference in New Issue
Block a user