diff --git a/docs/serving/online_serving/derenderer.md b/docs/serving/online_serving/derenderer.md index f85433a963..b7b5e5b8f0 100644 --- a/docs/serving/online_serving/derenderer.md +++ b/docs/serving/online_serving/derenderer.md @@ -89,7 +89,7 @@ The derenderer builds its tool and reasoning parsers from its own server flags p A mismatch doesn't fail. The parser just splits `reasoning`, `content` and `tool_calls` differently from what `vllm serve` would return. -`/v1/completions/derender` only detokenizes. It never runs tool or reasoning parsers, the same as `/v1/completions` on `vllm serve`. It only reads `skip_special_tokens` from `completion_request`. +`/v1/completions/derender` only detokenizes. It never runs tool or reasoning parsers, the same as `/v1/completions` on `vllm serve`. From `completion_request` it reads `skip_special_tokens`, and `logprobs` and `logprob_token_ids` to cut `top_logprobs` the way `/v1/completions` does. ## Streaming @@ -105,7 +105,7 @@ The server keeps no state between calls. Everything the next call needs is in `s - **Send `prompt_token_ids` on the first chunk, even without a parser.** It seeds detokenization so the first token's leading space matches the coupled endpoint (see [Request format](#request-format)). Later chunks already carry that context in `stream_state`. - **Don't forward `[DONE]`.** It marks the end of the generate stream and isn't a chunk. -`/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Streaming derender does the same per chunk on the plain detokenization path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them. +`/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Generate returns every top-k candidate the engine produced; derender keeps the ones the coupled endpoint would, with the same helpers: the first `top_logprobs` on chat (all of them for `top_logprobs=-1`) and the first `logprobs + 1` on completions (all of them for `logprobs=-1`). It reads those values from `chat_request` / `completion_request`, so without the original request, or with one that does not ask for logprobs, it returns every candidate. Requests that use `logprob_token_ids` get no output logprobs from generate (their `sampling_params.logprobs` is unset), so derender has none to return for them either. Streaming derender does the same per chunk on the plain detokenization path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them. ## Streaming cost diff --git a/docs/serving/online_serving/renderer.md b/docs/serving/online_serving/renderer.md index e44a9de319..adf9c4e7c1 100644 --- a/docs/serving/online_serving/renderer.md +++ b/docs/serving/online_serving/renderer.md @@ -108,10 +108,13 @@ choice carries a `GenerateLogProbs`; `output_mode: "text"` returns decoded - `content` has one entry per generated token, in generation order. - `top_logprobs` is a list, not a dict: JSON turns dict keys into strings and the ordering would be implicit. It follows the engine's order: the sampled - token first, then the remaining candidates in rank order. With non-greedy - sampling the sampled token can sit outside the top k (for example ranks - `[5, 1, 2]` at `logprobs=2`); it then takes one of the `logprobs` slots and - the rank-k candidate is left out, as on the OpenAI endpoints. + token first, then the remaining candidates in rank order. It holds every + candidate the engine returned and is not cut to `logprobs`: k entries when + the sampled token is in the top k, k + 1 when non-greedy sampling picked a + token outside it (for example ranks `[5, 1, 2]` at `logprobs=2`). The OpenAI + endpoints cut this differently (`/v1/chat/completions` keeps the first k, + `/v1/completions` the first k + 1), so [derender](derenderer.md) applies the + cut of the endpoint it renders for. - `rank` is the token's rank in the vocabulary distribution (1 = most likely) on every entry, the sampled one included; a top-k candidate's rank is its top-k position. The list is not sorted by it, so sort by `rank` if you need @@ -132,6 +135,16 @@ choice carries a `GenerateLogProbs`; `output_mode: "text"` returns decoded `return_tokens_as_token_ids` on `/v1/chat/completions` and `/v1/completions` is unchanged: it is a user-facing OpenAI option and still uses the `token_id:N` format. + `top_logprobs` is no longer cut to `max(logprobs, 1)` entries, in either + `output_mode`: it now holds every candidate the engine returned, which is + `logprobs + 1` entries when the sampled token is outside the top + `logprobs`. Clients that assume `len(top_logprobs) == logprobs` should + take the first `logprobs` entries after sorting by `rank`. + Derender applies each endpoint's cut from the original request, so send + `chat_request` / `completion_request` to derender whenever logprobs are + requested: without it derender returns every candidate, which on chat is + `top_logprobs + 1` entries where `/v1/chat/completions` returns + `top_logprobs`. ## Multimodal Render Features diff --git a/rust/src/server/src/routes/inference/generate.rs b/rust/src/server/src/routes/inference/generate.rs index 42716de8de..5bf434f522 100644 --- a/rust/src/server/src/routes/inference/generate.rs +++ b/rust/src/server/src/routes/inference/generate.rs @@ -5,7 +5,7 @@ mod convert; mod types; mod validate; -use std::collections::{HashMap, HashSet}; +use std::collections::HashMap; use std::convert::Infallible; use std::result::Result; use std::sync::Arc; @@ -158,7 +158,7 @@ async fn generate_chunk_stream( ResponseOptions { include_usage, include_continuous_usage, - logprobs: output_logprobs, + include_logprobs, // Ignored: raw generate streaming has no prompt-logprobs wire shape. include_prompt_logprobs: _, return_token_ids, @@ -213,15 +213,13 @@ async fn generate_chunk_stream( continue; } - let logprobs = if let Some(requested) = output_logprobs - && !token_ids.is_empty() - { + let logprobs = if include_logprobs && !token_ids.is_empty() { let logprobs = output.logprobs.as_ref().ok_or_else(|| { server_error!( "raw generate stream requested logprobs but generation returned none" ) })?; - Some(raw_logprobs_to_generate(logprobs, requested)?) + Some(raw_logprobs_to_generate(logprobs)?) } else { None }; @@ -289,19 +287,19 @@ fn collect_generate( include_usage: _, // Ignored: continuous usage is a streaming-only option. include_continuous_usage: _, - logprobs: output_logprobs, + include_logprobs, include_prompt_logprobs, return_token_ids, }: ResponseOptions, mm_placeholders: Option, ) -> Result { - let logprobs = if let Some(requested) = output_logprobs { + let logprobs = if include_logprobs { let logprobs = collected.logprobs.as_ref().ok_or_else(|| { ApiError::server_error( "raw generate response requested logprobs but generation returned none".to_string(), ) })?; - Some(raw_logprobs_to_generate(logprobs, requested)?) + Some(raw_logprobs_to_generate(logprobs)?) } else { None }; @@ -381,14 +379,11 @@ fn extract_mm_placeholders(features: Option<&[MmFeatureSpec]>) -> Option Result { +fn raw_logprobs_to_generate(logprobs: &Logprobs) -> Result { let content = logprobs .positions .iter() - .map(|position| position_to_generate_logprobs_content(position, requested)) + .map(position_to_generate_logprobs_content) .collect::, _>>()?; Ok(GenerateLogProbs { @@ -409,12 +404,12 @@ fn raw_prompt_logprobs_to_maps( .collect() } -/// `top_logprobs` matches the Python generate server: the engine's entries (the -/// sampled token first, then ranks 1..k) with repeated token ids dropped, cut to -/// `max(requested, 1)` entries, or all of them for `-1`. +/// `top_logprobs` matches the Python generate server: every candidate the +/// engine returned (the sampled token first, then ranks 1..k) with the repeated +/// sampled entry dropped, and no cut. Derender applies each OpenAI endpoint's +/// cut. fn position_to_generate_logprobs_content( position: &PositionLogprobs, - requested: i32, ) -> Result { let chosen = position.entries.first().ok_or_else(|| { ApiError::server_error( @@ -422,9 +417,9 @@ fn position_to_generate_logprobs_content( ) })?; - // One pass: the first entry of each token id wins, as in Python's dict. The - // sampled token is the only one the engine can list twice. - let mut seen = HashSet::with_capacity(position.entries.len()); + // The engine row is the sampled token followed by the top k, which are + // distinct; only the sampled token can repeat, at its own rank (anywhere in + // 1..=k). Skip that repeat, as Python's dict does: linear, no allocation. Ok(GenerateLogProbsContent { token_id: chosen.token_id, logprob: clamp_logprob(chosen.logprob), @@ -432,9 +427,9 @@ fn position_to_generate_logprobs_content( top_logprobs: position .entries .iter() - .filter(|entry| seen.insert(entry.token_id)) - .take(usize::try_from(requested).map_or(usize::MAX, |n| n.max(1))) - .map(|entry| GenerateLogProb { + .enumerate() + .filter(|(i, entry)| *i == 0 || entry.token_id != chosen.token_id) + .map(|(_, entry)| GenerateLogProb { token_id: entry.token_id, logprob: clamp_logprob(entry.logprob), rank: wire_rank(entry.rank), @@ -563,8 +558,8 @@ mod tests { } } - fn top(position: &PositionLogprobs, requested: i32) -> Vec<(u32, Option)> { - position_to_generate_logprobs_content(position, requested) + fn top(position: &PositionLogprobs) -> Vec<(u32, Option)> { + position_to_generate_logprobs_content(position) .unwrap() .top_logprobs .iter() @@ -576,24 +571,31 @@ mod tests { fn generate_top_logprobs_drop_the_repeated_sampled_token() { // Sampled token 7 is also top-1: the engine row is [7, 7, 8]. let pos = position(&[(7, -0.1, 1), (7, -0.1, 1), (8, -2.0, 2)]); - assert_eq!(top(&pos, 2), vec![(7, Some(1)), (8, Some(2))]); + assert_eq!(top(&pos), vec![(7, Some(1)), (8, Some(2))]); } #[test] - fn generate_top_logprobs_keep_sampled_first_when_outside_top_k() { - // Sampled token 50 has vocab rank 5; it takes a slot and rank 2 drops - // out, as on the Python generate server and the OpenAI endpoints. + fn generate_top_logprobs_keep_every_candidate_when_sampled_is_outside_top_k() { + // Sampled token 50 has vocab rank 5 at logprobs=2: sampled first, then + // ranks 1 and 2, all kept (derender cuts per endpoint). let pos = position(&[(50, -3.0, 5), (10, -0.2, 1), (20, -1.0, 2)]); - assert_eq!(top(&pos, 2), vec![(50, Some(5)), (10, Some(1))]); + assert_eq!(top(&pos), vec![(50, Some(5)), (10, Some(1)), (20, Some(2))]); } #[test] - fn generate_top_logprobs_respect_zero_and_all() { - let pos = position(&[(7, -0.1, 1), (7, -0.1, 1), (8, -2.0, 2), (9, -3.0, 3)]); - assert_eq!(top(&pos, 0), vec![(7, Some(1))]); + fn generate_top_logprobs_drop_the_sampled_repeat_at_its_own_rank() { + // Sampled token 30 is rank 3: the engine row is [30, 10, 20, 30, 40], + // so the repeat is at entries[3], not entries[1]. + let pos = position(&[ + (30, -1.0, 3), + (10, -0.2, 1), + (20, -0.5, 2), + (30, -1.0, 3), + (40, -2.0, 4), + ]); assert_eq!( - top(&pos, -1), - vec![(7, Some(1)), (8, Some(2)), (9, Some(3))] + top(&pos), + vec![(30, Some(3)), (10, Some(1)), (20, Some(2)), (40, Some(4))] ); } @@ -606,7 +608,7 @@ mod tests { entries.extend((0..VOCAB).map(|t| (t, -1.0 - t as f32, t + 1))); let pos = position(&entries); let start = std::time::Instant::now(); - let got = top(&pos, -1); + let got = top(&pos); assert!(start.elapsed() < std::time::Duration::from_secs(5)); assert_eq!(got.len(), VOCAB as usize); assert_eq!(got[0], (5, Some(1))); @@ -616,10 +618,10 @@ mod tests { fn generate_rank_zero_from_a_nan_logprob_is_none() { // The engine reports rank 0 for a sampled token whose logprob is NaN. let pos = position(&[(7, f32::NAN, 0), (8, -0.2, 1)]); - let content = position_to_generate_logprobs_content(&pos, 1).unwrap(); + let content = position_to_generate_logprobs_content(&pos).unwrap(); assert_eq!(content.rank, None); assert_eq!(content.logprob, -9999.0); - assert_eq!(top(&pos, 1), vec![(7, None)]); + assert_eq!(top(&pos), vec![(7, None), (8, Some(1))]); } #[tokio::test] diff --git a/rust/src/server/src/routes/inference/generate/convert.rs b/rust/src/server/src/routes/inference/generate/convert.rs index 313e9fa1f9..cd65682f6c 100644 --- a/rust/src/server/src/routes/inference/generate/convert.rs +++ b/rust/src/server/src/routes/inference/generate/convert.rs @@ -26,10 +26,8 @@ pub(super) struct ResponseOptions { pub include_usage: bool, /// Whether the caller asked for usage on every streamed chunk. pub include_continuous_usage: bool, - /// Requested output logprobs (`sampling_params.logprobs`): `None` omits - /// them, `Some(n)` returns `max(n, 1)` candidates per position, `Some(-1)` - /// all of them. - pub logprobs: Option, + /// Whether the caller requested output logprobs on generate choices. + pub include_logprobs: bool, /// Whether the caller requested top-level prompt logprobs. pub include_prompt_logprobs: bool, /// Whether the caller requested final prompt token metadata. @@ -58,7 +56,7 @@ pub(super) fn prepare_generate_request( .as_ref() .and_then(|options| options.continuous_usage_stats) .unwrap_or(false); - let logprobs = request.sampling_params.inner.logprobs; + let include_logprobs = request.sampling_params.inner.logprobs.is_some(); let include_prompt_logprobs = request.sampling_params.inner.prompt_logprobs.is_some(); let return_token_ids = request.return_token_ids.unwrap_or(false); let mut sampling_params = request.sampling_params.inner; @@ -98,7 +96,7 @@ pub(super) fn prepare_generate_request( options: ResponseOptions { include_usage, include_continuous_usage, - logprobs, + include_logprobs, include_prompt_logprobs, return_token_ids, }, diff --git a/tests/entrypoints/scale_out/derender/test_generate_logprobs_conversion.py b/tests/entrypoints/scale_out/derender/test_generate_logprobs_conversion.py index 96d937ad5a..221b2c3994 100644 --- a/tests/entrypoints/scale_out/derender/test_generate_logprobs_conversion.py +++ b/tests/entrypoints/scale_out/derender/test_generate_logprobs_conversion.py @@ -139,7 +139,7 @@ def test_sentencepiece_leading_space_matches_engine(): assert [t.token for t in entry.top_logprobs] == convert_ids_list_to_tokens( tokenizer, [spaced, bare] ) - completion = _convert_chat_logprobs_to_completion_logprobs(resolved) + completion = _convert_chat_logprobs_to_completion_logprobs(resolved, logprobs) assert completion.top_logprobs == [{" true": -0.5, "true": -1.5}] diff --git a/tests/entrypoints/scale_out/derender/test_logprobs_endpoint_parity.py b/tests/entrypoints/scale_out/derender/test_logprobs_endpoint_parity.py new file mode 100644 index 0000000000..fd3910851d --- /dev/null +++ b/tests/entrypoints/scale_out/derender/test_logprobs_endpoint_parity.py @@ -0,0 +1,212 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +"""generate + derender must return the same `top_logprobs` as the coupled +`/v1/chat/completions` and `/v1/completions` endpoints (#59513). + +Both sides start from the same per-position engine dicts, built with the +engine's own `append_logprobs_for_next_position` and detokenized the way the +engine does, and go through the real coupled serving objects on one side and +`/inference/v1/generate` + derender on the other. The cases include a sampled +token outside the top k, where chat keeps k entries and completions k + 1; the +greedy end-to-end parity tests never produce that case. +""" + +import pytest + +from tests.entrypoints.openai.chat_completion.test_serving_chat import ( + _build_mock_engine, + _build_serving_chat, +) +from tests.entrypoints.openai.completion.test_completion_error import ( + _build_serving_completion, +) +from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest +from vllm.entrypoints.openai.completion.protocol import CompletionRequest +from vllm.entrypoints.scale_out.token_in_token_out.protocol import GenerateLogProbs +from vllm.entrypoints.scale_out.token_in_token_out.serving import ServingTokens +from vllm.logprobs import append_logprobs_for_next_position +from vllm.renderers.online_derenderer import ( + _chat_top_logprobs_limit, + _completion_top_logprobs_limit, + _convert_chat_logprobs_to_completion_logprobs, + _resolve_logprobs, +) +from vllm.tokenizers import get_tokenizer +from vllm.tokenizers.detokenizer_utils import convert_ids_list_to_tokens + +MODEL_NAME = "hmellor/tiny-random-LlamaForCausalLM" + + +@pytest.fixture(scope="module") +def tokenizer(): + return get_tokenizer(MODEL_NAME) + + +@pytest.fixture(scope="module") +def serving_chat(): + return _build_serving_chat(_build_mock_engine()) + + +@pytest.fixture(scope="module") +def serving_completion(): + return _build_serving_completion(_build_mock_engine()) + + +@pytest.fixture(scope="module") +def vocab_ids(tokenizer): + """Ordinary word pieces (no byte-fallback tokens, which need the U+FFFD + context repair that is compared elsewhere).""" + ids = tokenizer.encode( + "the quick brown fox jumps over a lazy dog while seven cats watch " + "from an old wooden fence near the river bank", + add_special_tokens=False, + ) + pieces = tokenizer.convert_ids_to_tokens(ids) + return list(dict.fromkeys(i for i, p in zip(ids, pieces) if "<0x" not in p)) + + +def _positions(tokenizer, vocab_ids, k, sampled_ranks): + """Engine output for one completion: per position the sampled id and its + dict (sampled first, then ranks 1..k), with decoded tokens as the engine + fills them. k = -1 returns every candidate, as for the whole vocabulary.""" + token_ids, dicts = [], [] + for pos, sampled_rank in enumerate(sampled_ranks): + n_top = max(k, sampled_rank) if k != -1 else len(vocab_ids) - 1 + ranked = [vocab_ids[(pos + j) % len(vocab_ids)] for j in range(n_top + 1)] + sampled = ranked[sampled_rank - 1] + top = ranked[: len(ranked) - 1] if k == -1 else ranked[:k] + ids = [sampled, *top] + logprobs = [-0.25 * sampled_rank] + [-0.25 * (r + 1) for r in range(len(top))] + per_position: list = [] + append_logprobs_for_next_position( + per_position, + ids, + logprobs, + convert_ids_list_to_tokens(tokenizer, ids), + sampled_rank, + len(top), + ) + token_ids.append(sampled) + dicts.append(per_position[0]) + return token_ids, dicts + + +def _generate(token_ids, dicts) -> GenerateLogProbs: + out = ServingTokens._create_tokens_logprobs( + None, token_ids=token_ids, top_logprobs=dicts + ) + # Over the wire, as a derender client would send it back. + return GenerateLogProbs.model_validate_json(out.model_dump_json()) + + +# Sampled token rank per position: top-1, inside the top k, outside it. +RANKS = [1, 2, 1, 7, 3, 9, 1] + + +@pytest.mark.parametrize("k", [0, 1, 2, 5, -1]) +def test_chat_derender_matches_coupled_chat(tokenizer, vocab_ids, serving_chat, k): + token_ids, dicts = _positions(tokenizer, vocab_ids, k, RANKS) + coupled = serving_chat._create_chat_logprobs( + token_ids=token_ids, + top_logprobs=dicts, + tokenizer=tokenizer, + num_output_top_logprobs=k, + ) + request = ChatCompletionRequest( + model=MODEL_NAME, + messages=[{"role": "user", "content": "hi"}], + logprobs=True, + top_logprobs=k, + ) + derendered = _resolve_logprobs( + _generate(token_ids, dicts), + tokenizer, + top_limit=_chat_top_logprobs_limit(request), + ) + assert derendered.model_dump() == coupled.model_dump() + + +@pytest.mark.parametrize("k", [0, 1, 2, 5, -1]) +def test_completion_derender_matches_coupled_completion( + tokenizer, vocab_ids, serving_completion, k +): + token_ids, dicts = _positions(tokenizer, vocab_ids, k, RANKS) + coupled = serving_completion._create_completion_logprobs( + token_ids=token_ids, + top_logprobs=dicts, + num_output_top_logprobs=k, + tokenizer=tokenizer, + ) + request = CompletionRequest(model=MODEL_NAME, prompt="hi", logprobs=k) + generated = _generate(token_ids, dicts) + derendered = _convert_chat_logprobs_to_completion_logprobs( + _resolve_logprobs( + generated, + tokenizer, + top_limit=_completion_top_logprobs_limit(request), + ), + generated, + ) + assert derendered.model_dump() == coupled.model_dump() + if k > 0: + # The case #59513 is about: completions keeps k + 1 entries when the + # sampled token is outside the top k. + outside = [i for i, r in enumerate(RANKS) if r > k] + assert all(len(derendered.top_logprobs[i]) == k + 1 for i in outside) + if k == -1: + # /v1/completions returns every candidate for logprobs=-1 (#55717). + assert [len(top) for top in derendered.top_logprobs] == [len(d) for d in dicts] + + +@pytest.mark.parametrize( + "top_limit", + [ + _chat_top_logprobs_limit(None), + # A request that does not ask for logprobs says nothing about the cut. + _chat_top_logprobs_limit( + ChatCompletionRequest( + model=MODEL_NAME, messages=[{"role": "user", "content": "hi"}] + ) + ), + _completion_top_logprobs_limit(None), + _completion_top_logprobs_limit( + CompletionRequest(model=MODEL_NAME, prompt="hi") + ), + ], +) +def test_without_a_logprobs_request_derender_keeps_every_candidate( + tokenizer, vocab_ids, top_limit +): + token_ids, dicts = _positions(tokenizer, vocab_ids, 2, RANKS) + derendered = _resolve_logprobs( + _generate(token_ids, dicts), tokenizer, top_limit=top_limit + ) + assert [len(e.top_logprobs) for e in derendered.content] == [len(d) for d in dicts] + + +@pytest.mark.parametrize("k", [-1, 0, 2]) +def test_completion_position_without_logprobs_stays_none( + tokenizer, vocab_ids, serving_completion, k +): + """A position the engine returned no logprobs for is `None` in + `/v1/completions`, not `{}`; derender keeps it that way.""" + token_ids, dicts = _positions(tokenizer, vocab_ids, k, RANKS) + dicts[3] = None + coupled = serving_completion._create_completion_logprobs( + token_ids=token_ids, + top_logprobs=dicts, + num_output_top_logprobs=k, + tokenizer=tokenizer, + ) + request = CompletionRequest(model=MODEL_NAME, prompt="hi", logprobs=k) + generated = _generate(token_ids, dicts) + derendered = _convert_chat_logprobs_to_completion_logprobs( + _resolve_logprobs( + generated, + tokenizer, + top_limit=_completion_top_logprobs_limit(request), + ), + generated, + ) + assert derendered.top_logprobs == coupled.top_logprobs + assert derendered.top_logprobs[3] is None diff --git a/tests/entrypoints/scale_out/token_in_token_out/test_tokens_logprobs.py b/tests/entrypoints/scale_out/token_in_token_out/test_tokens_logprobs.py index 05dba167fe..ebbf6ac33c 100644 --- a/tests/entrypoints/scale_out/token_in_token_out/test_tokens_logprobs.py +++ b/tests/entrypoints/scale_out/token_in_token_out/test_tokens_logprobs.py @@ -14,11 +14,11 @@ from vllm.logprobs import Logprob def test_top_logprobs_alternatives_have_own_token_ids(): """Each top_logprobs alternative must carry its own integer token id.""" + # Engine dict at logprobs=2 with the sampled token 262 also top-1. result = ServingTokens._create_tokens_logprobs( None, token_ids=[262], - top_logprobs=[{262: Logprob(-0.1), 257: Logprob(-1.2), 428: Logprob(-2.3)}], - num_output_top_logprobs=2, + top_logprobs=[{262: Logprob(-0.1), 257: Logprob(-1.2)}], ) token_ids = {e.token_id for e in result.content[0].top_logprobs} assert token_ids == {262, 257}, f"got {token_ids}" @@ -32,7 +32,6 @@ def test_sampled_entry_carries_token_id_and_rank(): top_logprobs=[ {262: Logprob(-0.1, rank=1), 257: Logprob(-1.2, rank=2)}, ], - num_output_top_logprobs=2, ) entry = result.content[0] assert entry.token_id == 262 @@ -50,7 +49,6 @@ def test_sampled_token_absent_from_topk_uses_sentinel(): None, token_ids=[5], top_logprobs=[{7: Logprob(-0.9)}], - num_output_top_logprobs=1, ) entry = result.content[0] assert entry.token_id == 5 @@ -60,14 +58,13 @@ def test_sampled_token_absent_from_topk_uses_sentinel(): def test_logprobs_zero_emits_sampled_token(): - """logprobs=0 must still emit 1 entry (the sampled token).""" + """logprobs=0: the engine returns only the sampled token, and it is kept.""" result = ServingTokens._create_tokens_logprobs( None, token_ids=[7], - top_logprobs=[{7: Logprob(-0.9), 8: Logprob(-1.1)}], - num_output_top_logprobs=0, + top_logprobs=[{7: Logprob(-0.9)}], ) - assert len(result.content[0].top_logprobs) == 1 + assert [t.token_id for t in result.content[0].top_logprobs] == [7] def test_logprobs_minus_one_emits_all_tokens(): @@ -75,7 +72,6 @@ def test_logprobs_minus_one_emits_all_tokens(): None, token_ids=[7], top_logprobs=[{7: Logprob(-0.9), 8: Logprob(-1.1)}], - num_output_top_logprobs=-1, ) assert len(result.content[0].top_logprobs) == 2 @@ -86,7 +82,6 @@ def test_text_logprobs_without_tokenizer_are_placeholders_without_bytes(): None, token_ids=[7], top_logprobs=[{7: Logprob(-0.9), 8: Logprob(-1.1)}], - num_output_top_logprobs=2, ) entry = result.content[0] assert entry.token == "token_id:7" @@ -102,7 +97,6 @@ def test_resolved_logprobs_use_engine_decoded_tokens_and_bytes(): top_logprobs=[ {7: Logprob(-0.9, decoded_token="é"), 8: Logprob(-1.1, decoded_token="e")} ], - num_output_top_logprobs=2, tokenizer=tokenizer, ) entry = result.content[0] @@ -121,7 +115,6 @@ def test_resolved_logprobs_decode_tokens_the_engine_did_not(): None, token_ids=[7, 9], top_logprobs=[{7: Logprob(-0.9)}, None], - num_output_top_logprobs=1, tokenizer=tokenizer, ) assert [e.token for e in result.content] == ["<7>", "<9>"] @@ -130,9 +123,9 @@ def test_resolved_logprobs_decode_tokens_the_engine_did_not(): def test_sampled_token_outside_topk_comes_first(): - """The engine puts the sampled token first, then ranks 1..k. When the - sampled token is outside the top k it takes one of the k slots, so rank k - is left out (same as the OpenAI endpoints).""" + """The engine puts the sampled token first, then ranks 1..k. Generate keeps + all k + 1 candidates when the sampled token is outside the top k; derender + applies each endpoint's cut (#59513).""" result = ServingTokens._create_tokens_logprobs( None, token_ids=[50], @@ -143,9 +136,8 @@ def test_sampled_token_outside_topk_comes_first(): 20: Logprob(-1.0, rank=2), } ], - num_output_top_logprobs=2, ) - assert [t.rank for t in result.content[0].top_logprobs] == [5, 1] + assert [t.rank for t in result.content[0].top_logprobs] == [5, 1, 2] def test_logprob_is_required_on_the_wire(): @@ -160,7 +152,6 @@ def test_rank_zero_from_a_nan_logprob_is_none(): None, token_ids=[7], top_logprobs=[{7: Logprob(float("nan"), rank=0), 8: Logprob(-0.2, rank=1)}], - num_output_top_logprobs=1, ) entry = result.content[0] assert entry.rank is None @@ -176,7 +167,6 @@ def test_text_logprobs_clamp_a_nan_logprob(): None, token_ids=[7], top_logprobs=[{7: Logprob(float("nan"), rank=0, decoded_token="a")}], - num_output_top_logprobs=1, tokenizer=MagicMock(), ) entry = result.content[0] diff --git a/vllm/entrypoints/generate/base/serving.py b/vllm/entrypoints/generate/base/serving.py index a0eda6d05f..632f397711 100644 --- a/vllm/entrypoints/generate/base/serving.py +++ b/vllm/entrypoints/generate/base/serving.py @@ -356,6 +356,32 @@ def decode_token_ids( return out +def chat_top_logprobs_limit( + top_logprobs: int | None, return_all: bool = False +) -> int | None: + """How many of a position's logprob candidates `/v1/chat/completions` + returns, in the engine's order (sampled token first): the first + ``top_logprobs``, none when it is unset, and all of them (``None``) for + ``top_logprobs=-1`` or ``logprob_token_ids`` (``return_all``). Shared with + chat derender so the two cannot drift.""" + if return_all or top_logprobs == -1: + return None + return top_logprobs or 0 + + +def completion_top_logprobs_limit( + logprobs: int | None, return_all: bool = False +) -> int | None: + """How many of a position's logprob candidates `/v1/completions` returns: + the first ``logprobs + 1`` (the OpenAI completions convention), and all of + them (``None``) for ``logprobs=-1``, ``logprob_token_ids`` (``return_all``) + or when ``logprobs`` is unset. Shared with completions derender so the two + cannot drift.""" + if return_all or logprobs is None or logprobs == -1: + return None + return max(logprobs + 1, 0) + + def clamp_prompt_logprobs( prompt_logprobs: PromptLogprobs | None, ) -> PromptLogprobs | None: diff --git a/vllm/entrypoints/openai/chat_completion/serving.py b/vllm/entrypoints/openai/chat_completion/serving.py index 007c961ca8..9ddc20ed30 100644 --- a/vllm/entrypoints/openai/chat_completion/serving.py +++ b/vllm/entrypoints/openai/chat_completion/serving.py @@ -2,6 +2,7 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project import asyncio +import itertools import time from collections.abc import AsyncGenerator, AsyncIterator from collections.abc import Sequence as GenericSequence @@ -22,6 +23,7 @@ from vllm.entrypoints.generate.base.serving import ( GenerateBaseServing, build_per_request_timing_metrics, build_spec_decoding_metrics, + chat_top_logprobs_limit, clamp_prompt_logprobs, format_token_id_placeholder, ) @@ -1231,10 +1233,9 @@ class OpenAIServingChat(GenerateBaseServing): logprob=max(p[1].logprob, -9999.0), bytes=list(token.encode("utf-8", errors="replace")), ) - for i, p in enumerate(logprobs.items()) - if return_all - or top_logprobs == -1 - or (top_logprobs is not None and i < top_logprobs) + for p in itertools.islice( + logprobs.items(), chat_top_logprobs_limit(top_logprobs, return_all) + ) ] def _create_chat_logprobs( diff --git a/vllm/entrypoints/openai/completion/serving.py b/vllm/entrypoints/openai/completion/serving.py index e5ba20c52f..0306909237 100644 --- a/vllm/entrypoints/openai/completion/serving.py +++ b/vllm/entrypoints/openai/completion/serving.py @@ -2,6 +2,7 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project import asyncio +import itertools import time from collections.abc import AsyncGenerator, AsyncIterator from collections.abc import Sequence as GenericSequence @@ -19,6 +20,7 @@ from vllm.entrypoints.generate.base.serving import ( build_per_request_timing_metrics, build_spec_decoding_metrics, clamp_prompt_logprobs, + completion_top_logprobs_limit, format_token_id_placeholder, ) from vllm.entrypoints.openai.completion.protocol import ( @@ -721,10 +723,12 @@ class OpenAIServingCompletion(GenerateBaseServing): tokenizer, return_as_token_id=should_return_as_token_id, ): max(top_lp[1].logprob, -9999.0) - for i, top_lp in enumerate(step_top_logprobs.items()) - if logprob_token_ids - or num_output_top_logprobs == -1 - or num_output_top_logprobs >= i + for top_lp in itertools.islice( + step_top_logprobs.items(), + completion_top_logprobs_limit( + num_output_top_logprobs, bool(logprob_token_ids) + ), + ) } ) diff --git a/vllm/entrypoints/scale_out/token_in_token_out/serving.py b/vllm/entrypoints/scale_out/token_in_token_out/serving.py index f09ee6fad5..02a6810866 100644 --- a/vllm/entrypoints/scale_out/token_in_token_out/serving.py +++ b/vllm/entrypoints/scale_out/token_in_token_out/serving.py @@ -424,14 +424,12 @@ class ServingTokens(GenerateBaseServing): logprobs = self._create_text_logprobs( token_ids=token_ids, top_logprobs=out_logprobs, - num_output_top_logprobs=sampling_params.logprobs, tokenizer=tokenizer, ) else: logprobs = self._create_tokens_logprobs( token_ids=token_ids, top_logprobs=out_logprobs, - num_output_top_logprobs=sampling_params.logprobs, ) else: logprobs = None @@ -599,14 +597,12 @@ class ServingTokens(GenerateBaseServing): logprobs = self._create_text_logprobs( token_ids=delta_token_ids, top_logprobs=out_logprobs, - num_output_top_logprobs=sampling_params.logprobs, tokenizer=tokenizer, ) else: logprobs = self._create_tokens_logprobs( token_ids=delta_token_ids, top_logprobs=out_logprobs, - num_output_top_logprobs=sampling_params.logprobs, ) else: logprobs = None @@ -728,14 +724,15 @@ class ServingTokens(GenerateBaseServing): self, token_ids: GenericSequence[int], top_logprobs: GenericSequence[dict[int, Logprob] | None], - num_output_top_logprobs: int | None = None, tokenizer: TokenizerLike | None = None, ) -> ChatCompletionLogProbs: """Create OpenAI-style logprobs for ``output_mode="text"``. With a ``tokenizer`` the tokens are decoded strings and carry ``bytes``; without one (``--return-tokens-as-token-ids``) they are ``token_id:N`` - placeholders. + placeholders. Like tokens mode, ``top_logprobs`` carries every candidate + the engine returned (the sampled token first, then the top k), not cut + to the requested count. """ logprobs_content: list[ChatCompletionLogProbsContent] = [] @@ -757,14 +754,7 @@ class ServingTokens(GenerateBaseServing): bytes=token_bytes, top_logprobs=[ _top_logprob(top_id, logprob, tokenizer) - for i, (top_id, logprob) in enumerate( - step_top_logprobs.items() - ) - if num_output_top_logprobs is not None - and ( - num_output_top_logprobs == -1 - or i < max(num_output_top_logprobs, 1) - ) + for top_id, logprob in step_top_logprobs.items() ], ) ) @@ -775,10 +765,16 @@ class ServingTokens(GenerateBaseServing): self, token_ids: GenericSequence[int], top_logprobs: GenericSequence[dict[int, Logprob] | None], - num_output_top_logprobs: int | None = None, ) -> GenerateLogProbs: """Create generate-shaped logprobs (integer token ids, no tokenizer). + ``top_logprobs`` carries every candidate the engine returned for the + position: the sampled token first, then the top k, so k entries when the + sampled token is in the top k and k + 1 when it is not. It is not cut to + the requested count: `/v1/chat/completions` and + `/v1/completions` cut differently, and derender applies each endpoint's + cut the way the coupled server does. + The engine reports rank 0 for a sampled token whose logprob is NaN; that is not a rank, so it is sent as ``None`` (the logprob is clamped). """ @@ -806,14 +802,7 @@ class ServingTokens(GenerateBaseServing): logprob=_clamp_logprob(top_logprob.logprob), rank=top_logprob.rank or None, ) - for rank_index, (top_token_id, top_logprob) in enumerate( - step_top_logprobs.items() - ) - if num_output_top_logprobs is not None - and ( - num_output_top_logprobs == -1 - or rank_index < max(num_output_top_logprobs, 1) - ) + for top_token_id, top_logprob in step_top_logprobs.items() ], ) ) diff --git a/vllm/renderers/online_derenderer.py b/vllm/renderers/online_derenderer.py index 9a22df962e..0c9412816a 100644 --- a/vllm/renderers/online_derenderer.py +++ b/vllm/renderers/online_derenderer.py @@ -9,7 +9,11 @@ from vllm.entrypoints.generate.base.protocol import ( DeltaMessage, ToolCall, ) -from vllm.entrypoints.generate.base.serving import decode_token_ids +from vllm.entrypoints.generate.base.serving import ( + chat_top_logprobs_limit, + completion_top_logprobs_limit, + decode_token_ids, +) from vllm.entrypoints.openai.chat_completion.protocol import ( ChatCompletionLogProb, ChatCompletionLogProbs, @@ -204,12 +208,17 @@ class OnlineDerenderer: and chat_request.return_token_ids and (not has_parser or chat_request.include_reasoning) ) + top_limit = _chat_top_logprobs_limit(chat_request) for choice in generate_response.choices: if not choice.token_ids: raise ValueError(f"choice {choice.index} has empty or null token_ids") resolved_logprobs = ( - _resolve_logprobs(choice.logprobs, tokenizer) + _resolve_logprobs( + choice.logprobs, + tokenizer, + top_limit=top_limit, + ) if choice.logprobs is not None else None ) @@ -486,6 +495,7 @@ class OnlineDerenderer: updated_state = state return_token_ids = chat_request is not None and chat_request.return_token_ids + top_limit = _chat_top_logprobs_limit(chat_request) for choice in generate_chunk.choices: delta_tids = choice.token_ids or [] new_text, updated_state = await self._detokenize_delta_async( @@ -511,6 +521,7 @@ class OnlineDerenderer: choice.logprobs, tokenizer, initial_context_token_ids=state.logprob_context_token_ids, + top_limit=top_limit, ) include_role = not updated_state.role_sent @@ -794,6 +805,7 @@ class OnlineDerenderer: total_prompt_tokens = 0 total_completion_tokens = 0 index = 0 + top_limit = _completion_top_logprobs_limit(completion_request) for gen, pt, seed_ids_override in zip( generate_responses, prompt_tokens_list, prompt_token_ids_list @@ -822,9 +834,13 @@ class OnlineDerenderer: ) completion_logprobs = None if choice.logprobs is not None: - resolved = _resolve_logprobs(choice.logprobs, tokenizer) + resolved = _resolve_logprobs( + choice.logprobs, + tokenizer, + top_limit=top_limit, + ) completion_logprobs = _convert_chat_logprobs_to_completion_logprobs( - resolved + resolved, choice.logprobs ) choices.append( CompletionResponseChoice( @@ -897,6 +913,7 @@ class OnlineDerenderer: stream_choices: list[CompletionResponseStreamChoice] = [] updated_state = state + top_limit = _completion_top_logprobs_limit(completion_request) for choice in generate_chunk.choices: delta_tids = choice.token_ids or [] new_text, updated_state = await self._detokenize_delta_async( @@ -913,9 +930,12 @@ class OnlineDerenderer: choice.logprobs, tokenizer, initial_context_token_ids=state.logprob_context_token_ids, + top_limit=top_limit, ) completion_logprobs = _convert_chat_logprobs_to_completion_logprobs( - resolved, initial_text_offset=state.logprob_text_offset + resolved, + choice.logprobs, + initial_text_offset=state.logprob_text_offset, ) updated_state = updated_state.model_copy( @@ -1067,10 +1087,38 @@ def _correct_decoded_token( return "" +def _chat_top_logprobs_limit(chat_request: ChatCompletionRequest | None) -> int | None: + """The cut `/v1/chat/completions` applies (`chat_top_logprobs_limit`, the + same helper its `_get_top_logprobs` uses). None keeps every candidate, which + is also what happens without a `chat_request` or with one that does not ask + for logprobs: it then says nothing about the cut, so the candidates + generate returned are passed through.""" + if chat_request is None or not chat_request.logprobs: + return None + return chat_top_logprobs_limit( + chat_request.top_logprobs, bool(chat_request.logprob_token_ids) + ) + + +def _completion_top_logprobs_limit( + completion_request: CompletionRequest | None, +) -> int | None: + """The cut `/v1/completions` applies (`completion_top_logprobs_limit`, the + same helper `_create_completion_logprobs` uses). None keeps every candidate, + which is also what happens without a `completion_request` or with one that + does not ask for logprobs (`logprobs` unset).""" + if completion_request is None: + return None + return completion_top_logprobs_limit( + completion_request.logprobs, bool(completion_request.logprob_token_ids) + ) + + def _resolve_logprobs( logprobs: GenerateLogProbs, tokenizer: TokenizerLike, initial_context_token_ids: Sequence[int] = (), + top_limit: int | None = None, ) -> ChatCompletionLogProbs: """Convert generate's integer-id logprobs to the OpenAI chat shape. @@ -1080,6 +1128,11 @@ def _resolve_logprobs( ids as context). ``initial_context_token_ids`` seeds that context with sampled IDs from preceding chunks (streaming), so multi-byte characters split across chunk boundaries still resolve. + + Generate returns every candidate the engine produced for a position (the + sampled token first, then the top k). ``top_limit`` applies the target + endpoint's cut before decoding, so each derender endpoint returns what its + coupled counterpart would; None keeps all of them. """ if logprobs.content is None: return ChatCompletionLogProbs() @@ -1088,9 +1141,12 @@ def _resolve_logprobs( resolved_content = [] for entry in logprobs.content: - # One batch per position: the sampled id and its top-k ids. + top_entries = ( + entry.top_logprobs if top_limit is None else entry.top_logprobs[:top_limit] + ) + # One batch per position: the sampled id and its kept top-k ids. (token_str, token_bytes), *top_decoded = decode_token_ids( - [entry.token_id, *(top.token_id for top in entry.top_logprobs)], + [entry.token_id, *(top.token_id for top in top_entries)], tokenizer, ) @@ -1101,7 +1157,7 @@ def _resolve_logprobs( token_bytes = list(token_str.encode("utf-8")) resolved_top = [] - for top, (top_str, top_bytes) in zip(entry.top_logprobs, top_decoded): + for top, (top_str, top_bytes) in zip(top_entries, top_decoded): if top_str.endswith("\ufffd"): top_str = _correct_decoded_token( top.token_id, context_token_ids, tokenizer @@ -1131,11 +1187,17 @@ def _resolve_logprobs( def _convert_chat_logprobs_to_completion_logprobs( logprobs: ChatCompletionLogProbs, + generate_logprobs: GenerateLogProbs, initial_text_offset: int = 0, ) -> CompletionLogProbs: """Convert ChatCompletionLogProbs (per-token objects) to CompletionLogProbs (parallel flat lists) as required by the /v1/completions response schema. + ``generate_logprobs`` is what ``logprobs`` was resolved from. A position + with no candidates there had no logprobs in the engine output, which + /v1/completions reports as ``None``; every other position is a dict of + the candidates the cut kept. + ``initial_text_offset`` keeps ``text_offset`` absolute across streaming chunks, mirroring the generate streaming path. """ @@ -1148,13 +1210,13 @@ def _convert_chat_logprobs_to_completion_logprobs( text_offset: list[int] = [] offset = initial_text_offset - for entry in logprobs.content: + for entry, source in zip(logprobs.content, generate_logprobs.content or ()): text_offset.append(offset) tokens.append(entry.token) token_logprobs.append(entry.logprob) top_logprobs_list.append( {t.token: t.logprob for t in entry.top_logprobs} - if entry.top_logprobs + if source.top_logprobs else None ) offset += len(entry.token)