# Prompt Token ID Logprobs (Teacher Scoring) In on-policy distillation, the student records its top-K candidate tokens at every response position, and the teacher provides the log probabilities of exactly those candidates as the distillation target. The caller passes one candidate list per scored row as `prompt_logprob_token_ids`; the teacher runs a single prefill over the prompt plus response and gathers their log probabilities on the GPU, starting from `prompt_logprob_start` (typically the prompt length minus one, so only response positions are scored). The result is a `[rows, K]` matrix whose row `i` scores prompt token `prompt_logprob_start + i + 1` and whose columns follow that row's candidate order, so it lines up position by position with the student's top-K. ## Quick start ```python from vllm import LLM, SamplingParams from vllm.inputs import TokensPrompt llm = LLM(model) output = llm.generate( TokensPrompt(prompt_token_ids=prompt_ids + response_ids), SamplingParams( max_tokens=1, prompt_logprob_token_ids=student_topk_ids, # [len(response_ids), K] prompt_logprob_start=len(prompt_ids) - 1, ), ) scores = output[0].prompt_token_id_logprobs # scores.shape == (len(response_ids), K) # scores[i, j] = log p(student_topk_ids[i][j] | prompt_ids + response_ids[:i]) ``` The same parameters are accepted by the `/inference/v1/generate` HTTP endpoint (Python and Rust frontends), which returns the matrix as a base64-encoded `.npy` float32 array: ```python import base64, io import numpy as np scores = np.load(io.BytesIO(base64.b64decode(response["prompt_token_id_logprobs"]))) ``` `prompt_logprob_token_ids` is an integer array or a list of lists with exactly `prompt_len - 1 - prompt_logprob_start` rows; shorter rows are padded with `-1`, and every `-1` entry scores `-inf`. Pass an `int32` NumPy array, such as the student's top-K IDs, which is sent to the engine without conversion; nested lists of Python ints are much slower to copy and serialize. With a logits `--logprobs-mode`, the matrix holds logits instead of log probabilities. ## Requirements - The V2 model runner (`VLLM_USE_V2_MODEL_RUNNER=1`). - The longest row at most `--max-logprobs` candidates. - No `--kv-sharing-fast-prefill`. The request skips reading the prefix cache (local and KV connector), since cached rows have no logits to score. ## Limitations - Non-streaming only; `stream=true` requests are rejected. - Not exposed through the OpenAI-compatible endpoints. - If a prefill starts past the first scored row (e.g. `skip_reading_prefix_cache=False` with a cache hit), the result is `None` rather than a partial matrix.