Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com> Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
67 lines
2.6 KiB
Markdown
67 lines
2.6 KiB
Markdown
# Prompt Token ID Logprobs (Teacher Scoring)
|
|
|
|
In on-policy distillation, the student records its top-K candidate tokens at
|
|
every response position, and the teacher provides the log probabilities of
|
|
exactly those candidates as the distillation target. The caller passes one
|
|
candidate list per scored row as `prompt_logprob_token_ids`; the teacher runs a
|
|
single prefill over the prompt plus response and gathers their log
|
|
probabilities on the GPU, starting from `prompt_logprob_start` (typically the
|
|
prompt length minus one, so only response positions are scored). The result is
|
|
a `[rows, K]` matrix whose row `i` scores prompt token
|
|
`prompt_logprob_start + i + 1` and whose columns follow that row's candidate
|
|
order, so it lines up position by position with the student's top-K.
|
|
|
|
## Quick start
|
|
|
|
```python
|
|
from vllm import LLM, SamplingParams
|
|
from vllm.inputs import TokensPrompt
|
|
|
|
llm = LLM(model)
|
|
output = llm.generate(
|
|
TokensPrompt(prompt_token_ids=prompt_ids + response_ids),
|
|
SamplingParams(
|
|
max_tokens=1,
|
|
prompt_logprob_token_ids=student_topk_ids, # [len(response_ids), K]
|
|
prompt_logprob_start=len(prompt_ids) - 1,
|
|
),
|
|
)
|
|
scores = output[0].prompt_token_id_logprobs
|
|
# scores.shape == (len(response_ids), K)
|
|
# scores[i, j] = log p(student_topk_ids[i][j] | prompt_ids + response_ids[:i])
|
|
```
|
|
|
|
The same parameters are accepted by the `/inference/v1/generate` HTTP endpoint
|
|
(Python and Rust frontends), which returns the matrix as a base64-encoded
|
|
`.npy` float32 array:
|
|
|
|
```python
|
|
import base64, io
|
|
import numpy as np
|
|
|
|
scores = np.load(io.BytesIO(base64.b64decode(response["prompt_token_id_logprobs"])))
|
|
```
|
|
|
|
`prompt_logprob_token_ids` is an integer array or a list of lists with exactly
|
|
`prompt_len - 1 - prompt_logprob_start` rows; shorter rows are padded with `-1`,
|
|
and every `-1` entry scores `-inf`. Pass an `int32` NumPy array, such as the
|
|
student's top-K IDs, which is sent to the engine without conversion; nested
|
|
lists of Python ints are much slower to copy and serialize. With a logits
|
|
`--logprobs-mode`, the matrix holds logits instead of log probabilities.
|
|
|
|
## Requirements
|
|
|
|
- The V2 model runner (`VLLM_USE_V2_MODEL_RUNNER=1`).
|
|
- The longest row at most `--max-logprobs` candidates.
|
|
- No `--kv-sharing-fast-prefill`.
|
|
|
|
The request skips reading the prefix cache (local and KV connector), since
|
|
cached rows have no logits to score.
|
|
|
|
## Limitations
|
|
|
|
- Non-streaming only; `stream=true` requests are rejected.
|
|
- Not exposed through the OpenAI-compatible endpoints.
|
|
- If a prefill starts past the first scored row (e.g. `skip_reading_prefix_cache=False`
|
|
with a cache hit), the result is `None` rather than a partial matrix.
|