[Doc] Score centering via top-k processed logprobs (#59361)

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
vx120
2026-10-01 08:57:00 +08:00
committed by GitHub
co-authored by aoshen02 Claude Fable 5.1
parent c055c1e075
commit a47e4a5568
+33
View File
@@ -104,6 +104,39 @@ log_prob = log_softmax(masked_logits)[sampled_token_id]
Both sides normalize over the same token set, so the importance ratio is
consistent.
## Score centering with top-k logprobs
[Score centering](https://arxiv.org/abs/2609.20807) needs the sampler's top-k
token IDs and logprobs plus the sampled token's logprob; the paper uses
`k=128` and does not need the exact sampling support. vLLM already returns
this through the ordinary `logprobs` parameter:
```bash
vllm serve <model> --logprobs-mode processed_logprobs --max-logprobs 128
```
```json
{"sampling_params": {"temperature": 1.0, "top_p": 0.9, "logprobs": 128}}
```
Each position then carries the sampled token's logprob and the top-128
`(token_id, logprob)` pairs, normalized over the post-truncation support:
```json
{"token": "token_id:11", "logprob": -0.757,
"top_logprobs": [{"token": "token_id:11", "logprob": -0.757},
{"token": "token_id:13", "logprob": -0.632},
{"token": "token_id:0", "logprob": -9999.0}, ...]}
```
- `--max-logprobs` defaults to 20; raise it to the requested `k`.
- Tokens outside the support are returned as `-9999.0` with arbitrary IDs;
drop them. The finite entries are the sampling mask when it has at most `k`
tokens.
- The sampled token is always the first entry and may repeat inside the top-k.
- `/inference/v1/generate` encodes IDs as `token_id:<id>`; the OpenAI
endpoints need `return_tokens_as_token_ids`.
## Limitations
- **Engine-level flag:** `--return-sampling-mask` globally disables the