[Frontend] Score with the chat template declared by Sentence Transformers CrossEncoders (#60877)

Signed-off-by: rdlhio <contact@rdlh.io>
This commit is contained in:
Rémi Delhaye
2026-10-18 11:30:12 +00:00
committed by GitHub
parent 953fa442fa
commit 3c13653738
7 changed files with 192 additions and 27 deletions
+25
View File
@@ -466,6 +466,31 @@ This approach is more robust than index-based access (`messages[0]`, `messages[1
Example template file: [examples/pooling/score/template/nemotron-rerank.jinja](../../../examples/pooling/score/template/nemotron-rerank.jinja)
#### Templates declared by Sentence Transformers checkpoints
vLLM does not apply a checkpoint's chat template by default, since many rerankers inherit an unrelated template from their base LLM. The exception is a Sentence Transformers (v5.4+) `CrossEncoder` with a single `sequence-classification` Transformer module whose `sentence_bert_config.json` declares the `message` modality. Sentence Transformers formats the pairs of such checkpoints as `query`/`document` messages, so vLLM does the same with the saved template:
```json
{
"transformer_task": "sequence-classification",
"modality_config": {
"text": {"method": "forward", "method_output_name": "logits"},
"message": {"method": "forward", "method_output_name": "logits", "format": "flat"}
},
"module_output_name": "scores",
"processing_kwargs": {"chat_template": {"add_generation_prompt": true}}
}
```
- The default chat template is used, unless `processing_kwargs.chat_template.chat_template` names another template saved with the tokenizer (e.g. `additional_chat_templates/score.jinja` for `"score"`).
- The other `processing_kwargs.chat_template` settings, such as `add_generation_prompt`, are passed to the template; the request's `chat_template_kwargs` override them.
- Message contents are passed as strings, as with `--chat-template`.
- vLLM does not restore the template suffix after truncation (`restore_suffix`). Use `max_tokens_per_doc` rather than `truncate_prompt_tokens` to truncate long documents.
- `--chat-template` (or the `chat_template` argument of `LLM.score`) takes precedence over the saved template and its settings.
- Models with a built-in score template keep using it.
The template in use is logged at startup.
### Enable/disable activation
You can enable or disable activation via `use_activation` only works for cross-encoder models.
@@ -313,14 +313,12 @@ Start the server:
```shell
vllm serve nvidia/llama-nemotron-rerank-vl-1b-v2 \
--runner pooling \
--trust-remote-code \
--chat-template examples/pooling/score/template/nemotron-vl-rerank.jinja
--trust-remote-code
```
!!! note
The chat template bundled with this checkpoint's tokenizer is not suitable
for the Score/Rerank APIs. Use the provided override template when serving:
`examples/pooling/score/template/nemotron-vl-rerank.jinja`.
The checkpoint declares its score template in its Sentence Transformers config, so vLLM applies it automatically (see [Templates declared by Sentence Transformers checkpoints](scoring.md#templates-declared-by-sentence-transformers-checkpoints)).
Revisions older than October 2026 need `--chat-template examples/pooling/score/template/nemotron-vl-rerank.jinja`.
Score a text query against an image document: