[Frontend] Score with the chat template declared by Sentence Transformers CrossEncoders (#60877)

Signed-off-by: rdlhio <contact@rdlh.io>
This commit is contained in:
Rémi Delhaye
2026-10-18 11:30:12 +00:00
committed by GitHub
parent 953fa442fa
commit 3c13653738
7 changed files with 192 additions and 27 deletions
+25
View File
@@ -466,6 +466,31 @@ This approach is more robust than index-based access (`messages[0]`, `messages[1
Example template file: [examples/pooling/score/template/nemotron-rerank.jinja](../../../examples/pooling/score/template/nemotron-rerank.jinja)
#### Templates declared by Sentence Transformers checkpoints
vLLM does not apply a checkpoint's chat template by default, since many rerankers inherit an unrelated template from their base LLM. The exception is a Sentence Transformers (v5.4+) `CrossEncoder` with a single `sequence-classification` Transformer module whose `sentence_bert_config.json` declares the `message` modality. Sentence Transformers formats the pairs of such checkpoints as `query`/`document` messages, so vLLM does the same with the saved template:
```json
{
"transformer_task": "sequence-classification",
"modality_config": {
"text": {"method": "forward", "method_output_name": "logits"},
"message": {"method": "forward", "method_output_name": "logits", "format": "flat"}
},
"module_output_name": "scores",
"processing_kwargs": {"chat_template": {"add_generation_prompt": true}}
}
```
- The default chat template is used, unless `processing_kwargs.chat_template.chat_template` names another template saved with the tokenizer (e.g. `additional_chat_templates/score.jinja` for `"score"`).
- The other `processing_kwargs.chat_template` settings, such as `add_generation_prompt`, are passed to the template; the request's `chat_template_kwargs` override them.
- Message contents are passed as strings, as with `--chat-template`.
- vLLM does not restore the template suffix after truncation (`restore_suffix`). Use `max_tokens_per_doc` rather than `truncate_prompt_tokens` to truncate long documents.
- `--chat-template` (or the `chat_template` argument of `LLM.score`) takes precedence over the saved template and its settings.
- Models with a built-in score template keep using it.
The template in use is logged at startup.
### Enable/disable activation
You can enable or disable activation via `use_activation` only works for cross-encoder models.