[Frontend] Score with the chat template declared by Sentence Transformers CrossEncoders (#60877)
Signed-off-by: rdlhio <contact@rdlh.io>
This commit is contained in:
@@ -466,6 +466,31 @@ This approach is more robust than index-based access (`messages[0]`, `messages[1
|
||||
|
||||
Example template file: [examples/pooling/score/template/nemotron-rerank.jinja](../../../examples/pooling/score/template/nemotron-rerank.jinja)
|
||||
|
||||
#### Templates declared by Sentence Transformers checkpoints
|
||||
|
||||
vLLM does not apply a checkpoint's chat template by default, since many rerankers inherit an unrelated template from their base LLM. The exception is a Sentence Transformers (v5.4+) `CrossEncoder` with a single `sequence-classification` Transformer module whose `sentence_bert_config.json` declares the `message` modality. Sentence Transformers formats the pairs of such checkpoints as `query`/`document` messages, so vLLM does the same with the saved template:
|
||||
|
||||
```json
|
||||
{
|
||||
"transformer_task": "sequence-classification",
|
||||
"modality_config": {
|
||||
"text": {"method": "forward", "method_output_name": "logits"},
|
||||
"message": {"method": "forward", "method_output_name": "logits", "format": "flat"}
|
||||
},
|
||||
"module_output_name": "scores",
|
||||
"processing_kwargs": {"chat_template": {"add_generation_prompt": true}}
|
||||
}
|
||||
```
|
||||
|
||||
- The default chat template is used, unless `processing_kwargs.chat_template.chat_template` names another template saved with the tokenizer (e.g. `additional_chat_templates/score.jinja` for `"score"`).
|
||||
- The other `processing_kwargs.chat_template` settings, such as `add_generation_prompt`, are passed to the template; the request's `chat_template_kwargs` override them.
|
||||
- Message contents are passed as strings, as with `--chat-template`.
|
||||
- vLLM does not restore the template suffix after truncation (`restore_suffix`). Use `max_tokens_per_doc` rather than `truncate_prompt_tokens` to truncate long documents.
|
||||
- `--chat-template` (or the `chat_template` argument of `LLM.score`) takes precedence over the saved template and its settings.
|
||||
- Models with a built-in score template keep using it.
|
||||
|
||||
The template in use is logged at startup.
|
||||
|
||||
### Enable/disable activation
|
||||
|
||||
You can enable or disable activation via `use_activation` only works for cross-encoder models.
|
||||
|
||||
@@ -313,14 +313,12 @@ Start the server:
|
||||
```shell
|
||||
vllm serve nvidia/llama-nemotron-rerank-vl-1b-v2 \
|
||||
--runner pooling \
|
||||
--trust-remote-code \
|
||||
--chat-template examples/pooling/score/template/nemotron-vl-rerank.jinja
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
!!! note
|
||||
The chat template bundled with this checkpoint's tokenizer is not suitable
|
||||
for the Score/Rerank APIs. Use the provided override template when serving:
|
||||
`examples/pooling/score/template/nemotron-vl-rerank.jinja`.
|
||||
The checkpoint declares its score template in its Sentence Transformers config, so vLLM applies it automatically (see [Templates declared by Sentence Transformers checkpoints](scoring.md#templates-declared-by-sentence-transformers-checkpoints)).
|
||||
Revisions older than October 2026 need `--chat-template examples/pooling/score/template/nemotron-vl-rerank.jinja`.
|
||||
|
||||
Score a text query against an image document:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user