[Frontend][last/5] Improve pooling entrypoints | clean up. (#39675)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io> Signed-off-by: wang.yuqi <noooop@126.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
parent
5e5afafa21
commit
4e8c3f1c19
@@ -59,6 +59,16 @@ please refer to [IO Processor Plugins](../../design/io_processor_plugins.md).
|
||||
Within classification tasks, there is a specialized subcategory: Cross-encoder (aka reranker) models. These models
|
||||
are a subset of classification models that accept two prompts as input and output num_labels equal to 1.
|
||||
|
||||
### Pooling Types
|
||||
|
||||
| Pooling Tasks | Granularity | Description |
|
||||
|----------------|---------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `CLS` pooling | Sequence-wise | For BERT‑like (bidirectional self‑attention) models, CLS pooling is used by default. This means the last_hidden_states corresponding to the first token (the [CLS] token) is taken as the output. |
|
||||
| `LAST` pooling | Sequence-wise | For GPT‑like (causal self‑attention) models, LAST pooling is used by default. This means the last_hidden_states corresponding to the last token is taken as the output. |
|
||||
| `MEAN` pooling | Sequence-wise | Many studies have shown that averaging the last_hidden_states over all input tokens performs better on certain downstream tasks. Therefore, more and more models are using MEAN pooling. |
|
||||
| `ALL` pooling | Token-wise | Outputs the last_hidden_states for all input tokens. |
|
||||
| `STEP` pooling | Token-wise | Filters and outputs the last_hidden_states corresponding to the token IDs returned by returned_token_ids. |
|
||||
|
||||
### Score Types
|
||||
|
||||
The scoring models is designed to compute similarity scores between two input prompts. It supports three model types
|
||||
|
||||
@@ -160,6 +160,8 @@ The following Score API parameters are supported:
|
||||
--8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"
|
||||
--8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"
|
||||
--8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-extra-params"
|
||||
--8<-- "vllm/entrypoints/pooling/scoring/protocol.py:scoring-common-params"
|
||||
--8<-- "vllm/entrypoints/pooling/scoring/protocol.py:score-request-params"
|
||||
```
|
||||
|
||||
#### Examples
|
||||
@@ -370,6 +372,8 @@ The following rerank api parameters are supported:
|
||||
--8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-params"
|
||||
--8<-- "vllm/entrypoints/pooling/base/protocol.py:pooling-common-extra-params"
|
||||
--8<-- "vllm/entrypoints/pooling/base/protocol.py:classify-extra-params"
|
||||
--8<-- "vllm/entrypoints/pooling/scoring/protocol.py:scoring-common-params"
|
||||
--8<-- "vllm/entrypoints/pooling/scoring/protocol.py:rerank-request-params"
|
||||
```
|
||||
|
||||
#### Examples
|
||||
|
||||
@@ -68,7 +68,7 @@ If your model is not in the above list, we will try to automatically convert the
|
||||
Forced alignment usage requires `--hf-overrides '{"architectures": ["Qwen3ASRForcedAlignerForTokenClassification"]}'`.
|
||||
Please refer to [examples/pooling/token_classify/forced_alignment_offline.py](../../../examples/pooling/token_classify/forced_alignment_offline.py).
|
||||
|
||||
### As Reward Models
|
||||
### Reward Models
|
||||
|
||||
Using token classification models as reward models. For details on reward models, see [Reward Models](reward.md).
|
||||
|
||||
|
||||
@@ -467,28 +467,11 @@ It consists of two endpoints:
|
||||
- `/tokenize` corresponds to calling `tokenizer.encode()`.
|
||||
- `/detokenize` corresponds to calling `tokenizer.decode()`.
|
||||
|
||||
### Score API
|
||||
|
||||
#### Score Template
|
||||
|
||||
Some scoring models require a specific prompt format to work correctly. You can specify a custom score template using the `--chat-template` parameter (see [Chat Template](#chat-template)).
|
||||
|
||||
Score templates are supported for **cross-encoder** models only. If you are using an **embedding** model for scoring, vLLM does not apply a score template.
|
||||
|
||||
Like chat templates, the score template receives a `messages` list. For scoring, each message has a `role` attribute—either `"query"` or `"document"`. For the usual kind of point-wise cross-encoder, you can expect exactly two messages: one query and one document. To access the query and document content, use Jinja's `selectattr` filter:
|
||||
|
||||
- **Query**: `{{ (messages | selectattr("role", "eq", "query") | first).content }}`
|
||||
- **Document**: `{{ (messages | selectattr("role", "eq", "document") | first).content }}`
|
||||
|
||||
This approach is more robust than index-based access (`messages[0]`, `messages[1]`) because it selects messages by their semantic role. It also avoids assumptions about message ordering if additional message types are added to `messages` in the future.
|
||||
|
||||
Example template file: [examples/pooling/score/template/nemotron-rerank.jinja](../../examples/pooling/score/template/nemotron-rerank.jinja)
|
||||
|
||||
### Generative Scoring API
|
||||
|
||||
The `/generative_scoring` endpoint uses a CausalLM model (e.g., Llama, Qwen, Mistral) to compute the probability of specified token IDs appearing as the next token. Each item (document) is concatenated with the query to form a prompt, and the model predicts how likely each label token is as the next token after that prompt. This lets you score items against a query — for example, asking "Is this the capital of France?" and scoring each city by how likely the model is to answer "Yes".
|
||||
|
||||
This endpoint is automatically available when the server is started with a generative model (task `"generate"`). It is separate from the pooling-based [Score API](#score-api), which uses cross-encoder, bi-encoder, or late-interaction models.
|
||||
This endpoint is automatically available when the server is started with a generative model (task `"generate"`). It is separate from the pooling-based [Score API](../models/pooling_models/scoring.md#score-api), which uses cross-encoder, bi-encoder, or late-interaction models.
|
||||
|
||||
**Requirements:**
|
||||
|
||||
|
||||
Reference in New Issue
Block a user