[Docs] Reorganize online serving docs. (#41907)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io> Signed-off-by: wang.yuqi <noooop@126.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
This commit is contained in:
co-authored by
gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Cyrus Leung
Harry Mellor
parent
4a4fdabe28
commit
257af77bc2
@@ -141,24 +141,24 @@ enabling the corresponding APIs.
|
||||
|
||||
### `LLM.classify`
|
||||
|
||||
The [classify][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.classify] method outputs a probability vector for each prompt.
|
||||
The [classify][vllm.LLM.classify] method outputs a probability vector for each prompt.
|
||||
It is primarily designed for [classification models](classify.md).
|
||||
For more information about `LLM.embed`, see [this page](classify.md#offline-inference).
|
||||
|
||||
### `LLM.embed`
|
||||
|
||||
The [embed][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.embed] method outputs an embedding vector for each prompt.
|
||||
The [embed][vllm.LLM.embed] method outputs an embedding vector for each prompt.
|
||||
It is primarily designed for [embedding models](embed.md).
|
||||
For more information about `LLM.embed`, see [this page](embed.md#offline-inference).
|
||||
|
||||
### `LLM.score`
|
||||
|
||||
The [score][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.score] method outputs similarity scores between sentence pairs.
|
||||
The [score][vllm.LLM.score] method outputs similarity scores between sentence pairs.
|
||||
It is primarily designed for [score models](scoring.md).
|
||||
|
||||
### `LLM.encode`
|
||||
|
||||
The [encode][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.encode] method is available to all pooling models in vLLM.
|
||||
The [encode][vllm.LLM.encode] method is available to all pooling models in vLLM.
|
||||
|
||||
Please use one of the more specific methods or set the task directly when using `LLM.encode`, refer to the [table above](#offline-apis-corresponding-to-pooling-usages).
|
||||
|
||||
@@ -180,12 +180,12 @@ Our online Server provides endpoints that correspond to the offline APIs:
|
||||
|
||||
- Corresponding to `LLM.embed`:
|
||||
- [Cohere Embed API](embed.md#cohere-embed-api) (`/v2/embed`)
|
||||
- [Openai-compatible Embeddings API](embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`)
|
||||
- [OpenAI-compatible Embeddings API](embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`)
|
||||
- Corresponding to `LLM.classify`:
|
||||
- [Classification API](classify.md#online-serving)(`/classify`)
|
||||
- Corresponding to `LLM.score`:
|
||||
- [Score API](scoring.md#score-api)(`/score`)
|
||||
- [Rerank API](scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`)
|
||||
- [Cohere Rerank API](scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`)
|
||||
- Pooling API (`/pooling`) is similar to `LLM.encode`, being applicable to all types of pooling models.
|
||||
|
||||
The following introduces the Pooling API. For other APIs, please refer to the link above.
|
||||
|
||||
@@ -12,7 +12,7 @@ Embedding models are a class of machine learning models designed to transform un
|
||||
- `LLM.score(...)`
|
||||
- Online APIs:
|
||||
- [Cohere Embed API](embed.md#cohere-embed-api) (`/v2/embed`)
|
||||
- [Openai-compatible Embeddings API](embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`)
|
||||
- [OpenAI-compatible Embeddings API](embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`)
|
||||
- Pooling API (`/pooling`)
|
||||
|
||||
The primary distinction between (sequence) embedding and token embedding lies in their output granularity: (sequence) embedding produces a single embedding vector for an entire input sequence, whereas token embedding generates an embedding for each individual token within the sequence.
|
||||
@@ -228,7 +228,7 @@ these extra parameters are supported instead:
|
||||
|
||||
#### Examples
|
||||
|
||||
If the model has a [chat template](../../serving/openai_compatible_server.md#chat-template), you can replace `inputs` with a list of `messages` (same schema as [Chat API](../../serving/openai_compatible_server.md#chat-api))
|
||||
If the model has a [chat template](../../serving/online_serving/README.md#chat-template), you can replace `inputs` with a list of `messages` (same schema as [Chat API](../../serving/online_serving/openai_compatible_server.md#chat-api))
|
||||
which will be treated as a single prompt to the model. Here is a convenience function for calling the API while retaining OpenAI's type annotations:
|
||||
|
||||
??? code
|
||||
|
||||
@@ -133,7 +133,7 @@ print(f"Data: {data!r}")
|
||||
|
||||
## Online Serving
|
||||
|
||||
Please refer to the [pooling API](README.md#pooling-api). Pooling task corresponding to reward model types refer to the [table above](#summary).
|
||||
Please refer to the [Pooling API](README.md#pooling-api). Pooling task corresponding to reward model types refer to the [table above](#summary).
|
||||
|
||||
## More examples
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@ The score models is designed to compute similarity scores between two input prom
|
||||
- `LLM.score`
|
||||
- Online APIs:
|
||||
- [Score API](scoring.md#score-api) (`/score`)
|
||||
- [Rerank API](scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`)
|
||||
- [Cohere Rerank API](scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`)
|
||||
|
||||
!!! note
|
||||
Only when a classification model outputs num_labels equal to 1 can it be used as a scoring model and have its scoring API enabled.
|
||||
@@ -363,7 +363,7 @@ Full example:
|
||||
- [examples/pooling/score/vision_score_api_online.py](../../../examples/pooling/score/vision_score_api_online.py)
|
||||
- [examples/pooling/score/vision_rerank_api_online.py](../../../examples/pooling/score/vision_rerank_api_online.py)
|
||||
|
||||
### Rerank API
|
||||
### Cohere Rerank API
|
||||
|
||||
`/rerank`, `/v1/rerank`, and `/v2/rerank` APIs are compatible with both [Jina AI's rerank API interface](https://jina.ai/reranker/) and
|
||||
[Cohere's rerank API interface](https://docs.cohere.com/v2/reference/rerank) to ensure compatibility with
|
||||
@@ -446,7 +446,7 @@ AS cross-encoder models are a subset of classification models that accept two pr
|
||||
|
||||
Score templates are supported for **cross-encoder** models only. If you are using an **embedding** model for scoring, vLLM does not apply a score template.
|
||||
|
||||
Some scoring models require a specific prompt format to work correctly. You can specify a custom score template using the `--chat-template` parameter (see [Chat Template](../../serving/openai_compatible_server.md#chat-template)).
|
||||
Some scoring models require a specific prompt format to work correctly. You can specify a custom score template using the `--chat-template` parameter (see [Chat Template](../../serving/online_serving/README.md#chat-template)).
|
||||
|
||||
Like chat templates, the score template receives a `messages` list. For scoring, each message has a `role` attribute—either `"query"` or `"document"`. For the usual kind of point-wise cross-encoder, you can expect exactly two messages: one query and one document. To access the query and document content, use Jinja's `selectattr` filter:
|
||||
|
||||
|
||||
@@ -60,7 +60,7 @@ curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{
|
||||
}'
|
||||
```
|
||||
|
||||
You can also get the raw token embeddings using the pooling API with `token_embed` task:
|
||||
You can also get the raw token embeddings using the Pooling API with `token_embed` task:
|
||||
|
||||
```shell
|
||||
curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{
|
||||
@@ -174,7 +174,7 @@ curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{
|
||||
}'
|
||||
```
|
||||
|
||||
For **image inputs** via the pooling API, use the chat-style `messages` field:
|
||||
For **image inputs** via the Pooling API, use the chat-style `messages` field:
|
||||
|
||||
```shell
|
||||
curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{
|
||||
|
||||
@@ -103,7 +103,7 @@ print(f"Data: {data!r}")
|
||||
|
||||
## Online Serving
|
||||
|
||||
Please refer to the [pooling API](README.md#pooling-api) and use `"task":"token_classify"`.
|
||||
Please refer to the [Pooling API](README.md#pooling-api) and use `"task":"token_classify"`.
|
||||
|
||||
## More examples
|
||||
|
||||
|
||||
@@ -129,7 +129,7 @@ print(f"Score: {score}")
|
||||
|
||||
## Online Serving
|
||||
|
||||
Please refer to the [pooling API](README.md#pooling-api) and use `"task":"token_embed"`.
|
||||
Please refer to the [Pooling API](README.md#pooling-api) and use `"task":"token_embed"`.
|
||||
|
||||
## More examples
|
||||
|
||||
|
||||
Reference in New Issue
Block a user