diff --git a/docs/.nav.yml b/docs/.nav.yml index 89584442e3..7d985fdeb5 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -11,7 +11,7 @@ nav: - usage/* - Inference and Serving: - serving/offline_inference.md - - serving/openai_compatible_server.md + - Online Serving: serving/online_serving - serving/* - serving/integrations - Deployment: diff --git a/docs/assets/models/pooling_models/cheat_sheet.svg b/docs/assets/models/pooling_models/cheat_sheet.svg index a4f108cca1..f8fc17569c 100644 --- a/docs/assets/models/pooling_models/cheat_sheet.svg +++ b/docs/assets/models/pooling_models/cheat_sheet.svg @@ -1,28 +1,28 @@ - - relationship + + cheat_sheet - - - - - + + + + + - + - - - - - - - + + + + + + + @@ -30,96 +30,98 @@ - - - - - - - + + + + + + + - - - - - - + + + + + + + - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + - + - - + + - - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - + + + + + + + + + + + + - - - - + + + + + - - - - - - - - - - - + + + + + + + + + + + @@ -127,77 +129,78 @@ - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - - - - - - - + + + + + + + + + + + + - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + - - - - + + + + + - - - - - - - - - - - - - + + + + + + + + + + + + + @@ -205,173 +208,177 @@ - - - - - - - - - - - - - - - - + + + + + + + + + + - - - - - - - - - - + + + + + + + - - - - - - - + + + + + + + + + + + + + + + + - - - - - - - - + + + + + + + + + + + + + - - - - - - - - - - - - - + + + + + + + + - - - - + + + + + - - - - - - - - - - - - + + + + + + + + + + + + - - + + - - - - - - - - - - - - - - - + + + + + + + + + + + + + + - - - - - - - - - - - + + + + + + + + + + + - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - - - - - - - + + + + + + + + + + + - - - - + + + + + - - - - - - - - - - - - - + + + + + + + + + + + + + @@ -379,407 +386,411 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - + + + + + - - - - - - - - - - - - - + + + + + + + + + + + + + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - + + - - + + - - + + - - + + - - - + + - - + + + + + \ No newline at end of file diff --git a/docs/configuration/README.md b/docs/configuration/README.md index 85ae642ba6..9a1c3c9c0f 100644 --- a/docs/configuration/README.md +++ b/docs/configuration/README.md @@ -4,6 +4,6 @@ This section lists the most common options for running vLLM. There are three main levels of configuration, from highest priority to lowest priority: -- [Request parameters](../serving/openai_compatible_server.md#completions-api) and [input arguments](../api/README.md#inference-parameters) +- [Request parameters](../serving/online_serving/openai_compatible_server.md#completions-api) and [input arguments](../api/README.md#inference-parameters) - [Engine arguments](./engine_args.md) - [Environment variables](./env_vars.md) diff --git a/docs/configuration/engine_args.md b/docs/configuration/engine_args.md index 1458947882..b619cbf3db 100644 --- a/docs/configuration/engine_args.md +++ b/docs/configuration/engine_args.md @@ -7,7 +7,7 @@ toc_depth: 3 Engine arguments control the behavior of the vLLM engine. - For [offline inference](../serving/offline_inference.md), they are part of the arguments to [LLM][vllm.LLM] class. -- For [online serving](../serving/openai_compatible_server.md), they are part of the arguments to `vllm serve`. +- For [online serving](../serving/online_serving/README.md), they are part of the arguments to `vllm serve`. The engine argument classes, [EngineArgs][vllm.engine.arg_utils.EngineArgs] and [AsyncEngineArgs][vllm.engine.arg_utils.AsyncEngineArgs], are a combination of the configuration classes defined in [vllm.config][]. Therefore, if you are interested in developer documentation, we recommend looking at these configuration classes as they are the source of truth for types, defaults and docstrings. diff --git a/docs/design/arch_overview.md b/docs/design/arch_overview.md index f8bc66d6d4..e419104bae 100644 --- a/docs/design/arch_overview.md +++ b/docs/design/arch_overview.md @@ -51,9 +51,9 @@ More API details can be found in the [Offline Inference](../api/README.md#offlin The code for the `LLM` class can be found in [vllm/entrypoints/llm.py](../../vllm/entrypoints/llm.py). -### OpenAI-Compatible API Server +### Online Serving -The second primary interface to vLLM is via its OpenAI-compatible API server. +The second primary interface to vLLM is via its online server. This server can be started using the `vllm serve` command. ```bash @@ -76,7 +76,7 @@ python -m vllm.entrypoints.openai.api_server --model That code can be found in [vllm/entrypoints/openai/api_server.py](../../vllm/entrypoints/openai/api_server.py). -More details on the API server can be found in the [OpenAI-Compatible Server](../serving/openai_compatible_server.md) document. +More details on the API server can be found in the [Online Serving](../serving/online_serving/README.md) document. ## V1 Process Architecture diff --git a/docs/features/structured_outputs.md b/docs/features/structured_outputs.md index fa39f7ae6e..a46138238d 100644 --- a/docs/features/structured_outputs.md +++ b/docs/features/structured_outputs.md @@ -29,7 +29,7 @@ The following parameters are supported, which must be added as extra parameters: - `grammar`: the output will follow the context free grammar. - `structural_tag`: Follow a JSON schema within a set of specified tags within the generated text. -You can see the complete list of supported parameters on the [OpenAI-Compatible Server](../serving/openai_compatible_server.md) page. +You can see the complete list of supported parameters on the [OpenAI-Compatible Server](../serving/online_serving/openai_compatible_server.md) page. Structured outputs are supported by default in the OpenAI-Compatible Server. You may choose to specify the backend to use by setting the diff --git a/docs/getting_started/installation/gpu.apple.inc.md b/docs/getting_started/installation/gpu.apple.inc.md index 30e6244e05..1a1d37b53b 100644 --- a/docs/getting_started/installation/gpu.apple.inc.md +++ b/docs/getting_started/installation/gpu.apple.inc.md @@ -92,7 +92,7 @@ response = client.chat.completions.create( print(response.choices[0].message.content) ``` -For more details on the `vllm` CLI commands, see the [OpenAI-compatible server documentation](../../serving/openai_compatible_server.md). +For more details on the `vllm` CLI commands, see the [OpenAI-compatible server documentation](../../serving/online_serving/openai_compatible_server.md). --8<-- [end:set-up-using-python] --8<-- [start:pre-built-wheels] diff --git a/docs/getting_started/quickstart.md b/docs/getting_started/quickstart.md index a748ba4a93..6b2a1cf414 100644 --- a/docs/getting_started/quickstart.md +++ b/docs/getting_started/quickstart.md @@ -3,7 +3,7 @@ This guide will help you quickly get started with vLLM to perform: - [Offline batched inference](#offline-batched-inference) -- [Online serving using OpenAI-compatible server](#openai-compatible-server) +- [Online serving](#online-serving) ## Prerequisites @@ -182,7 +182,7 @@ for output in outputs: print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}") ``` -## OpenAI-Compatible Server +## Online Serving vLLM can be deployed as a server that implements the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API. By default, it starts the server at `http://localhost:8000`. You can specify the address with `--host` and `--port` arguments. The server currently hosts one model at a time and implements endpoints such as [list models](https://platform.openai.com/docs/api-reference/models/list), [create chat completion](https://platform.openai.com/docs/api-reference/chat/completions/create), and [create completion](https://platform.openai.com/docs/api-reference/completions/create) endpoints. @@ -195,7 +195,7 @@ vllm serve Qwen/Qwen2.5-1.5B-Instruct !!! note By default, the server uses a predefined chat template stored in the tokenizer. - You can learn about overriding it [here](../serving/openai_compatible_server.md#chat-template). + You can learn about overriding it [here](../serving/online_serving/README.md#chat-template). !!! important By default, the server applies `generation_config.json` from the huggingface model repository if it exists. This means the default values of certain sampling parameters can be overridden by those recommended by the model creator. diff --git a/docs/models/generative_models.md b/docs/models/generative_models.md index 76dba59771..46f8f4096d 100644 --- a/docs/models/generative_models.md +++ b/docs/models/generative_models.md @@ -138,7 +138,7 @@ outputs = llm.chat(conversation, chat_template=custom_template) ## Online Serving -Our [OpenAI-Compatible Server](../serving/openai_compatible_server.md) provides endpoints that correspond to the offline APIs: +Our [OpenAI-Compatible Server](../serving/online_serving/openai_compatible_server.md) provides endpoints that correspond to the offline APIs: -- [Completions API](../serving/openai_compatible_server.md#completions-api) is similar to `LLM.generate` but only accepts text. -- [Chat API](../serving/openai_compatible_server.md#chat-api) is similar to `LLM.chat`, accepting both text and [multi-modal inputs](../features/multimodal_inputs.md) for models with a chat template. +- [Completions API](../serving/online_serving/openai_compatible_server.md#completions-api) is similar to `LLM.generate` but only accepts text. +- [Chat API](../serving/online_serving/openai_compatible_server.md#chat-api) is similar to `LLM.chat`, accepting both text and [multi-modal inputs](../features/multimodal_inputs.md) for models with a chat template. diff --git a/docs/models/pooling_models/README.md b/docs/models/pooling_models/README.md index 7fdf505d5e..2a5357e4fe 100644 --- a/docs/models/pooling_models/README.md +++ b/docs/models/pooling_models/README.md @@ -141,24 +141,24 @@ enabling the corresponding APIs. ### `LLM.classify` -The [classify][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.classify] method outputs a probability vector for each prompt. +The [classify][vllm.LLM.classify] method outputs a probability vector for each prompt. It is primarily designed for [classification models](classify.md). For more information about `LLM.embed`, see [this page](classify.md#offline-inference). ### `LLM.embed` -The [embed][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.embed] method outputs an embedding vector for each prompt. +The [embed][vllm.LLM.embed] method outputs an embedding vector for each prompt. It is primarily designed for [embedding models](embed.md). For more information about `LLM.embed`, see [this page](embed.md#offline-inference). ### `LLM.score` -The [score][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.score] method outputs similarity scores between sentence pairs. +The [score][vllm.LLM.score] method outputs similarity scores between sentence pairs. It is primarily designed for [score models](scoring.md). ### `LLM.encode` -The [encode][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.encode] method is available to all pooling models in vLLM. +The [encode][vllm.LLM.encode] method is available to all pooling models in vLLM. Please use one of the more specific methods or set the task directly when using `LLM.encode`, refer to the [table above](#offline-apis-corresponding-to-pooling-usages). @@ -180,12 +180,12 @@ Our online Server provides endpoints that correspond to the offline APIs: - Corresponding to `LLM.embed`: - [Cohere Embed API](embed.md#cohere-embed-api) (`/v2/embed`) - - [Openai-compatible Embeddings API](embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`) + - [OpenAI-compatible Embeddings API](embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`) - Corresponding to `LLM.classify`: - [Classification API](classify.md#online-serving)(`/classify`) - Corresponding to `LLM.score`: - [Score API](scoring.md#score-api)(`/score`) - - [Rerank API](scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`) + - [Cohere Rerank API](scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`) - Pooling API (`/pooling`) is similar to `LLM.encode`, being applicable to all types of pooling models. The following introduces the Pooling API. For other APIs, please refer to the link above. diff --git a/docs/models/pooling_models/embed.md b/docs/models/pooling_models/embed.md index 34cd3c0c2a..6f78be5e23 100644 --- a/docs/models/pooling_models/embed.md +++ b/docs/models/pooling_models/embed.md @@ -12,7 +12,7 @@ Embedding models are a class of machine learning models designed to transform un - `LLM.score(...)` - Online APIs: - [Cohere Embed API](embed.md#cohere-embed-api) (`/v2/embed`) - - [Openai-compatible Embeddings API](embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`) + - [OpenAI-compatible Embeddings API](embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`) - Pooling API (`/pooling`) The primary distinction between (sequence) embedding and token embedding lies in their output granularity: (sequence) embedding produces a single embedding vector for an entire input sequence, whereas token embedding generates an embedding for each individual token within the sequence. @@ -228,7 +228,7 @@ these extra parameters are supported instead: #### Examples -If the model has a [chat template](../../serving/openai_compatible_server.md#chat-template), you can replace `inputs` with a list of `messages` (same schema as [Chat API](../../serving/openai_compatible_server.md#chat-api)) +If the model has a [chat template](../../serving/online_serving/README.md#chat-template), you can replace `inputs` with a list of `messages` (same schema as [Chat API](../../serving/online_serving/openai_compatible_server.md#chat-api)) which will be treated as a single prompt to the model. Here is a convenience function for calling the API while retaining OpenAI's type annotations: ??? code diff --git a/docs/models/pooling_models/reward.md b/docs/models/pooling_models/reward.md index 79ab716cb7..4acacda500 100644 --- a/docs/models/pooling_models/reward.md +++ b/docs/models/pooling_models/reward.md @@ -133,7 +133,7 @@ print(f"Data: {data!r}") ## Online Serving -Please refer to the [pooling API](README.md#pooling-api). Pooling task corresponding to reward model types refer to the [table above](#summary). +Please refer to the [Pooling API](README.md#pooling-api). Pooling task corresponding to reward model types refer to the [table above](#summary). ## More examples diff --git a/docs/models/pooling_models/scoring.md b/docs/models/pooling_models/scoring.md index 75744c2f56..76e41d00cb 100644 --- a/docs/models/pooling_models/scoring.md +++ b/docs/models/pooling_models/scoring.md @@ -20,7 +20,7 @@ The score models is designed to compute similarity scores between two input prom - `LLM.score` - Online APIs: - [Score API](scoring.md#score-api) (`/score`) - - [Rerank API](scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`) + - [Cohere Rerank API](scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`) !!! note Only when a classification model outputs num_labels equal to 1 can it be used as a scoring model and have its scoring API enabled. @@ -363,7 +363,7 @@ Full example: - [examples/pooling/score/vision_score_api_online.py](../../../examples/pooling/score/vision_score_api_online.py) - [examples/pooling/score/vision_rerank_api_online.py](../../../examples/pooling/score/vision_rerank_api_online.py) -### Rerank API +### Cohere Rerank API `/rerank`, `/v1/rerank`, and `/v2/rerank` APIs are compatible with both [Jina AI's rerank API interface](https://jina.ai/reranker/) and [Cohere's rerank API interface](https://docs.cohere.com/v2/reference/rerank) to ensure compatibility with @@ -446,7 +446,7 @@ AS cross-encoder models are a subset of classification models that accept two pr Score templates are supported for **cross-encoder** models only. If you are using an **embedding** model for scoring, vLLM does not apply a score template. -Some scoring models require a specific prompt format to work correctly. You can specify a custom score template using the `--chat-template` parameter (see [Chat Template](../../serving/openai_compatible_server.md#chat-template)). +Some scoring models require a specific prompt format to work correctly. You can specify a custom score template using the `--chat-template` parameter (see [Chat Template](../../serving/online_serving/README.md#chat-template)). Like chat templates, the score template receives a `messages` list. For scoring, each message has a `role` attribute—either `"query"` or `"document"`. For the usual kind of point-wise cross-encoder, you can expect exactly two messages: one query and one document. To access the query and document content, use Jinja's `selectattr` filter: diff --git a/docs/models/pooling_models/specific_models.md b/docs/models/pooling_models/specific_models.md index 0d908c1aa1..8753f1fd07 100644 --- a/docs/models/pooling_models/specific_models.md +++ b/docs/models/pooling_models/specific_models.md @@ -60,7 +60,7 @@ curl -s http://localhost:8000/score -H "Content-Type: application/json" -d '{ }' ``` -You can also get the raw token embeddings using the pooling API with `token_embed` task: +You can also get the raw token embeddings using the Pooling API with `token_embed` task: ```shell curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{ @@ -174,7 +174,7 @@ curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{ }' ``` -For **image inputs** via the pooling API, use the chat-style `messages` field: +For **image inputs** via the Pooling API, use the chat-style `messages` field: ```shell curl -s http://localhost:8000/pooling -H "Content-Type: application/json" -d '{ diff --git a/docs/models/pooling_models/token_classify.md b/docs/models/pooling_models/token_classify.md index 1b2902cc80..40c8aa1eab 100644 --- a/docs/models/pooling_models/token_classify.md +++ b/docs/models/pooling_models/token_classify.md @@ -103,7 +103,7 @@ print(f"Data: {data!r}") ## Online Serving -Please refer to the [pooling API](README.md#pooling-api) and use `"task":"token_classify"`. +Please refer to the [Pooling API](README.md#pooling-api) and use `"task":"token_classify"`. ## More examples diff --git a/docs/models/pooling_models/token_embed.md b/docs/models/pooling_models/token_embed.md index 23625e2853..ba44ad3ca3 100644 --- a/docs/models/pooling_models/token_embed.md +++ b/docs/models/pooling_models/token_embed.md @@ -129,7 +129,7 @@ print(f"Score: {score}") ## Online Serving -Please refer to the [pooling API](README.md#pooling-api) and use `"task":"token_embed"`. +Please refer to the [Pooling API](README.md#pooling-api) and use `"task":"token_embed"`. ## More examples diff --git a/docs/models/supported_models.md b/docs/models/supported_models.md index d9df96bea4..e2aaae95f9 100644 --- a/docs/models/supported_models.md +++ b/docs/models/supported_models.md @@ -44,7 +44,7 @@ llm.apply_model(lambda model: print(type(model))) If the printed type starts with `Transformers...` then it's using the Transformers model implementation! -If a model has a vLLM implementation but you would prefer to use the Transformers implementation via the Transformers modeling backend, set `model_impl="transformers"` for [offline inference](../serving/offline_inference.md) or `--model-impl transformers` for the [online serving](../serving/openai_compatible_server.md). +If a model has a vLLM implementation but you would prefer to use the Transformers implementation via the Transformers modeling backend, set `model_impl="transformers"` for [offline inference](../serving/offline_inference.md) or `--model-impl transformers` for the [online serving](../serving/online_serving/README.md). !!! note For vision-language models, if you are loading with `dtype="auto"`, vLLM loads the whole model with config's `dtype` if it exists. In contrast the native Transformers will respect the `dtype` attribute of each backbone in the model. That might cause a slight difference in performance. @@ -63,8 +63,8 @@ For a model to be compatible with the Transformers modeling backend for vLLM it If the compatible model is: -- on the Hugging Face Model Hub, simply set `trust_remote_code=True` for [offline-inference](../serving/offline_inference.md) or `--trust-remote-code` for the [openai-compatible-server](../serving/openai_compatible_server.md). -- in a local directory, simply pass directory path to `model=` for [offline-inference](../serving/offline_inference.md) or `vllm serve ` for the [openai-compatible-server](../serving/openai_compatible_server.md). +- on the Hugging Face Model Hub, simply set `trust_remote_code=True` for [offline-inference](../serving/offline_inference.md) or `--trust-remote-code` for the [online serving](../serving/online_serving/README.md). +- in a local directory, simply pass directory path to `model=` for [offline-inference](../serving/offline_inference.md) or `vllm serve ` for the [online serving](../serving/online_serving/README.md). This means that, with the Transformers modeling backend for vLLM, new models can be used before they are officially supported in Transformers or vLLM! @@ -709,7 +709,7 @@ Speech2Text models trained specifically for Automatic Speech Recognition. #### Realtime Transcription Speech models that support streaming transcription via the -[`/v1/realtime`](../serving/openai_compatible_server.md#realtime-api) +[`/v1/realtime`](../serving/online_serving/speech_to_text.md#realtime-api) WebSocket endpoint. | Architecture | Models | Example HF Models | [LoRA](../features/lora.md) | [PP](../serving/parallelism_scaling.md) | diff --git a/docs/serving/online_serving/README.md b/docs/serving/online_serving/README.md new file mode 100644 index 0000000000..c843770444 --- /dev/null +++ b/docs/serving/online_serving/README.md @@ -0,0 +1,182 @@ +# Online Serving + +vLLM provides an HTTP server that is compatible with many interfaces! + +## OpenAI-Compatible Server + +We currently support the following OpenAI APIs: + +- [Completions API](./openai_compatible_server.md#completions-api) (`/v1/completions`) + - Only applicable to [text generation models](../../models/generative_models.md). + - *Note: `suffix` parameter is not supported.* +- [Responses API](./openai_compatible_server.md#responses-api) (`/v1/responses`) + - Only applicable to [text generation models](../../models/generative_models.md). +- [Chat Completions API](./openai_compatible_server.md#chat-api) (`/v1/chat/completions`) + - Only applicable to [text generation models](../../models/generative_models.md) with a [chat template](./openai_compatible_server.md#chat-template). + - *Note: `user` parameter is ignored.* + - *Note:* Setting the `parallel_tool_calls` parameter to `false` ensures vLLM only returns zero or one tool call per request. Setting it to `true` (the default) allows returning more than one tool call per request. There is no guarantee more than one tool call will be returned if this is set to `true`, as that behavior is model dependent and not all models are designed to support parallel tool calls. +- [Embeddings API](../../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`) + - Only applicable to [embedding models](../../models/pooling_models/embed.md). +- [Transcriptions API](./speech_to_text.md#transcriptions-api) (`/v1/audio/transcriptions`) + - Only applicable to [Automatic Speech Recognition (ASR) models](../../models/supported_models.md#transcription). +- [Translation API](./speech_to_text.md#translations-api) (`/v1/audio/translations`) + - Only applicable to [Automatic Speech Recognition (ASR) models](../../models/supported_models.md#transcription). + +## Anthropic APIs + +- Anthropic messages API (`/v1/messages`) + +## Cohere APIs + +- [Cohere Embed API](../../models/pooling_models/embed.md#cohere-embed-api) (`/v2/embed`) + - Compatible with [Cohere's Embed API](https://docs.cohere.com/reference/embed) + - Works with any [embedding model](../../models/pooling_models/embed.md#supported-models), including multimodal models. +- [Cohere Rerank API](../../models/pooling_models/scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`) + - Implements [Jina AI's v1 rerank API](https://jina.ai/reranker/) + - compatible with [Cohere's v1 & v2 rerank APIs](https://docs.cohere.com/v2/reference/rerank) + +## SageMaker APIs + +- `/invocations` - SageMaker-compatible endpoint (routes to the same inference functions as `/v1` endpoints) + +## Pooling APIs + +For further details on pooling models, please refer to [this page](../../models/pooling_models/README.md). + +- [Classification Usages](../../models/pooling_models/classify.md) + - [Classification API](../../models/pooling_models/classify.md#online-serving) (`/classify`) + - Only applicable to [classification models](../../models/pooling_models/classify.md). +- [Embedding Usages](../../models/pooling_models/embed.md) + - [Cohere Embed API](../../models/pooling_models/embed.md#cohere-embed-api) (`/v2/embed`) + - [OpenAI-compatible Embeddings API](../../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`) + - Only applicable to [embedding models](../../models/pooling_models/embed.md). +- [Scoring Usages](../../models/pooling_models/scoring.md) + - [Score API](../../models/pooling_models/scoring.md#score-api) (`/score`) + - [Cohere Rerank API](../../models/pooling_models/scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`) + - Applicable to [score models](../../models/pooling_models/scoring.md) (cross-encoder, bi-encoder, late-interaction). +- [Pooling API](../../models/pooling_models/README.md#pooling-api) (`/pooling`) + - Applicable to all [pooling models](../../models/pooling_models/README.md). + +## Speech to Text APIs + +For further details on speech to text, please refer to [this page](speech_to_text.md). + +- [Transcriptions API](./speech_to_text.md#transcriptions-api) (`/v1/audio/transcriptions`) + - Only applicable to [Automatic Speech Recognition (ASR) models](../../models/supported_models.md#transcription). +- [Translation API](./speech_to_text.md#translations-api) (`/v1/audio/translations`) + - Only applicable to [Automatic Speech Recognition (ASR) models](../../models/supported_models.md#transcription). +- [Realtime API](./speech_to_text.md#realtime-api) (`/v1/realtime`) + - Only applicable to [Automatic Speech Recognition (ASR) models](../../models/supported_models.md#realtime-transcription). + +## Disaggregated APIs + +### Renderer APIs + +For further details on renderer APIs, please refer to [this page](renderer.md). + +- [Completions Render API](renderer.md) (`/v1/completions/render`) + - Render completion requests +- [Chat Completions Render API](renderer.md) (`/v1/chat/completions/render`) + - Render chat completions + +## Custom APIs + +- [Classification API](../../models/pooling_models/classify.md#classification-api) (`/classify`) + - Only applicable to [classification models](../../models/pooling_models/classify.md). +- [Score API](../../models/pooling_models/scoring.md#score-api) (`/score`, `/v1/score`) + - Applicable to [score models](../../models/pooling_models/scoring.md) (cross-encoder, bi-encoder, late-interaction). +- [Pooling API](../../models/pooling_models/README.md#pooling-api) (`/pooling`) + - Applicable to all [pooling models](../../models/pooling_models/README.md). +- [Generative Scoring API](generative_scoring.md#generative-scoring-api) (`/generative_scoring`) + - Applicable to [CausalLM models](../../models/generative_models.md) (task `"generate"`). + - Computes next-token probabilities for specified `label_token_ids`. + +## Utility APIs + +- `/tokenize` - Tokenize text +- `/detokenize` - Detokenize tokens +- `/health` - Health check +- `/ping` - SageMaker health check +- `/version` - Version information +- `/load` - Server load metrics + +## Sleep Mode APIs + +For further details on sleep mode, please refer to [this page](../../features/sleep_mode.md). + +- `/sleep` - Put engine to sleep (causes denial of service) +- `/wake_up` - Wake engine from sleep +- `/is_sleeping` - Check if engine is sleeping +- `/collective_rpc` - Execute arbitrary RPC methods on the engine (extremely dangerous) + +## Chat Template + +In order for the language model to support chat protocol, vLLM requires the model to include +a chat template in its tokenizer configuration. The chat template is a Jinja2 template that +specifies how roles, messages, and other chat-specific tokens are encoded in the input. + +An example chat template for `NousResearch/Meta-Llama-3-8B-Instruct` can be found [here](https://llama.com/docs/model-cards-and-prompt-formats/meta-llama-3/#prompt-template-for-meta-llama-3) + +Some models do not provide a chat template even though they are instruction/chat fine-tuned. For those models, +you can manually specify their chat template in the `--chat-template` parameter with the file path to the chat +template, or the template in string form. Without a chat template, the server will not be able to process chat +and all chat requests will error. + +```bash +vllm serve --chat-template ./path-to-chat-template.jinja +``` + +vLLM community provides a set of chat templates for popular models. You can find them under the [examples](../../../examples) directory. + +With the inclusion of multi-modal chat APIs, the OpenAI spec now accepts chat messages in a new format which specifies +both a `type` and a `text` field. An example is provided below: + +```python +completion = client.chat.completions.create( + model="NousResearch/Meta-Llama-3-8B-Instruct", + messages=[ + { + "role": "user", + "content": [ + {"type": "text", "text": "Classify this sentiment: vLLM is wonderful!"}, + ], + }, + ], +) +``` + +Most chat templates for LLMs expect the `content` field to be a string, but there are some newer models like +`meta-llama/Llama-Guard-3-1B` that expect the content to be formatted according to the OpenAI schema in the +request. vLLM provides best-effort support to detect this automatically, which is logged as a string like +*"Detected the chat template content format to be..."*, and internally converts incoming requests to match +the detected format, which can be one of: + +- `"string"`: A string. + - Example: `"Hello world"` +- `"openai"`: A list of dictionaries, similar to OpenAI schema. + - Example: `[{"type": "text", "text": "Hello world!"}]` + +If the result is not what you expect, you can set the `--chat-template-content-format` CLI argument +to override which format to use. + +## Offline API Documentation + +The FastAPI `/docs` endpoint requires an internet connection by default. To enable offline access in air-gapped environments, use the `--enable-offline-docs` flag: + +```bash +vllm serve NousResearch/Meta-Llama-3-8B-Instruct --enable-offline-docs +``` + +## Ray Serve LLM + +Ray Serve LLM enables scalable, production-grade serving of the vLLM engine. It integrates tightly with vLLM and extends it with features such as auto-scaling, load balancing, and back-pressure. + +Key capabilities: + +- Exposes an OpenAI-compatible HTTP API as well as a Pythonic API. +- Scales from a single GPU to a multi-node cluster without code changes. +- Provides observability and autoscaling policies through Ray dashboards and metrics. + +The following example shows how to deploy a large model like DeepSeek R1 with Ray Serve LLM: [examples/ray_serving/ray_serve_deepseek.py](../../../examples/ray_serving/ray_serve_deepseek.py). + +Learn more about Ray Serve LLM with the official [Ray Serve LLM documentation](https://docs.ray.io/en/latest/serve/llm/index.html). diff --git a/docs/serving/online_serving/generative_scoring.md b/docs/serving/online_serving/generative_scoring.md new file mode 100644 index 0000000000..d0c578f210 --- /dev/null +++ b/docs/serving/online_serving/generative_scoring.md @@ -0,0 +1,64 @@ +# Generative Scoring + +The `/generative_scoring` endpoint uses a CausalLM model (e.g., Llama, Qwen, Mistral) to compute the probability of specified token IDs appearing as the next token. Each item (document) is concatenated with the query to form a prompt, and the model predicts how likely each label token is as the next token after that prompt. This lets you score items against a query — for example, asking "Is this the capital of France?" and scoring each city by how likely the model is to answer "Yes". + +This endpoint is automatically available when the server is started with a generative model (task `"generate"`). It is separate from the pooling-based [Score API](../../models/pooling_models/scoring.md#score-api), which uses cross-encoder, bi-encoder, or late-interaction models. + +**Requirements:** + +- The `label_token_ids` parameter is **required** and must contain **at least 1 token ID**. +- When 2 label tokens are provided, the score equals `P(label_token_ids[0]) / (P(label_token_ids[0]) + P(label_token_ids[1]))` (softmax over the two labels). +- When more labels are provided, the score is the softmax-normalized probability of the first label token across all label tokens. + +## How it works + +1. **Prompt Construction**: For each item, builds `prompt = query + item` (or `item + query` if `item_first=true`) +2. **Forward Pass**: Runs the model on each prompt to get next-token logits +3. **Probability Extraction**: Extracts logprobs for the specified `label_token_ids` +4. **Softmax Normalization**: Applies softmax over only the label tokens (when `apply_softmax=true`) +5. **Score**: Returns the normalized probability of the first label token + +## Finding Token IDs + +To find the token IDs for your labels, use the tokenizer: + +```python +from transformers import AutoTokenizer + +tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B") +yes_id = tokenizer.encode("Yes", add_special_tokens=False)[0] +no_id = tokenizer.encode("No", add_special_tokens=False)[0] +print(f"Yes: {yes_id}, No: {no_id}") +``` + +## Example + +```bash +curl -X POST http://localhost:8000/generative_scoring \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen/Qwen3-0.6B", + "query": "Is this city the capital of France?", + "items": ["Paris", "London", "Berlin"], + "label_token_ids": [9454, 2753] + }' +``` + +Here, each item is appended to the query to form prompts like `"Is this city the capital of France? Paris"`, `"... London"`, etc. The model then predicts the next token, and the score reflects the probability of "Yes" (token 9454) vs "No" (token 2753). + +??? console "Response" + + ```json + { + "id": "generative-scoring-abc123", + "object": "list", + "created": 1234567890, + "model": "Qwen/Qwen3-0.6B", + "data": [ + {"index": 0, "object": "score", "score": 0.95}, + {"index": 1, "object": "score", "score": 0.12}, + {"index": 2, "object": "score", "score": 0.08} + ], + "usage": {"prompt_tokens": 45, "total_tokens": 48, "completion_tokens": 3} + } + ``` diff --git a/docs/serving/online_serving/openai_compatible_server.md b/docs/serving/online_serving/openai_compatible_server.md new file mode 100644 index 0000000000..245de012bf --- /dev/null +++ b/docs/serving/online_serving/openai_compatible_server.md @@ -0,0 +1,193 @@ +# OpenAI-Compatible Server + +vLLM provides an HTTP server that implements OpenAI's [Completions API](https://platform.openai.com/docs/api-reference/completions), [Chat API](https://platform.openai.com/docs/api-reference/chat), and more! This functionality lets you serve models and interact with them using an HTTP client. + +## Supported APIs + +We currently support the following OpenAI APIs: + +- [Completions API](#completions-api) (`/v1/completions`) + - Only applicable to [text generation models](../../models/generative_models.md). + - *Note: `suffix` parameter is not supported.* +- [Responses API](#responses-api) (`/v1/responses`) + - Only applicable to [text generation models](../../models/generative_models.md). +- [Chat Completions API](#chat-api) (`/v1/chat/completions`) + - Only applicable to [text generation models](../../models/generative_models.md) with a [chat template](../online_serving/README.md#chat-template). + - *Note: `user` parameter is ignored.* + - *Note:* Setting the `parallel_tool_calls` parameter to `false` ensures vLLM only returns zero or one tool call per request. Setting it to `true` (the default) allows returning more than one tool call per request. There is no guarantee more than one tool call will be returned if this is set to `true`, as that behavior is model dependent and not all models are designed to support parallel tool calls. +- [Embeddings API](../../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`) + - Only applicable to [embedding models](../../models/pooling_models/embed.md). +- [Transcriptions API](./speech_to_text.md#transcriptions-api) (`/v1/audio/transcriptions`) + - Only applicable to [Automatic Speech Recognition (ASR) models](../../models/supported_models.md#transcription). +- [Translation API](./speech_to_text.md#translations-api) (`/v1/audio/translations`) + - Only applicable to [Automatic Speech Recognition (ASR) models](../../models/supported_models.md#transcription). + +## Completions API + +In your terminal, you can [install](../../getting_started/installation/README.md) vLLM, then start the server with the [`vllm serve`](../../configuration/serve_args.md) command. (You can also use our [Docker](../../deployment/docker.md) image.) + +```bash +vllm serve NousResearch/Meta-Llama-3-8B-Instruct \ + --dtype auto \ + --api-key token-abc123 +``` + +To call the server, in your preferred text editor, create a script that uses an HTTP client. Include any messages that you want to send to the model. Then run that script. Below is an example script using the [official OpenAI Python client](https://github.com/openai/openai-python). + +??? code + + ```python + from openai import OpenAI + client = OpenAI( + base_url="http://localhost:8000/v1", + api_key="token-abc123", + ) + + completion = client.chat.completions.create( + model="NousResearch/Meta-Llama-3-8B-Instruct", + messages=[ + {"role": "user", "content": "Hello!"}, + ], + ) + + print(completion.choices[0].message) + ``` + +!!! tip + vLLM supports some parameters that are not supported by OpenAI, `top_k` for example. + You can pass these parameters to vLLM using the OpenAI client in the `extra_body` parameter of your requests, i.e. `extra_body={"top_k": 50}` for `top_k`. + +!!! important + By default, the server applies `generation_config.json` from the Hugging Face model repository if it exists. This means the default values of certain sampling parameters can be overridden by those recommended by the model creator. + + To disable this behavior, please pass `--generation-config vllm` when launching the server. + +## Extra Parameters + +vLLM supports a set of parameters that are not part of the OpenAI API. +In order to use them, you can pass them as extra parameters in the OpenAI client. +Or directly merge them into the JSON payload if you are using HTTP call directly. + +```python +completion = client.chat.completions.create( + model="NousResearch/Meta-Llama-3-8B-Instruct", + messages=[ + {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}, + ], + extra_body={ + "structured_outputs": {"choice": ["positive", "negative"]}, + }, +) +``` + +## Extra HTTP Headers + +Only `X-Request-Id` HTTP request header is supported for now. It can be enabled +with `--enable-request-id-headers`. + +??? code + + ```python + completion = client.chat.completions.create( + model="NousResearch/Meta-Llama-3-8B-Instruct", + messages=[ + {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}, + ], + extra_headers={ + "x-request-id": "sentiment-classification-00001", + }, + ) + print(completion._request_id) + + completion = client.completions.create( + model="NousResearch/Meta-Llama-3-8B-Instruct", + prompt="A robot may not injure a human being", + extra_headers={ + "x-request-id": "completion-test", + }, + ) + print(completion._request_id) + ``` + +## API Reference + +### Completions API + +Our Completions API is compatible with [OpenAI's Completions API](https://platform.openai.com/docs/api-reference/completions); +you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it. + +Code example: [examples/basic/online_serving/openai_completion_client.py](../../../examples/basic/online_serving/openai_completion_client.py) + +#### Extra parameters + +The following [sampling parameters](../../api/README.md#inference-parameters) are supported. + +??? code + + ```python + --8<-- "vllm/entrypoints/openai/completion/protocol.py:completion-sampling-params" + ``` + +The following extra parameters are supported: + +??? code + + ```python + --8<-- "vllm/entrypoints/openai/completion/protocol.py:completion-extra-params" + ``` + +### Chat API + +Our Chat API is compatible with [OpenAI's Chat Completions API](https://platform.openai.com/docs/api-reference/chat); +you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it. + +We support both [Vision](https://platform.openai.com/docs/guides/vision)- and +[Audio](https://platform.openai.com/docs/guides/audio?audio-generation-quickstart-example=audio-in)-related parameters; +see our [Multimodal Inputs](../../features/multimodal_inputs.md) guide for more information. + +- *Note: `image_url.detail` parameter is not supported.* + +Code example: [examples/basic/online_serving/openai_chat_completion_client.py](../../../examples/basic/online_serving/openai_chat_completion_client.py) + +#### Extra parameters + +The following [sampling parameters](../../api/README.md#inference-parameters) are supported. + +??? code + + ```python + --8<-- "vllm/entrypoints/openai/chat_completion/protocol.py:chat-completion-sampling-params" + ``` + +The following extra parameters are supported: + +??? code + + ```python + --8<-- "vllm/entrypoints/openai/chat_completion/protocol.py:chat-completion-extra-params" + ``` + +### Responses API + +Our Responses API is compatible with [OpenAI's Responses API](https://platform.openai.com/docs/api-reference/responses); +you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it. + +Code example: [examples/tool_calling/openai_responses_client_with_tools.py](../../../examples/tool_calling/openai_responses_client_with_tools.py) + +#### Extra parameters + +The following extra parameters in the request object are supported: + +??? code + + ```python + --8<-- "vllm/entrypoints/openai/responses/protocol.py:responses-extra-params" + ``` + +The following extra parameters in the response object are supported: + +??? code + + ```python + --8<-- "vllm/entrypoints/openai/responses/protocol.py:responses-response-extra-params" + ``` diff --git a/docs/serving/online_serving/renderer.md b/docs/serving/online_serving/renderer.md new file mode 100644 index 0000000000..9ea2f369db --- /dev/null +++ b/docs/serving/online_serving/renderer.md @@ -0,0 +1,14 @@ +# Renderer APIs + +Our renderer API is designed to disaggregate the render phase(preprocessing) and enable a token-in / token-out API server. + +- GPU-less deployment of frontend: Allow preprocessing (tokenization, MM input processing) and postprocessing (detokenization, tool call parsing, reasoning parsing) to run without GPU. +- Disaggregated tokenization: Support use cases such as llm-d, Dynamo, and custom frontends that need to leverage vLLM's preprocessing logic without running the full inference engine. +- Tokens-in / tokens-out engine: Make the engine a pure token-in / token-out service, decoupled from request preprocessing. + +## API Reference + +- [Completions Render API](renderer.md) (`/v1/completions/render`) + - Render completion requests +- [Chat Completions Render API](renderer.md) (`/v1/chat/completions/render`) + - Render chat completions diff --git a/docs/serving/online_serving/speech_to_text.md b/docs/serving/online_serving/speech_to_text.md new file mode 100644 index 0000000000..d503923c4f --- /dev/null +++ b/docs/serving/online_serving/speech_to_text.md @@ -0,0 +1,189 @@ +# Speech to Text APIs + +## Transcriptions API + +Our Transcriptions API is compatible with [OpenAI's Transcriptions API](https://platform.openai.com/docs/api-reference/audio/createTranscription); +you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it. + +!!! note + To use the Transcriptions API, please install with extra audio dependencies using `pip install vllm[audio]`. + +Code example: [examples/speech_to_text/openai/openai_transcription_client.py](../../../examples/speech_to_text/openai/openai_transcription_client.py) + +NOTE: beam search is currently supported in the transcriptions endpoint for encoder-decoder multimodal models, e.g., whisper, but highly inefficient as work for handling the encoder/decoder cache is actively ongoing. This is an active point of ongoing optimization and will be handled properly in the very near future. + +### API Enforced Limits + +Set the maximum audio file size (in MB) that VLLM will accept, via the +`VLLM_MAX_AUDIO_CLIP_FILESIZE_MB` environment variable. Default is 25 MB. + +### Uploading Audio Files + +The Transcriptions API supports uploading audio files in various formats including FLAC, MP3, MP4, MPEG, MPGA, M4A, OGG, WAV, and WEBM. + +**Using OpenAI Python Client:** + +??? code + + ```python + from openai import OpenAI + + client = OpenAI( + base_url="http://localhost:8000/v1", + api_key="token-abc123", + ) + + # Upload audio file from disk + with open("audio.mp3", "rb") as audio_file: + transcription = client.audio.transcriptions.create( + model="openai/whisper-large-v3-turbo", + file=audio_file, + language="en", + response_format="verbose_json", + ) + + print(transcription.text) + ``` + +**Using curl with multipart/form-data:** + +??? code + + ```bash + curl -X POST "http://localhost:8000/v1/audio/transcriptions" \ + -H "Authorization: Bearer token-abc123" \ + -F "file=@audio.mp3" \ + -F "model=openai/whisper-large-v3-turbo" \ + -F "language=en" \ + -F "response_format=verbose_json" + ``` + +**Supported Parameters:** + +- `file`: The audio file to transcribe (required) +- `model`: The model to use for transcription (required) +- `language`: The language code (e.g., "en", "zh") (optional) +- `prompt`: Optional text to guide the transcription style (optional) +- `response_format`: Format of the response ("json", "text") (optional) +- `temperature`: Sampling temperature between 0 and 1 (optional) + +For the complete list of supported parameters including sampling parameters and vLLM extensions, see the [protocol definitions](https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/openai/protocol.py#L2182). + +**Response Format:** + +For `verbose_json` response format: + +??? code + + ```json + { + "text": "Hello, this is a transcription of the audio file.", + "language": "en", + "duration": 5.42, + "segments": [ + { + "id": 0, + "seek": 0, + "start": 0.0, + "end": 2.5, + "text": "Hello, this is a transcription", + "tokens": [50364, 938, 428, 307, 275, 28347], + "temperature": 0.0, + "avg_logprob": -0.245, + "compression_ratio": 1.235, + "no_speech_prob": 0.012 + } + ] + } + ``` +Currently “verbose_json” response format doesn’t support no_speech_prob. + +### Extra Parameters + +The following [sampling parameters](../../api/README.md#inference-parameters) are supported. + +??? code + + ```python + --8<-- "vllm/entrypoints/speech_to_text/transcription/protocol.py:transcription-sampling-params" + ``` + +The following extra parameters are supported: + +??? code + + ```python + --8<-- "vllm/entrypoints/speech_to_text/transcription/protocol.py:transcription-extra-params" + ``` + +## Translations API + +Our Translation API is compatible with [OpenAI's Translations API](https://platform.openai.com/docs/api-reference/audio/createTranslation); +you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it. +Whisper models can translate audio from one of the 55 non-English supported languages into English. +Please mind that the popular `openai/whisper-large-v3-turbo` model does not support translating. + +!!! note + To use the Translation API, please install with extra audio dependencies using `pip install vllm[audio]`. + +Code example: [examples/speech_to_text/openai/openai_translation_client.py](../../../examples/speech_to_text/openai/openai_translation_client.py) + +### Extra Parameters + +The following [sampling parameters](../../api/README.md#inference-parameters) are supported. + +```python +--8<-- "vllm/entrypoints/speech_to_text/translation/protocol.py:translation-sampling-params" +``` + +The following extra parameters are supported: + +```python +--8<-- "vllm/entrypoints/speech_to_text/translation/protocol.py:translation-extra-params" +``` + +## Realtime API + +The Realtime API provides WebSocket-based streaming audio transcription, allowing real-time speech-to-text as audio is being recorded. + +!!! note + To use the Realtime API, please install with extra audio dependencies using `uv pip install vllm[audio]`. + +### Audio Format + +Audio must be sent as base64-encoded PCM16 audio at 16kHz sample rate, mono channel. + +### Protocol Overview + +1. Client connects to `ws://host/v1/realtime` +2. Server sends `session.created` event +3. Client optionally sends `session.update` with model/params +4. Client sends `input_audio_buffer.commit` when ready +5. Client sends `input_audio_buffer.append` events with base64 PCM16 chunks +6. Server sends `transcription.delta` events with incremental text +7. Server sends `transcription.done` with final text + usage +8. Repeat from step 5 for next utterance +9. Optionally, client sends input_audio_buffer.commit with final=True + to signal audio input is finished. Useful when streaming audio files + +### Client → Server Events + +| Event | Description | +| ----- | ----------- | +| `input_audio_buffer.append` | Send base64-encoded audio chunk: `{"type": "input_audio_buffer.append", "audio": ""}` | +| `input_audio_buffer.commit` | Trigger transcription processing or end: `{"type": "input_audio_buffer.commit", "final": bool}` | +| `session.update` | Configure session: `{"type": "session.update", "model": "model-name"}` | + +### Server → Client Events + +| Event | Description | +| ----- | ----------- | +| `session.created` | Connection established with session ID and timestamp | +| `transcription.delta` | Incremental transcription text: `{"type": "transcription.delta", "delta": "text"}` | +| `transcription.done` | Final transcription with usage stats | +| `error` | Error notification with message and optional code | + +#### Example Clients + +- [openai_realtime_client.py](https://github.com/vllm-project/vllm/tree/main/examples/speech_to_text/realtime/openai_realtime_client.py) - Upload and transcribe an audio file +- [openai_realtime_microphone_client.py](https://github.com/vllm-project/vllm/tree/main/examples/speech_to_text/realtime/openai_realtime_microphone_client.py) - Gradio demo for live microphone transcription diff --git a/docs/serving/openai_compatible_server.md b/docs/serving/openai_compatible_server.md deleted file mode 100644 index 973c2ecdcb..0000000000 --- a/docs/serving/openai_compatible_server.md +++ /dev/null @@ -1,547 +0,0 @@ -# OpenAI-Compatible Server - -vLLM provides an HTTP server that implements OpenAI's [Completions API](https://platform.openai.com/docs/api-reference/completions), [Chat API](https://platform.openai.com/docs/api-reference/chat), and more! This functionality lets you serve models and interact with them using an HTTP client. - -In your terminal, you can [install](../getting_started/installation/README.md) vLLM, then start the server with the [`vllm serve`](../configuration/serve_args.md) command. (You can also use our [Docker](../deployment/docker.md) image.) - -```bash -vllm serve NousResearch/Meta-Llama-3-8B-Instruct \ - --dtype auto \ - --api-key token-abc123 -``` - -To call the server, in your preferred text editor, create a script that uses an HTTP client. Include any messages that you want to send to the model. Then run that script. Below is an example script using the [official OpenAI Python client](https://github.com/openai/openai-python). - -??? code - - ```python - from openai import OpenAI - client = OpenAI( - base_url="http://localhost:8000/v1", - api_key="token-abc123", - ) - - completion = client.chat.completions.create( - model="NousResearch/Meta-Llama-3-8B-Instruct", - messages=[ - {"role": "user", "content": "Hello!"}, - ], - ) - - print(completion.choices[0].message) - ``` - -!!! tip - vLLM supports some parameters that are not supported by OpenAI, `top_k` for example. - You can pass these parameters to vLLM using the OpenAI client in the `extra_body` parameter of your requests, i.e. `extra_body={"top_k": 50}` for `top_k`. - -!!! important - By default, the server applies `generation_config.json` from the Hugging Face model repository if it exists. This means the default values of certain sampling parameters can be overridden by those recommended by the model creator. - - To disable this behavior, please pass `--generation-config vllm` when launching the server. - -## Supported APIs - -We currently support the following OpenAI APIs: - -- [Completions API](#completions-api) (`/v1/completions`) - - Only applicable to [text generation models](../models/generative_models.md). - - *Note: `suffix` parameter is not supported.* -- [Responses API](#responses-api) (`/v1/responses`) - - Only applicable to [text generation models](../models/generative_models.md). -- [Chat Completions API](#chat-api) (`/v1/chat/completions`) - - Only applicable to [text generation models](../models/generative_models.md) with a [chat template](../serving/openai_compatible_server.md#chat-template). - - *Note: `user` parameter is ignored.* - - *Note:* Setting the `parallel_tool_calls` parameter to `false` ensures vLLM only returns zero or one tool call per request. Setting it to `true` (the default) allows returning more than one tool call per request. There is no guarantee more than one tool call will be returned if this is set to `true`, as that behavior is model dependent and not all models are designed to support parallel tool calls. -- [Embeddings API](../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`) - - Only applicable to [embedding models](../models/pooling_models/embed.md). -- [Transcriptions API](#transcriptions-api) (`/v1/audio/transcriptions`) - - Only applicable to [Automatic Speech Recognition (ASR) models](../models/supported_models.md#transcription). -- [Translation API](#translations-api) (`/v1/audio/translations`) - - Only applicable to [Automatic Speech Recognition (ASR) models](../models/supported_models.md#transcription). -- [Realtime API](#realtime-api) (`/v1/realtime`) - - Only applicable to [Automatic Speech Recognition (ASR) models](../models/supported_models.md#realtime-transcription). - -In addition, we have the following custom APIs: - -- [Tokenizer API](#tokenizer-api) (`/tokenize`, `/detokenize`) - - Applicable to any model with a tokenizer. -- [pooling API](../models/pooling_models/README.md#pooling-api) (`/pooling`) - - Applicable to all [pooling models](../models/pooling_models/README.md). -- [Classification API](../models/pooling_models/classify.md#classification-api) (`/classify`) - - Only applicable to [classification models](../models/pooling_models/classify.md). -- [Cohere Embed API](../models/pooling_models/embed.md#cohere-embed-api) (`/v2/embed`) - - Compatible with [Cohere's Embed API](https://docs.cohere.com/reference/embed) - - Works with any [embedding model](../models/pooling_models/embed.md#supported-models), including multimodal models. -- [Score API](../models/pooling_models/scoring.md#score-api) (`/score`, `/v1/score`) - - Applicable to [score models](../models/pooling_models/scoring.md) (cross-encoder, bi-encoder, late-interaction). -- [Generative Scoring API](#generative-scoring-api) (`/generative_scoring`) - - Applicable to [CausalLM models](../models/generative_models.md) (task `"generate"`). - - Computes next-token probabilities for specified `label_token_ids`. -- [Rerank API](../models/pooling_models/scoring.md#rerank-api) (`/rerank`, `/v1/rerank`, `/v2/rerank`) - - Implements [Jina AI's v1 rerank API](https://jina.ai/reranker/) - - Also compatible with [Cohere's v1 & v2 rerank APIs](https://docs.cohere.com/v2/reference/rerank) - - Jina and Cohere's APIs are very similar; Jina's includes extra information in the rerank endpoint's response. - -## Chat Template - -In order for the language model to support chat protocol, vLLM requires the model to include -a chat template in its tokenizer configuration. The chat template is a Jinja2 template that -specifies how roles, messages, and other chat-specific tokens are encoded in the input. - -An example chat template for `NousResearch/Meta-Llama-3-8B-Instruct` can be found [here](https://llama.com/docs/model-cards-and-prompt-formats/meta-llama-3/#prompt-template-for-meta-llama-3) - -Some models do not provide a chat template even though they are instruction/chat fine-tuned. For those models, -you can manually specify their chat template in the `--chat-template` parameter with the file path to the chat -template, or the template in string form. Without a chat template, the server will not be able to process chat -and all chat requests will error. - -```bash -vllm serve --chat-template ./path-to-chat-template.jinja -``` - -vLLM community provides a set of chat templates for popular models. You can find them under the [examples](../../examples) directory. - -With the inclusion of multi-modal chat APIs, the OpenAI spec now accepts chat messages in a new format which specifies -both a `type` and a `text` field. An example is provided below: - -```python -completion = client.chat.completions.create( - model="NousResearch/Meta-Llama-3-8B-Instruct", - messages=[ - { - "role": "user", - "content": [ - {"type": "text", "text": "Classify this sentiment: vLLM is wonderful!"}, - ], - }, - ], -) -``` - -Most chat templates for LLMs expect the `content` field to be a string, but there are some newer models like -`meta-llama/Llama-Guard-3-1B` that expect the content to be formatted according to the OpenAI schema in the -request. vLLM provides best-effort support to detect this automatically, which is logged as a string like -*"Detected the chat template content format to be..."*, and internally converts incoming requests to match -the detected format, which can be one of: - -- `"string"`: A string. - - Example: `"Hello world"` -- `"openai"`: A list of dictionaries, similar to OpenAI schema. - - Example: `[{"type": "text", "text": "Hello world!"}]` - -If the result is not what you expect, you can set the `--chat-template-content-format` CLI argument -to override which format to use. - -## Extra Parameters - -vLLM supports a set of parameters that are not part of the OpenAI API. -In order to use them, you can pass them as extra parameters in the OpenAI client. -Or directly merge them into the JSON payload if you are using HTTP call directly. - -```python -completion = client.chat.completions.create( - model="NousResearch/Meta-Llama-3-8B-Instruct", - messages=[ - {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}, - ], - extra_body={ - "structured_outputs": {"choice": ["positive", "negative"]}, - }, -) -``` - -## Extra HTTP Headers - -Only `X-Request-Id` HTTP request header is supported for now. It can be enabled -with `--enable-request-id-headers`. - -??? code - - ```python - completion = client.chat.completions.create( - model="NousResearch/Meta-Llama-3-8B-Instruct", - messages=[ - {"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}, - ], - extra_headers={ - "x-request-id": "sentiment-classification-00001", - }, - ) - print(completion._request_id) - - completion = client.completions.create( - model="NousResearch/Meta-Llama-3-8B-Instruct", - prompt="A robot may not injure a human being", - extra_headers={ - "x-request-id": "completion-test", - }, - ) - print(completion._request_id) - ``` - -## Offline API Documentation - -The FastAPI `/docs` endpoint requires an internet connection by default. To enable offline access in air-gapped environments, use the `--enable-offline-docs` flag: - -```bash -vllm serve NousResearch/Meta-Llama-3-8B-Instruct --enable-offline-docs -``` - -## API Reference - -### Completions API - -Our Completions API is compatible with [OpenAI's Completions API](https://platform.openai.com/docs/api-reference/completions); -you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it. - -Code example: [examples/basic/online_serving/openai_completion_client.py](../../examples/basic/online_serving/openai_completion_client.py) - -#### Extra parameters - -The following [sampling parameters](../api/README.md#inference-parameters) are supported. - -??? code - - ```python - --8<-- "vllm/entrypoints/openai/completion/protocol.py:completion-sampling-params" - ``` - -The following extra parameters are supported: - -??? code - - ```python - --8<-- "vllm/entrypoints/openai/completion/protocol.py:completion-extra-params" - ``` - -### Chat API - -Our Chat API is compatible with [OpenAI's Chat Completions API](https://platform.openai.com/docs/api-reference/chat); -you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it. - -We support both [Vision](https://platform.openai.com/docs/guides/vision)- and -[Audio](https://platform.openai.com/docs/guides/audio?audio-generation-quickstart-example=audio-in)-related parameters; -see our [Multimodal Inputs](../features/multimodal_inputs.md) guide for more information. - -- *Note: `image_url.detail` parameter is not supported.* - -Code example: [examples/basic/online_serving/openai_chat_completion_client.py](../../examples/basic/online_serving/openai_chat_completion_client.py) - -#### Extra parameters - -The following [sampling parameters](../api/README.md#inference-parameters) are supported. - -??? code - - ```python - --8<-- "vllm/entrypoints/openai/chat_completion/protocol.py:chat-completion-sampling-params" - ``` - -The following extra parameters are supported: - -??? code - - ```python - --8<-- "vllm/entrypoints/openai/chat_completion/protocol.py:chat-completion-extra-params" - ``` - -### Responses API - -Our Responses API is compatible with [OpenAI's Responses API](https://platform.openai.com/docs/api-reference/responses); -you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it. - -Code example: [examples/tool_calling/openai_responses_client_with_tools.py](../../examples/tool_calling/openai_responses_client_with_tools.py) - -#### Extra parameters - -The following extra parameters in the request object are supported: - -??? code - - ```python - --8<-- "vllm/entrypoints/openai/responses/protocol.py:responses-extra-params" - ``` - -The following extra parameters in the response object are supported: - -??? code - - ```python - --8<-- "vllm/entrypoints/openai/responses/protocol.py:responses-response-extra-params" - ``` - -### Transcriptions API - -Our Transcriptions API is compatible with [OpenAI's Transcriptions API](https://platform.openai.com/docs/api-reference/audio/createTranscription); -you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it. - -!!! note - To use the Transcriptions API, please install with extra audio dependencies using `pip install vllm[audio]`. - -Code example: [examples/speech_to_text/openai/openai_transcription_client.py](../../examples/speech_to_text/openai/openai_transcription_client.py) - -NOTE: beam search is currently supported in the transcriptions endpoint for encoder-decoder multimodal models, e.g., whisper, but highly inefficient as work for handling the encoder/decoder cache is actively ongoing. This is an active point of ongoing optimization and will be handled properly in the very near future. - -#### API Enforced Limits - -Set the maximum audio file size (in MB) that VLLM will accept, via the -`VLLM_MAX_AUDIO_CLIP_FILESIZE_MB` environment variable. Default is 25 MB. - -#### Uploading Audio Files - -The Transcriptions API supports uploading audio files in various formats including FLAC, MP3, MP4, MPEG, MPGA, M4A, OGG, WAV, and WEBM. - -**Using OpenAI Python Client:** - -??? code - - ```python - from openai import OpenAI - - client = OpenAI( - base_url="http://localhost:8000/v1", - api_key="token-abc123", - ) - - # Upload audio file from disk - with open("audio.mp3", "rb") as audio_file: - transcription = client.audio.transcriptions.create( - model="openai/whisper-large-v3-turbo", - file=audio_file, - language="en", - response_format="verbose_json", - ) - - print(transcription.text) - ``` - -**Using curl with multipart/form-data:** - -??? code - - ```bash - curl -X POST "http://localhost:8000/v1/audio/transcriptions" \ - -H "Authorization: Bearer token-abc123" \ - -F "file=@audio.mp3" \ - -F "model=openai/whisper-large-v3-turbo" \ - -F "language=en" \ - -F "response_format=verbose_json" - ``` - -**Supported Parameters:** - -- `file`: The audio file to transcribe (required) -- `model`: The model to use for transcription (required) -- `language`: The language code (e.g., "en", "zh") (optional) -- `prompt`: Optional text to guide the transcription style (optional) -- `response_format`: Format of the response ("json", "text") (optional) -- `temperature`: Sampling temperature between 0 and 1 (optional) - -For the complete list of supported parameters including sampling parameters and vLLM extensions, see the [protocol definitions](https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/openai/protocol.py#L2182). - -**Response Format:** - -For `verbose_json` response format: - -??? code - - ```json - { - "text": "Hello, this is a transcription of the audio file.", - "language": "en", - "duration": 5.42, - "segments": [ - { - "id": 0, - "seek": 0, - "start": 0.0, - "end": 2.5, - "text": "Hello, this is a transcription", - "tokens": [50364, 938, 428, 307, 275, 28347], - "temperature": 0.0, - "avg_logprob": -0.245, - "compression_ratio": 1.235, - "no_speech_prob": 0.012 - } - ] - } - ``` -Currently “verbose_json” response format doesn’t support no_speech_prob. - -#### Extra Parameters - -The following [sampling parameters](../api/README.md#inference-parameters) are supported. - -??? code - - ```python - --8<-- "vllm/entrypoints/openai/speech_to_text/protocol.py:transcription-sampling-params" - ``` - -The following extra parameters are supported: - -??? code - - ```python - --8<-- "vllm/entrypoints/openai/speech_to_text/protocol.py:transcription-extra-params" - ``` - -### Translations API - -Our Translation API is compatible with [OpenAI's Translations API](https://platform.openai.com/docs/api-reference/audio/createTranslation); -you can use the [official OpenAI Python client](https://github.com/openai/openai-python) to interact with it. -Whisper models can translate audio from one of the 55 non-English supported languages into English. -Please mind that the popular `openai/whisper-large-v3-turbo` model does not support translating. - -!!! note - To use the Translation API, please install with extra audio dependencies using `pip install vllm[audio]`. - -Code example: [examples/speech_to_text/openai/openai_translation_client.py](../../examples/speech_to_text/openai/openai_translation_client.py) - -#### Extra Parameters - -The following [sampling parameters](../api/README.md#inference-parameters) are supported. - -```python ---8<-- "vllm/entrypoints/openai/speech_to_text/protocol.py:translation-sampling-params" -``` - -The following extra parameters are supported: - -```python ---8<-- "vllm/entrypoints/openai/speech_to_text/protocol.py:translation-extra-params" -``` - -### Realtime API - -The Realtime API provides WebSocket-based streaming audio transcription, allowing real-time speech-to-text as audio is being recorded. - -!!! note - To use the Realtime API, please install with extra audio dependencies using `uv pip install vllm[audio]`. - -#### Audio Format - -Audio must be sent as base64-encoded PCM16 audio at 16kHz sample rate, mono channel. - -#### Protocol Overview - -1. Client connects to `ws://host/v1/realtime` -2. Server sends `session.created` event -3. Client optionally sends `session.update` with model/params -4. Client sends `input_audio_buffer.commit` when ready -5. Client sends `input_audio_buffer.append` events with base64 PCM16 chunks -6. Server sends `transcription.delta` events with incremental text -7. Server sends `transcription.done` with final text + usage -8. Repeat from step 5 for next utterance -9. Optionally, client sends input_audio_buffer.commit with final=True - to signal audio input is finished. Useful when streaming audio files - -#### Client → Server Events - -| Event | Description | -| ----- | ----------- | -| `input_audio_buffer.append` | Send base64-encoded audio chunk: `{"type": "input_audio_buffer.append", "audio": ""}` | -| `input_audio_buffer.commit` | Trigger transcription processing or end: `{"type": "input_audio_buffer.commit", "final": bool}` | -| `session.update` | Configure session: `{"type": "session.update", "model": "model-name"}` | - -#### Server → Client Events - -| Event | Description | -| ----- | ----------- | -| `session.created` | Connection established with session ID and timestamp | -| `transcription.delta` | Incremental transcription text: `{"type": "transcription.delta", "delta": "text"}` | -| `transcription.done` | Final transcription with usage stats | -| `error` | Error notification with message and optional code | - -#### Example Clients - -- [openai_realtime_client.py](https://github.com/vllm-project/vllm/tree/main/examples/speech_to_text/realtime/openai_realtime_client.py) - Upload and transcribe an audio file -- [openai_realtime_microphone_client.py](https://github.com/vllm-project/vllm/tree/main/examples/speech_to_text/realtime/openai_realtime_microphone_client.py) - Gradio demo for live microphone transcription - -### Tokenizer API - -Our Tokenizer API is a simple wrapper over [HuggingFace-style tokenizers](https://huggingface.co/docs/transformers/en/main_classes/tokenizer). -It consists of two endpoints: - -- `/tokenize` corresponds to calling `tokenizer.encode()`. -- `/detokenize` corresponds to calling `tokenizer.decode()`. - -### Generative Scoring API - -The `/generative_scoring` endpoint uses a CausalLM model (e.g., Llama, Qwen, Mistral) to compute the probability of specified token IDs appearing as the next token. Each item (document) is concatenated with the query to form a prompt, and the model predicts how likely each label token is as the next token after that prompt. This lets you score items against a query — for example, asking "Is this the capital of France?" and scoring each city by how likely the model is to answer "Yes". - -This endpoint is automatically available when the server is started with a generative model (task `"generate"`). It is separate from the pooling-based [Score API](../models/pooling_models/scoring.md#score-api), which uses cross-encoder, bi-encoder, or late-interaction models. - -**Requirements:** - -- The `label_token_ids` parameter is **required** and must contain **at least 1 token ID**. -- When 2 label tokens are provided, the score equals `P(label_token_ids[0]) / (P(label_token_ids[0]) + P(label_token_ids[1]))` (softmax over the two labels). -- When more labels are provided, the score is the softmax-normalized probability of the first label token across all label tokens. - -#### Example - -```bash -curl -X POST http://localhost:8000/generative_scoring \ - -H "Content-Type: application/json" \ - -d '{ - "model": "Qwen/Qwen3-0.6B", - "query": "Is this city the capital of France?", - "items": ["Paris", "London", "Berlin"], - "label_token_ids": [9454, 2753] - }' -``` - -Here, each item is appended to the query to form prompts like `"Is this city the capital of France? Paris"`, `"... London"`, etc. The model then predicts the next token, and the score reflects the probability of "Yes" (token 9454) vs "No" (token 2753). - -??? console "Response" - - ```json - { - "id": "generative-scoring-abc123", - "object": "list", - "created": 1234567890, - "model": "Qwen/Qwen3-0.6B", - "data": [ - {"index": 0, "object": "score", "score": 0.95}, - {"index": 1, "object": "score", "score": 0.12}, - {"index": 2, "object": "score", "score": 0.08} - ], - "usage": {"prompt_tokens": 45, "total_tokens": 48, "completion_tokens": 3} - } - ``` - -#### How it works - -1. **Prompt Construction**: For each item, builds `prompt = query + item` (or `item + query` if `item_first=true`) -2. **Forward Pass**: Runs the model on each prompt to get next-token logits -3. **Probability Extraction**: Extracts logprobs for the specified `label_token_ids` -4. **Softmax Normalization**: Applies softmax over only the label tokens (when `apply_softmax=true`) -5. **Score**: Returns the normalized probability of the first label token - -#### Finding Token IDs - -To find the token IDs for your labels, use the tokenizer: - -```python -from transformers import AutoTokenizer - -tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B") -yes_id = tokenizer.encode("Yes", add_special_tokens=False)[0] -no_id = tokenizer.encode("No", add_special_tokens=False)[0] -print(f"Yes: {yes_id}, No: {no_id}") -``` - -## Ray Serve LLM - -Ray Serve LLM enables scalable, production-grade serving of the vLLM engine. It integrates tightly with vLLM and extends it with features such as auto-scaling, load balancing, and back-pressure. - -Key capabilities: - -- Exposes an OpenAI-compatible HTTP API as well as a Pythonic API. -- Scales from a single GPU to a multi-node cluster without code changes. -- Provides observability and autoscaling policies through Ray dashboards and metrics. - -The following example shows how to deploy a large model like DeepSeek R1 with Ray Serve LLM: [examples/ray_serving/ray_serve_deepseek.py](../../examples/ray_serving/ray_serve_deepseek.py). - -Learn more about Ray Serve LLM with the official [Ray Serve LLM documentation](https://docs.ray.io/en/latest/serve/llm/index.html). diff --git a/mkdocs.yaml b/mkdocs.yaml index 4b06b31ebe..88d99bf244 100644 --- a/mkdocs.yaml +++ b/mkdocs.yaml @@ -110,6 +110,7 @@ plugins: redirect_maps: features/spec_decode/README.md: features/speculative_decoding/README.md features/spec_decode/speculators.md: features/speculative_decoding/speculators.md + serving/openai_compatible_server.md: serving/online_serving/README.md markdown_extensions: - attr_list