[Docs] Reorganize online serving docs. (#41907)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io> Signed-off-by: wang.yuqi <noooop@126.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
This commit is contained in:
co-authored by
gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Cyrus Leung
Harry Mellor
parent
4a4fdabe28
commit
257af77bc2
@@ -92,7 +92,7 @@ response = client.chat.completions.create(
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
For more details on the `vllm` CLI commands, see the [OpenAI-compatible server documentation](../../serving/openai_compatible_server.md).
|
||||
For more details on the `vllm` CLI commands, see the [OpenAI-compatible server documentation](../../serving/online_serving/openai_compatible_server.md).
|
||||
|
||||
--8<-- [end:set-up-using-python]
|
||||
--8<-- [start:pre-built-wheels]
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
This guide will help you quickly get started with vLLM to perform:
|
||||
|
||||
- [Offline batched inference](#offline-batched-inference)
|
||||
- [Online serving using OpenAI-compatible server](#openai-compatible-server)
|
||||
- [Online serving](#online-serving)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -182,7 +182,7 @@ for output in outputs:
|
||||
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
|
||||
```
|
||||
|
||||
## OpenAI-Compatible Server
|
||||
## Online Serving
|
||||
|
||||
vLLM can be deployed as a server that implements the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API.
|
||||
By default, it starts the server at `http://localhost:8000`. You can specify the address with `--host` and `--port` arguments. The server currently hosts one model at a time and implements endpoints such as [list models](https://platform.openai.com/docs/api-reference/models/list), [create chat completion](https://platform.openai.com/docs/api-reference/chat/completions/create), and [create completion](https://platform.openai.com/docs/api-reference/completions/create) endpoints.
|
||||
@@ -195,7 +195,7 @@ vllm serve Qwen/Qwen2.5-1.5B-Instruct
|
||||
|
||||
!!! note
|
||||
By default, the server uses a predefined chat template stored in the tokenizer.
|
||||
You can learn about overriding it [here](../serving/openai_compatible_server.md#chat-template).
|
||||
You can learn about overriding it [here](../serving/online_serving/README.md#chat-template).
|
||||
!!! important
|
||||
By default, the server applies `generation_config.json` from the huggingface model repository if it exists. This means the default values of certain sampling parameters can be overridden by those recommended by the model creator.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user