[Examples][last/6] Resettle examples. (#41084)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io> Signed-off-by: wang.yuqi <noooop@126.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
parent
57c2f724c1
commit
1d694e78c9
+1
-1
@@ -5,7 +5,7 @@ Start vLLM API server:
|
||||
vllm serve meta-llama/Llama-2-7b-chat-hf
|
||||
|
||||
Start Gradio OpenAI Chatbot Webserver:
|
||||
python examples/online_serving/gradio_openai_chatbot_webserver.py \
|
||||
python examples/applications/chatbot/gradio_openai_chatbot_webserver.py \
|
||||
-m meta-llama/Llama-2-7b-chat-hf
|
||||
|
||||
Note that `pip install --upgrade gradio` is needed to run this example.
|
||||
+1
-1
@@ -6,7 +6,7 @@ Start vLLM API server:
|
||||
--model meta-llama/Llama-2-7b-chat-hf
|
||||
|
||||
Start Webserver:
|
||||
python examples/online_serving/gradio_webserver.py
|
||||
python examples/applications/chatbot/gradio_webserver.py
|
||||
|
||||
Note that `pip install --upgrade gradio` is needed to run this example.
|
||||
More details: https://github.com/gradio-app/gradio
|
||||
+1
-1
@@ -8,7 +8,7 @@ token-by-token output in offline inference scenarios. It demonstrates DELTA mode
|
||||
streaming where you receive new tokens as they are generated.
|
||||
|
||||
Usage:
|
||||
python examples/offline_inference/async_llm_streaming.py
|
||||
python examples/deployment/async_llm_streaming.py
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
@@ -33,7 +33,7 @@ https://github.com/coreweave/tensorizer
|
||||
To serialize a model, install vLLM from source, then run something
|
||||
like this from the root level of this repository:
|
||||
|
||||
python examples/others/tensorize_vllm_model.py \
|
||||
python examples/features/tensorize_vllm_model.py \
|
||||
--model facebook/opt-125m \
|
||||
serialize \
|
||||
--serialized-directory s3://my-bucket \
|
||||
@@ -53,7 +53,7 @@ providing a `--keyfile` argument.
|
||||
To deserialize a model, you can run something like this from the root
|
||||
level of this repository:
|
||||
|
||||
python examples/others/tensorize_vllm_model.py \
|
||||
python examples/features/tensorize_vllm_model.py \
|
||||
--model EleutherAI/gpt-j-6B \
|
||||
--dtype float16 \
|
||||
deserialize \
|
||||
@@ -71,11 +71,11 @@ shard's rank. Sharded models serialized with this script will be named as
|
||||
model-rank-%03d.tensors
|
||||
|
||||
For more information on the available arguments for serializing, run
|
||||
`python -m examples.others.tensorize_vllm_model serialize --help`.
|
||||
`python -m examples.features.tensorize_vllm_model serialize --help`.
|
||||
|
||||
Or for deserializing:
|
||||
|
||||
`python examples/others/tensorize_vllm_model.py deserialize --help`.
|
||||
`python examples/features/tensorize_vllm_model.py deserialize --help`.
|
||||
|
||||
Once a model is serialized, tensorizer can be invoked with the `LLM` class
|
||||
directly to load models:
|
||||
@@ -100,7 +100,7 @@ vllm serve s3://my-bucket/vllm/facebook/opt-125m/v1 \
|
||||
In order to see all of the available arguments usable to configure
|
||||
loading with tensorizer that are given to `TensorizerConfig`, run:
|
||||
|
||||
`python examples/others/tensorize_vllm_model.py deserialize --help`
|
||||
`python examples/features/tensorize_vllm_model.py deserialize --help`
|
||||
|
||||
under the `tensorizer options` section. These can also be used for
|
||||
deserialization in this example script, although `--tensorizer-uri` and
|
||||
@@ -43,7 +43,7 @@ Both platforms provide equivalent monitoring capabilities:
|
||||
First, navigate to this example's directory:
|
||||
|
||||
```bash
|
||||
cd examples/online_serving/dashboards
|
||||
cd examples/observability/dashboards
|
||||
```
|
||||
|
||||
### Grafana
|
||||
|
||||
@@ -1,26 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
from openai import APIConnectionError, OpenAI
|
||||
from openai.pagination import SyncPage
|
||||
from openai.types.model import Model
|
||||
|
||||
|
||||
def get_first_model(client: OpenAI) -> str:
|
||||
"""
|
||||
Get the first model from the vLLM server.
|
||||
"""
|
||||
try:
|
||||
models: SyncPage[Model] = client.models.list()
|
||||
except APIConnectionError as e:
|
||||
raise RuntimeError(
|
||||
"Failed to get the list of models from the vLLM server at "
|
||||
f"{client.base_url} with API key {client.api_key}. Check\n"
|
||||
"1. the server is running\n"
|
||||
"2. the server URL is correct\n"
|
||||
"3. the API key is correct"
|
||||
) from e
|
||||
|
||||
if len(models.data) == 0:
|
||||
raise RuntimeError(f"No models found on the vLLM server at {client.base_url}")
|
||||
|
||||
return models.data[0].id
|
||||
@@ -119,7 +119,7 @@ echo " - API Key: $API_KEY"
|
||||
echo " - Native Pooling: $POOLING_TYPE | Cross-chunk: MEAN"
|
||||
echo ""
|
||||
echo "🧪 Test the server with:"
|
||||
echo " python examples/online_serving/openai_embedding_long_text/client.py"
|
||||
echo " python examples/pooling/embed/openai_embedding_long_text/client.py"
|
||||
echo ""
|
||||
echo "📚 Enhanced features enabled:"
|
||||
echo " ✅ Intelligent native pooling type detection"
|
||||
|
||||
@@ -15,14 +15,14 @@ Start the vLLM server:
|
||||
Then run this script:
|
||||
|
||||
# Use the built-in sample audio
|
||||
python examples/online_serving/openai_lid_client.py
|
||||
python examples/speech_to_text/lid/openai_lid_client.py
|
||||
|
||||
# Use your own audio file(s)
|
||||
python examples/online_serving/openai_lid_client.py \
|
||||
python examples/speech_to_text/lid/openai_lid_client.py \
|
||||
--audio_paths audio_en.wav audio_zh.wav audio_fr.wav
|
||||
|
||||
# Batch-identify multiple files in one run
|
||||
python examples/online_serving/openai_lid_client.py \
|
||||
python examples/speech_to_text/lid/openai_lid_client.py \
|
||||
--audio_paths /path/to/dir/*.wav
|
||||
|
||||
Requirements:
|
||||
|
||||
Reference in New Issue
Block a user