Refactor system architecture (#109)

This commit is contained in:
Woosuk Kwon
2023-05-20 13:06:59 -07:00
committed by GitHub
parent 7297fa6f7c
commit c3442c1f6f
24 changed files with 1017 additions and 1034 deletions
+8 -5
View File
@@ -10,13 +10,17 @@ pip install -e . # This may take several minutes.
## Test simple server
```bash
# Single-GPU inference.
python examples/simple_server.py # --model <your_model>
# Multi-GPU inference (e.g., 2 GPUs).
ray start --head
python simple_server.py
python examples/simple_server.py -tp 2 # --model <your_model>
```
The detailed arguments for `simple_server.py` can be found by:
```bash
python simple_server.py --help
python examples/simple_server.py --help
```
## FastAPI server
@@ -24,12 +28,12 @@ python simple_server.py --help
To start the server:
```bash
ray start --head
python -m cacheflow.http_frontend.fastapi_frontend
python -m cacheflow.entrypoints.fastapi_server # --model <your_model>
```
To test the server:
```bash
python -m cacheflow.http_frontend.test_cli_client
python test_cli_client.py
```
## Gradio web server
@@ -55,7 +59,6 @@ Since LLaMA weight is not fully public, we cannot directly download the LLaMA we
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
--input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path/llama-7b
```
Please make sure that `llama` is included in the output directory name.
2. For all the commands above, specify the model with `--model /output/path/llama-7b` to load the model. For example:
```bash
python simple_server.py --model /output/path/llama-7b