Refactor system architecture (#109)
This commit is contained in:
@@ -10,13 +10,17 @@ pip install -e . # This may take several minutes.
|
||||
## Test simple server
|
||||
|
||||
```bash
|
||||
# Single-GPU inference.
|
||||
python examples/simple_server.py # --model <your_model>
|
||||
|
||||
# Multi-GPU inference (e.g., 2 GPUs).
|
||||
ray start --head
|
||||
python simple_server.py
|
||||
python examples/simple_server.py -tp 2 # --model <your_model>
|
||||
```
|
||||
|
||||
The detailed arguments for `simple_server.py` can be found by:
|
||||
```bash
|
||||
python simple_server.py --help
|
||||
python examples/simple_server.py --help
|
||||
```
|
||||
|
||||
## FastAPI server
|
||||
@@ -24,12 +28,12 @@ python simple_server.py --help
|
||||
To start the server:
|
||||
```bash
|
||||
ray start --head
|
||||
python -m cacheflow.http_frontend.fastapi_frontend
|
||||
python -m cacheflow.entrypoints.fastapi_server # --model <your_model>
|
||||
```
|
||||
|
||||
To test the server:
|
||||
```bash
|
||||
python -m cacheflow.http_frontend.test_cli_client
|
||||
python test_cli_client.py
|
||||
```
|
||||
|
||||
## Gradio web server
|
||||
@@ -55,7 +59,6 @@ Since LLaMA weight is not fully public, we cannot directly download the LLaMA we
|
||||
python src/transformers/models/llama/convert_llama_weights_to_hf.py \
|
||||
--input_dir /path/to/downloaded/llama/weights --model_size 7B --output_dir /output/path/llama-7b
|
||||
```
|
||||
Please make sure that `llama` is included in the output directory name.
|
||||
2. For all the commands above, specify the model with `--model /output/path/llama-7b` to load the model. For example:
|
||||
```bash
|
||||
python simple_server.py --model /output/path/llama-7b
|
||||
|
||||
Reference in New Issue
Block a user