[Rust Frontend] Move code from vllm-frontend-rs (#43283)
Signed-off-by: Bugen Zhao <i@bugenzhao.com> Signed-off-by: Nick Hill <nickhill123@gmail.com> Signed-off-by: Eric Curtin <eric.curtin@docker.com> Signed-off-by: Dev-X25874 <283057883+Dev-X25874@users.noreply.github.com> Signed-off-by: Will.hou <1205157517@qq.com> Signed-off-by: Will.hou <willamhou@ceresman.com> Co-authored-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Eric Curtin <eric.curtin@docker.com> Co-authored-by: Dev-X25874 <283057883+Dev-X25874@users.noreply.github.com> Co-authored-by: Will.hou <1205157517@qq.com> Co-authored-by: Will.hou <willamhou@ceresman.com> Please see https://github.com/Inferact/vllm-frontend-rs for full original commit history.
This commit is contained in:
co-authored by
Nick Hill
Eric Curtin
Dev-X25874
Will.hou
Will.hou
parent
39d5fa96a7
commit
39910f2b25
@@ -0,0 +1,44 @@
|
||||
# `vllm-rs` CLI Quick Start
|
||||
|
||||
Start Qwen3 with one managed `vllm-rs serve` command from the repo root:
|
||||
|
||||
```bash
|
||||
HF_HUB_OFFLINE=1 \
|
||||
VLLM_CPU_KVCACHE_SPACE=2 \
|
||||
VLLM_HOST_IP=127.0.0.1 \
|
||||
VLLM_LOOPBACK_IP=127.0.0.1 \
|
||||
cargo run --bin vllm-rs -- serve \
|
||||
Qwen/Qwen3-0.6B \
|
||||
--python ../vllm/.venv/bin/python \
|
||||
--max-model-len 512 \
|
||||
-- \
|
||||
--dtype float16
|
||||
```
|
||||
|
||||
This launches:
|
||||
|
||||
- a managed headless Python `vllm` engine
|
||||
- the Rust OpenAI-compatible frontend on `127.0.0.1:8000`
|
||||
|
||||
All Python engine arguments must be placed after `--`. Arguments before `--` are parsed by the Rust
|
||||
frontend itself.
|
||||
|
||||
You can then send OpenAI-style requests to the Rust frontend:
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"messages": [{"role": "user", "content": "What is the capital of France?"}],
|
||||
"stream": true
|
||||
}'
|
||||
```
|
||||
|
||||
If you already started headless `vllm` yourself, use `frontend` instead:
|
||||
|
||||
```bash
|
||||
cargo run --bin vllm-rs -- frontend \
|
||||
--handshake-address tcp://127.0.0.1:62100 \
|
||||
Qwen/Qwen3-0.6B
|
||||
```
|
||||
Reference in New Issue
Block a user