[Rust Frontend] Add standalone Rust renderer (#50289)

This commit is contained in:
Sage
2026-08-06 11:23:26 +00:00
committed by GitHub
parent ef2615c2e0
commit 865781e627
15 changed files with 720 additions and 40 deletions
+39 -1
View File
@@ -15,6 +15,7 @@ The component is organized as a Cargo workspace with several crates, layered bot
│ vllm-cmd / vllm-rs │ CLI entrypoint:
│ │ Python vLLM frontend subprocess
│ │ Rust managed-engine serve mode
│ │ Engine-free render mode
├─────────────────────────────────┤
│ vllm-server │ OpenAI-compatible HTTP API (axum)
├─────────────────────────────────┤
@@ -74,9 +75,46 @@ To build the `vllm-rs` in isolation:
./build_rust.sh
```
### Engine-free renderer
`vllm-rs render` serves text-only request preprocessing without starting or
connecting to a Python inference engine:
```bash
cargo run --manifest-path rust/Cargo.toml -p vllm-cmd --release -- \
render Qwen/Qwen3-32B \
--host 127.0.0.1 --max-model-len 32768
```
It exposes `/v1/chat/completions/render` and `/v1/completions/render`. Only
tokenizer and model configuration files are loaded; model weights, PyTorch,
and vLLM kernels are not required.
The render endpoints return the public token-in `GenerateRequest` consumed by
the Rust `/inference/v1/generate` endpoint. A chat render response, or one item
from a completion render response, can be submitted to that endpoint without
changing its fields.
The render and inference paths use the same `vllm-chat` and `vllm-text`
request-preparation logic; render mode stops before engine submission.
Tool-call and reasoning parsers use model-based auto-detection by default. Use
`--tool-call-parser` and `--reasoning-parser` to override either selection;
unified parsers require the same selection for both options.
```bash
curl http://127.0.0.1:8000/v1/chat/completions/render \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-32B",
"messages": [{"role": "user", "content": "Hello"}],
"max_completion_tokens": 16
}'
```
### Example Request
After either startup path, you can use any OpenAI-compatible client:
After either full-frontend startup path, you can use any OpenAI-compatible
client against the inference endpoints:
```bash
curl http://127.0.0.1:8000/v1/chat/completions \