Vllm recipes tool improve (#53325)
Signed-off-by: louie-tsai <louie.tsai@intel.com>
This commit is contained in:
Louie Tsai
+83
-111
@@ -1,87 +1,46 @@
|
||||
# vLLM Recipes Tools
|
||||
|
||||
Utilities for consuming deployment configurations from [vLLM Recipes](https://recipes.vllm.ai/) and converting them into files that can be used directly with vLLM.
|
||||
Convert a vLLM Recipes deployment rendering into files for `vllm serve`.
|
||||
|
||||
## `recipe_json_to_vllm_config.py`
|
||||
## Optimized Deployment Flow
|
||||
|
||||
Converts a hardware-specific vLLM Recipes JSON rendering into:
|
||||
```mermaid
|
||||
flowchart LR
|
||||
R["vLLM Recipe"] --> C["Recipe Converter"]
|
||||
H["Hardware Info (optional)"] --> C
|
||||
W["Workload Info (optional)"] --> C
|
||||
C --> F["config.yml + env.sh"]
|
||||
C -.-> S["Sweep Tuning (optional)"]
|
||||
S -.-> F
|
||||
F --> D["vLLM Docker Image"]
|
||||
D --> E["OpenAI Endpoint"]
|
||||
|
||||
- `config.yml` — native configuration for `vllm serve --config`
|
||||
- `env.sh` — environment variables required by the recipe
|
||||
style S stroke-dasharray: 5 5
|
||||
```
|
||||
|
||||
The converter uses the Recipes JSON API as the source of truth. It does not reimplement model-variant, hardware, or strategy compatibility rules.
|
||||
The recipe is the baseline. Hardware and workload information can optionally
|
||||
refine the initial configuration. Sweep tuning is an optional validation step.
|
||||
|
||||
## Install Dependency
|
||||
## 1. vLLM Recipes Only
|
||||
|
||||
Use the converter directly when the recipe already contains the deployment
|
||||
settings you need. This path requires only PyYAML; the vLLM Python package is
|
||||
not required unless optional runtime tuning or sweep generation is requested.
|
||||
|
||||
```bash
|
||||
pip install pyyaml
|
||||
```
|
||||
|
||||
## Recipe Selection Model
|
||||
Choose whichever recipe-selection method fits the workflow:
|
||||
|
||||
Discovery follows the JSON links published by vLLM Recipes:
|
||||
|
||||
```text
|
||||
/models.json
|
||||
-> /{hf_id}.json
|
||||
-> recommended_command.by_hardware[{hardware}]
|
||||
-> /{hf_id}/hw/{hardware}.json
|
||||
-> recommended strategy
|
||||
-> alternatives[{strategy}]
|
||||
```
|
||||
|
||||
The per-hardware JSON is the recommended deployment rendering for that model and hardware. Alternative strategy renderings are referenced through the JSON's `alternatives` map.
|
||||
|
||||
The converter follows those exact links instead of constructing strategy paths locally. This also handles the Recipes API's legacy default-hardware strategy path automatically.
|
||||
|
||||
## Interactive Recipe Discovery
|
||||
|
||||
If you do not know the recipe JSON URL, run the script without an input:
|
||||
**Interactive discovery** — search models, then choose hardware and strategy:
|
||||
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py
|
||||
```
|
||||
|
||||
The script will:
|
||||
|
||||
1. Search the Recipes model index, including promoted variant model IDs.
|
||||
2. Ask which model to use.
|
||||
3. Show the hardware configurations available for that model.
|
||||
4. Show the recommended strategy and generated alternative strategies.
|
||||
5. Resolve the exact Recipes JSON endpoint.
|
||||
6. Generate `config.yml` and `env.sh`.
|
||||
|
||||
Example interaction:
|
||||
|
||||
```text
|
||||
No recipe JSON supplied; starting Recipes API discovery.
|
||||
Model search (for example: llama 3.1): llama 3.1 fp8
|
||||
|
||||
Matching models:
|
||||
[1] nvidia/Llama-3.1-8B-Instruct-FP8 — Llama-3.1-8B-Instruct [Meta]
|
||||
Select model: 1
|
||||
|
||||
Available hardware:
|
||||
[1] arc_pro_b70
|
||||
[2] b200
|
||||
[3] h100
|
||||
Select hardware: 1
|
||||
|
||||
Available strategies:
|
||||
[1] single_node_tp (recommended)
|
||||
[2] multi_node_tp
|
||||
Select strategy: 1
|
||||
|
||||
Resolved recipe:
|
||||
Model: nvidia/Llama-3.1-8B-Instruct-FP8
|
||||
Hardware: arc_pro_b70
|
||||
Strategy: single_node_tp
|
||||
JSON: https://recipes.vllm.ai/nvidia/Llama-3.1-8B-Instruct-FP8/hw/arc_pro_b70.json
|
||||
```
|
||||
|
||||
## Non-Interactive Recipe Discovery
|
||||
|
||||
For automation or CI, specify the model and hardware directly:
|
||||
**Non-interactive discovery** — provide model and hardware and use the
|
||||
Recipes-recommended strategy:
|
||||
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py \
|
||||
@@ -89,68 +48,81 @@ python3 tools/recipes/recipe_json_to_vllm_config.py \
|
||||
--hardware xeon6
|
||||
```
|
||||
|
||||
When `--strategy` is omitted with both `--model` and `--hardware`, the converter uses the Recipes-recommended strategy from the per-hardware JSON.
|
||||
|
||||
To request a generated alternative strategy explicitly:
|
||||
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py \
|
||||
--model nvidia/Llama-3.1-8B-Instruct-FP8 \
|
||||
--hardware arc_pro_b70 \
|
||||
--strategy multi_node_tp
|
||||
```
|
||||
|
||||
The converter does not synthesize a URL such as `.../strategies/multi_node_tp.json`. It reads the selected hardware JSON and follows the exact `alternatives["multi_node_tp"]` link published by the Recipes API.
|
||||
|
||||
## Direct Recipe JSON Input
|
||||
|
||||
If you already know the recipe JSON URL:
|
||||
**Direct JSON input** — use a Recipes JSON URL or a local JSON file:
|
||||
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py \
|
||||
https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct/hw/xeon6.json
|
||||
```
|
||||
|
||||
A local JSON file is also supported:
|
||||
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py recipe.json
|
||||
```
|
||||
|
||||
When a positional JSON source is supplied, do not combine it with `--model`, `--hardware`, or `--strategy`.
|
||||
All paths generate `config.yml` and `env.sh`. See
|
||||
[REFERENCE.md](REFERENCE.md) for recipe discovery, strategy selection, direct
|
||||
JSON input, custom output files, and deployment scope.
|
||||
|
||||
## Start vLLM
|
||||
## 2. Hardware Information (Optional)
|
||||
|
||||
After generating the files:
|
||||
|
||||
```bash
|
||||
source env.sh
|
||||
vllm serve --config config.yml
|
||||
```
|
||||
|
||||
Example generated `config.yml`:
|
||||
|
||||
```yaml
|
||||
model: meta-llama/Llama-3.1-8B-Instruct
|
||||
tensor-parallel-size: 1
|
||||
```
|
||||
|
||||
If the selected recipe does not require environment variables, `env.sh` will contain no additional exports.
|
||||
|
||||
## Custom Output Files
|
||||
Add `--detect-hardware` when the target host's effective CPU/NUMA/memory
|
||||
resources should refine deployment-sensitive values such as
|
||||
`tensor-parallel-size` and `gpu-memory-utilization`.
|
||||
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--hardware xeon6 \
|
||||
--config-out llama31-xeon6.yml \
|
||||
--env-out llama31-xeon6-env.sh
|
||||
--detect-hardware
|
||||
```
|
||||
|
||||
## Strategy and Deployment Scope
|
||||
Hardware detection is optional and uses vLLM CPU resource utilities only when
|
||||
requested. See [RUNTIME_TUNING.md](RUNTIME_TUNING.md#hardware-information).
|
||||
|
||||
Strategy discovery and config conversion are separate concerns.
|
||||
## 3. Workload Information (Optional)
|
||||
|
||||
The converter can resolve any generated strategy JSON exposed by the Recipes API. It currently emits one `config.yml`, so the selected rendering must contain a single `vllm serve` `argv`.
|
||||
Workload hints can refine scheduler settings for one initial deployment
|
||||
suggestion. Inputs include token lengths, concurrency, and optional latency or
|
||||
capacity objectives.
|
||||
|
||||
Single-process renderings such as `single_node_tp` can be converted directly. Multi-node, disaggregated prefill/decode, or other multi-process renderings expose fields such as `head_argv`, `worker_argv`, `prefill`, or `decode`; the converter intentionally exits instead of generating an incomplete single-process config.
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--hardware xeon6 \
|
||||
--input-tokens 2048 \
|
||||
--output-tokens 128 \
|
||||
--concurrency 32 \
|
||||
--ttft-sla-ms 3000 \
|
||||
--tpot-sla-ms 100
|
||||
```
|
||||
|
||||
Hardware detection and workload information are independent optional inputs;
|
||||
they can also be supplied together. See
|
||||
[RUNTIME_TUNING.md](RUNTIME_TUNING.md#workload-information) for the supported
|
||||
inputs and how runtime parameters are calculated.
|
||||
|
||||
## 4. Sweep Tuning (Optional)
|
||||
|
||||
Use `--generate-sweep` when the initial scheduler suggestion should be validated
|
||||
with `vllm bench sweep serve`. The sweep benchmarks nearby scheduler values and
|
||||
`recommend.py` produces one measured `recommended-config.yml` plus the
|
||||
selection evidence in `recommendation.json`.
|
||||
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--hardware xeon6 \
|
||||
--detect-hardware \
|
||||
--input-tokens 2048 \
|
||||
--output-tokens 128 \
|
||||
--concurrency 32 \
|
||||
--generate-sweep
|
||||
```
|
||||
|
||||
See [SWEEP_TUNING.md](SWEEP_TUNING.md) for the benchmark, recommendation, and
|
||||
vLLM CPU Docker-shell workflow.
|
||||
|
||||
## Start vLLM
|
||||
|
||||
```bash
|
||||
source env.sh
|
||||
vllm serve --config config.yml
|
||||
```
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
# vLLM Recipes Tools
|
||||
|
||||
Utilities for consuming deployment configurations from [vLLM Recipes](https://recipes.vllm.ai/) and converting them into files that can be used directly with vLLM.
|
||||
|
||||
## Optimized Deployment Flow
|
||||
|
||||
The recipe provides the validated deployment baseline. Hardware and workload
|
||||
information are optional inputs that can refine deployment-sensitive values
|
||||
before the generated configuration is passed to vLLM.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
R["vLLM Recipe"] --> C["Recipe Converter"]
|
||||
H["Hardware Info (optional)"] --> C
|
||||
W["Workload Info (optional)"] --> C
|
||||
C --> F["config.yml + env.sh"]
|
||||
F --> D["vLLM Docker Image"]
|
||||
D --> E["OpenAI Endpoint"]
|
||||
```
|
||||
|
||||
## `recipe_json_to_vllm_config.py`
|
||||
|
||||
Converts a hardware-specific vLLM Recipes JSON rendering into:
|
||||
|
||||
- `config.yml` — native configuration for `vllm serve --config`
|
||||
- `env.sh` — environment variables required by the recipe
|
||||
|
||||
The converter uses the Recipes JSON API as the source of truth. It does not reimplement model-variant, hardware, or strategy compatibility rules.
|
||||
|
||||
## Install Dependency
|
||||
|
||||
```bash
|
||||
pip install pyyaml
|
||||
```
|
||||
|
||||
## Recipe Selection Model
|
||||
|
||||
Discovery follows the JSON links published by vLLM Recipes:
|
||||
|
||||
```text
|
||||
/models.json
|
||||
-> /{hf_id}.json
|
||||
-> recommended_command.by_hardware[{hardware}]
|
||||
-> /{hf_id}/hw/{hardware}.json
|
||||
-> recommended strategy
|
||||
-> alternatives[{strategy}]
|
||||
```
|
||||
|
||||
The per-hardware JSON is the recommended deployment rendering for that model and hardware. Alternative strategy renderings are referenced through the JSON's `alternatives` map.
|
||||
|
||||
The converter follows those exact links instead of constructing strategy paths locally. This also handles the Recipes API's legacy default-hardware strategy path automatically.
|
||||
|
||||
## Interactive Recipe Discovery
|
||||
|
||||
If you do not know the recipe JSON URL, run the script without an input:
|
||||
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py
|
||||
```
|
||||
|
||||
The script will:
|
||||
|
||||
1. Search the Recipes model index, including promoted variant model IDs.
|
||||
2. Ask which model to use.
|
||||
3. Show the hardware configurations available for that model.
|
||||
4. Show the recommended strategy and generated alternative strategies.
|
||||
5. Resolve the exact Recipes JSON endpoint.
|
||||
6. Generate `config.yml` and `env.sh`.
|
||||
|
||||
Example interaction:
|
||||
|
||||
```text
|
||||
No recipe JSON supplied; starting Recipes API discovery.
|
||||
Model search (for example: llama 3.1): llama 3.1 fp8
|
||||
|
||||
Matching models:
|
||||
[1] nvidia/Llama-3.1-8B-Instruct-FP8 — Llama-3.1-8B-Instruct [Meta]
|
||||
Select model: 1
|
||||
|
||||
Available hardware:
|
||||
[1] arc_pro_b70
|
||||
[2] b200
|
||||
[3] h100
|
||||
Select hardware: 1
|
||||
|
||||
Available strategies:
|
||||
[1] single_node_tp (recommended)
|
||||
[2] multi_node_tp
|
||||
Select strategy: 1
|
||||
|
||||
Resolved recipe:
|
||||
Model: nvidia/Llama-3.1-8B-Instruct-FP8
|
||||
Hardware: arc_pro_b70
|
||||
Strategy: single_node_tp
|
||||
JSON: https://recipes.vllm.ai/nvidia/Llama-3.1-8B-Instruct-FP8/hw/arc_pro_b70.json
|
||||
```
|
||||
|
||||
## Non-Interactive Recipe Discovery
|
||||
|
||||
For automation or CI, specify the model and hardware directly:
|
||||
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--hardware xeon6
|
||||
```
|
||||
|
||||
When `--strategy` is omitted with both `--model` and `--hardware`, the converter uses the Recipes-recommended strategy from the per-hardware JSON.
|
||||
|
||||
To request a generated alternative strategy explicitly:
|
||||
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py \
|
||||
--model nvidia/Llama-3.1-8B-Instruct-FP8 \
|
||||
--hardware arc_pro_b70 \
|
||||
--strategy multi_node_tp
|
||||
```
|
||||
|
||||
The converter does not synthesize a URL such as `.../strategies/multi_node_tp.json`. It reads the selected hardware JSON and follows the exact `alternatives["multi_node_tp"]` link published by the Recipes API.
|
||||
|
||||
## Direct Recipe JSON Input
|
||||
|
||||
If you already know the recipe JSON URL:
|
||||
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py \
|
||||
https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct/hw/xeon6.json
|
||||
```
|
||||
|
||||
A local JSON file is also supported:
|
||||
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py recipe.json
|
||||
```
|
||||
|
||||
When a positional JSON source is supplied, do not combine it with `--model`, `--hardware`, or `--strategy`.
|
||||
|
||||
## Start vLLM
|
||||
|
||||
After generating the files:
|
||||
|
||||
```bash
|
||||
source env.sh
|
||||
vllm serve --config config.yml
|
||||
```
|
||||
|
||||
Example generated `config.yml`:
|
||||
|
||||
```yaml
|
||||
model: meta-llama/Llama-3.1-8B-Instruct
|
||||
tensor-parallel-size: 1
|
||||
```
|
||||
|
||||
If the selected recipe does not require environment variables, `env.sh` will contain no additional exports.
|
||||
|
||||
## Custom Output Files
|
||||
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--hardware xeon6 \
|
||||
--config-out llama31-xeon6.yml \
|
||||
--env-out llama31-xeon6-env.sh
|
||||
```
|
||||
|
||||
## Strategy and Deployment Scope
|
||||
|
||||
Strategy discovery and config conversion are separate concerns.
|
||||
|
||||
The converter can resolve any generated strategy JSON exposed by the Recipes API. It currently emits one `config.yml`, so the selected rendering must contain a single `vllm serve` `argv`.
|
||||
|
||||
Single-process renderings such as `single_node_tp` can be converted directly. Multi-node, disaggregated prefill/decode, or other multi-process renderings expose fields such as `head_argv`, `worker_argv`, `prefill`, or `decode`; the converter intentionally exits instead of generating an incomplete single-process config.
|
||||
|
||||
## Related Guides
|
||||
|
||||
- [RUNTIME_TUNING.md](RUNTIME_TUNING.md) — optional hardware and workload inputs, runtime parameter policies, and calculations.
|
||||
- [SWEEP_TUNING.md](SWEEP_TUNING.md) — optional benchmark sweep and measured recommendation workflow.
|
||||
@@ -0,0 +1,292 @@
|
||||
# Runtime Tuning
|
||||
|
||||
The Recipes JSON remains the baseline. vLLM Recipes already provide validated
|
||||
model, hardware, strategy, environment variables, and serving arguments.
|
||||
Runtime tuning is optional and is intended for parameters whose best value can
|
||||
depend on the actual deployment resources or expected request workload.
|
||||
|
||||
## Information Sources
|
||||
|
||||
Runtime tuning combines up to three inputs:
|
||||
|
||||
- **vLLM Recipe** — the required baseline from the Recipes JSON API or a direct
|
||||
recipe JSON file.
|
||||
- **Hardware information (optional)** — detected with `--detect-hardware`,
|
||||
including effective CPU/NUMA topology and memory availability.
|
||||
- **Workload information (optional)** — supplied through CLI hints such as input
|
||||
and output token lengths, concurrency, TTFT/TPOT objectives, and target QPS.
|
||||
|
||||
If no optional hardware or workload information is supplied, the converter keeps
|
||||
the normal Recipes conversion behavior.
|
||||
|
||||
See [Deployment-Time Parameters](#deployment-time-parameters) for the mapping
|
||||
between these inputs and individual runtime parameters.
|
||||
|
||||
## Hardware Information
|
||||
|
||||
Hardware detection is enabled only with `--detect-hardware`. It reports the
|
||||
effective resources available to the process or container, including CPU/NUMA
|
||||
topology and memory information. The current policy can use those values to
|
||||
refine `tensor-parallel-size` and `gpu-memory-utilization`.
|
||||
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--hardware xeon6 \
|
||||
--detect-hardware
|
||||
```
|
||||
|
||||
Hardware detection is optional. Recipe hardware selects the tuning policy; host
|
||||
hardware detection provides resource information to that policy.
|
||||
|
||||
## Workload Information
|
||||
|
||||
Workload information is supplied explicitly because the converter runs before a
|
||||
vLLM server exists. Supported hints include:
|
||||
|
||||
- `--input-tokens`
|
||||
- `--output-tokens`
|
||||
- `--concurrency`
|
||||
- `--ttft-sla-ms`
|
||||
- `--tpot-sla-ms`
|
||||
- `--target-qps`
|
||||
|
||||
Example:
|
||||
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--hardware xeon6 \
|
||||
--input-tokens 2048 \
|
||||
--output-tokens 128 \
|
||||
--concurrency 32 \
|
||||
--ttft-sla-ms 3000 \
|
||||
--tpot-sla-ms 100
|
||||
```
|
||||
|
||||
`--output-tokens` is used to estimate steady-state request turnover for
|
||||
`max-num-batched-tokens`. When both `--target-qps` and `--tpot-sla-ms` are
|
||||
available, they are also used to estimate prompt arrival pressure per scheduler
|
||||
step. `--ttft-sla-ms` is collected but is not yet converted directly into a
|
||||
batch-size formula because the relationship between TTFT and scheduler budget is
|
||||
model- and hardware-dependent.
|
||||
|
||||
## Deployment-Time Parameters
|
||||
|
||||
The parameters below may already exist in a recipe. They are candidates for
|
||||
deployment-time refinement when the recipe value is missing, generic, or based
|
||||
on a validation environment that differs from the user's target deployment.
|
||||
|
||||
| Runtime parameter | Why it may need deployment-time refinement | Main decision input | Current policy |
|
||||
| --- | --- | --- | --- |
|
||||
| `tensor-parallel-size` | The effective CPU/NUMA topology available to a container or pod can differ from the system used to validate the recipe. | Hardware topology | Use the largest power-of-two TP value that does not exceed the effective NUMA-node count. |
|
||||
| `gpu-memory-utilization` | Available memory can differ by machine size, container limits, and other memory use. The vLLM option name is also used by the CPU backend. | Hardware memory + recipe baseline | Calculate a conservative fraction from the most constrained NUMA node. |
|
||||
| `max-num-seqs` | The useful scheduler concurrency depends on the number of requests expected to be active at the same time. | Workload concurrency | Set `max-num-seqs` to `--concurrency` when supplied. |
|
||||
| `max-num-batched-tokens` | Each scheduler iteration must share its token budget between active decodes and incoming prefills. | Input/output token shape, concurrency, and optional QPS/TPOT | Calculate decode budget + expected prefill demand, with vLLM scheduler constraints as floors. |
|
||||
| `data-parallel-size` | The required replica count depends on the requested throughput and the capacity of one replica. | Capacity target | Keep the recipe value today because per-replica SLA capacity is not known. |
|
||||
|
||||
## How Each Runtime Parameter Is Calculated
|
||||
|
||||
### `tensor-parallel-size`
|
||||
|
||||
TP is derived from the effective NUMA topology reported by hardware detection:
|
||||
|
||||
```text
|
||||
TP = largest power of two <= effective NUMA-node count
|
||||
```
|
||||
|
||||
For example, 2 effective NUMA nodes produce TP=2, while 6 nodes currently
|
||||
produce TP=4. This is a topology-based starting point and avoids automatically
|
||||
selecting unusual non-power-of-two TP sizes before they are validated.
|
||||
|
||||
### `gpu-memory-utilization`
|
||||
|
||||
For every effective NUMA memory node, the policy calculates:
|
||||
|
||||
```text
|
||||
node_safe_fraction = available_memory / total_memory - 0.10
|
||||
safe_fraction = min(0.90, minimum node_safe_fraction)
|
||||
```
|
||||
|
||||
The 10% reserve leaves memory for model/runtime overhead outside the vLLM cache
|
||||
budget. If the recipe already contains a smaller value, the policy preserves the
|
||||
smaller value. If it is absent, `0.80` is used as the initial ceiling:
|
||||
|
||||
```text
|
||||
candidate = min(recipe value or 0.80, safe_fraction)
|
||||
```
|
||||
|
||||
### `max-num-seqs`
|
||||
|
||||
When `--concurrency` is supplied, it directly represents the requested maximum
|
||||
number of simultaneously active requests:
|
||||
|
||||
```text
|
||||
max-num-seqs = concurrency
|
||||
```
|
||||
|
||||
If concurrency is not supplied, the converter does not override this parameter.
|
||||
|
||||
### `max-num-batched-tokens`
|
||||
|
||||
The token budget is workload-derived and no longer assumes a fixed number of
|
||||
parallel prefills.
|
||||
|
||||
First, the policy determines the active sequence count:
|
||||
|
||||
```text
|
||||
active_sequences =
|
||||
--concurrency
|
||||
or recipe max-num-seqs
|
||||
or vLLM default max_num_seqs (128)
|
||||
```
|
||||
|
||||
Decode requests consume approximately one token per active sequence in a
|
||||
scheduler iteration:
|
||||
|
||||
```text
|
||||
decode_budget = active_sequences
|
||||
```
|
||||
|
||||
The policy then estimates how many new prompts need prefill work per scheduler
|
||||
step:
|
||||
|
||||
```text
|
||||
prefills_per_step = max(
|
||||
1,
|
||||
active_sequences / output_tokens,
|
||||
target_qps * tpot_sla_ms / 1000
|
||||
)
|
||||
|
||||
prefills_per_step = min(active_sequences, prefills_per_step)
|
||||
prefill_budget = ceil(input_tokens * prefills_per_step)
|
||||
```
|
||||
|
||||
The final scheduler budget is:
|
||||
|
||||
```text
|
||||
max-num-batched-tokens = max(
|
||||
vLLM default max_num_batched_tokens (2048),
|
||||
active_sequences,
|
||||
decode_budget + prefill_budget
|
||||
)
|
||||
```
|
||||
|
||||
For example, with 2048 input tokens, 128 output tokens, and concurrency 32:
|
||||
|
||||
```text
|
||||
decode_budget = 32
|
||||
prefills_per_step = max(1, 32 / 128) = 1
|
||||
prefill_budget = 2048
|
||||
candidate = max(2048, 32, 32 + 2048) = 2080
|
||||
```
|
||||
|
||||
If chunked prefill is explicitly disabled and `max-model-len` is available, the
|
||||
policy also ensures:
|
||||
|
||||
```text
|
||||
max-num-batched-tokens >= max-model-len
|
||||
```
|
||||
|
||||
### `data-parallel-size`
|
||||
|
||||
`--target-qps` alone is not enough to choose DP safely. A correct capacity rule
|
||||
also needs measured per-replica throughput that still satisfies TTFT/TPOT:
|
||||
|
||||
```text
|
||||
DP = ceil(target_qps / qps_per_replica_at_SLO)
|
||||
```
|
||||
|
||||
Because the converter does not have that measured capacity yet, it intentionally
|
||||
keeps the recipe DP value rather than guessing.
|
||||
|
||||
## Precedence
|
||||
|
||||
```text
|
||||
vLLM defaults
|
||||
-> vLLM Recipes baseline
|
||||
-> hardware refinement (optional)
|
||||
-> workload / SLO refinement (optional)
|
||||
-> config.yml + env.sh
|
||||
```
|
||||
|
||||
## Runtime-Tuning Hardware Scope
|
||||
|
||||
Runtime tuning is selected from the resolved recipe JSON's `hardware` field,
|
||||
rather than by inspecting which physical devices happen to be present on the
|
||||
host. This matters because a GPU server also exposes its host CPU topology.
|
||||
|
||||
The current runtime-tuning policy registry contains `xeon6`. A tuning request
|
||||
for unregistered recipe hardware, such as `b200`, fails before host hardware
|
||||
detection:
|
||||
|
||||
```text
|
||||
ERROR: Runtime tuning is not supported for recipe hardware 'b200'.
|
||||
Currently supported: xeon6.
|
||||
```
|
||||
|
||||
Plain recipe conversion remains available for all recipe hardware. The gate only
|
||||
applies when optional runtime-tuning inputs are requested.
|
||||
|
||||
## vLLM CPU Docker Shell
|
||||
|
||||
Run hardware detection and runtime tuning inside the target vLLM CPU container
|
||||
so detected CPU, NUMA, memory, and cgroup limits match the deployment
|
||||
environment.
|
||||
|
||||
From the vLLM source tree:
|
||||
|
||||
```bash
|
||||
mkdir -p recipe-output
|
||||
|
||||
docker run --rm -it \
|
||||
--entrypoint bash \
|
||||
--security-opt seccomp=unconfined \
|
||||
--cap-add SYS_NICE \
|
||||
--shm-size=4g \
|
||||
-p 8000:8000 \
|
||||
-v "$PWD/tools/recipes:/recipes:ro" \
|
||||
-v "$PWD/recipe-output:/output" \
|
||||
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
|
||||
-w /output \
|
||||
vllm/vllm-openai-cpu:latest-x86_64
|
||||
```
|
||||
|
||||
`/output` is writable; `/recipes` remains read-only.
|
||||
|
||||
Inside the container, generate one runtime-tuned initial configuration:
|
||||
|
||||
```bash
|
||||
python3 /recipes/recipe_json_to_vllm_config.py \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--hardware xeon6 \
|
||||
--detect-hardware \
|
||||
--input-tokens 2048 \
|
||||
--output-tokens 128 \
|
||||
--concurrency 32 \
|
||||
--ttft-sla-ms 3000 \
|
||||
--tpot-sla-ms 100 \
|
||||
--config-out /output/config.yml \
|
||||
--env-out /output/env.sh
|
||||
```
|
||||
|
||||
Deploy the generated initial configuration:
|
||||
|
||||
```bash
|
||||
source /output/env.sh
|
||||
vllm serve --config /output/config.yml
|
||||
```
|
||||
|
||||
For optional benchmark-backed validation of the scheduler settings, continue
|
||||
with [SWEEP_TUNING.md](SWEEP_TUNING.md).
|
||||
|
||||
## Implementation
|
||||
|
||||
- `hardware_detection.py` collects effective CPU/NUMA/memory information.
|
||||
- `runtime_tuning.py` owns hardware-policy selection and independent parameter
|
||||
tuning functions.
|
||||
- `recipe_json_to_vllm_config.py` resolves the recipe, collects optional inputs,
|
||||
applies the selected policy, and generates `config.yml` and `env.sh`.
|
||||
|
||||
For benchmark-backed validation of the initial scheduler suggestion, see
|
||||
[SWEEP_TUNING.md](SWEEP_TUNING.md).
|
||||
@@ -0,0 +1,124 @@
|
||||
# Sweep Tuning
|
||||
|
||||
Sweep tuning is optional. The converter always creates one initial `config.yml`
|
||||
first. Users can deploy it immediately or benchmark nearby scheduler settings
|
||||
and generate one measured recommendation.
|
||||
|
||||
## Generate
|
||||
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--hardware xeon6 \
|
||||
--detect-hardware \
|
||||
--input-tokens 2048 \
|
||||
--output-tokens 128 \
|
||||
--concurrency 32 \
|
||||
--ttft-sla-ms 3000 \
|
||||
--tpot-sla-ms 100 \
|
||||
--generate-sweep
|
||||
```
|
||||
|
||||
The optional package is:
|
||||
|
||||
```text
|
||||
sweep/
|
||||
├── serve_params.json
|
||||
├── bench_params.json
|
||||
├── run_sweep.sh
|
||||
├── recommend.py
|
||||
└── SWEEP.md
|
||||
```
|
||||
|
||||
The sweep currently varies only `max-num-seqs` and
|
||||
`max-num-batched-tokens`.
|
||||
|
||||
## Run and Recommend
|
||||
|
||||
```bash
|
||||
sweep/run_sweep.sh --dry-run
|
||||
sweep/run_sweep.sh
|
||||
sweep/recommend.py
|
||||
```
|
||||
|
||||
Resume an interrupted sweep with:
|
||||
|
||||
```bash
|
||||
sweep/run_sweep.sh --resume
|
||||
```
|
||||
|
||||
`recommend.py` writes:
|
||||
|
||||
```text
|
||||
sweep/recommended-config.yml
|
||||
sweep/recommendation.json
|
||||
```
|
||||
|
||||
With TTFT/TPOT objectives, the benchmark uses vLLM `--goodput`, and the
|
||||
recommender selects highest mean request goodput across repeated runs. Without
|
||||
latency objectives it selects highest mean output-token throughput. Failed
|
||||
benchmark configurations are excluded.
|
||||
|
||||
`recommended-config.yml` changes only the two swept scheduler parameters.
|
||||
`recommendation.json` records the measured evidence.
|
||||
|
||||
Deploy:
|
||||
|
||||
```bash
|
||||
source env.sh
|
||||
vllm serve --config sweep/recommended-config.yml
|
||||
```
|
||||
|
||||
## vLLM CPU Docker Shell
|
||||
|
||||
Use the common CPU container setup in
|
||||
[RUNTIME_TUNING.md](RUNTIME_TUNING.md#vllm-cpu-docker-shell). It ensures
|
||||
hardware detection runs against the same effective CPU, NUMA, memory, and
|
||||
cgroup limits used by the deployment.
|
||||
|
||||
Inside that container, generate the initial suggestion plus sweep package:
|
||||
|
||||
```bash
|
||||
python3 /recipes/recipe_json_to_vllm_config.py \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--hardware xeon6 \
|
||||
--detect-hardware \
|
||||
--input-tokens 2048 \
|
||||
--output-tokens 128 \
|
||||
--concurrency 32 \
|
||||
--ttft-sla-ms 3000 \
|
||||
--tpot-sla-ms 100 \
|
||||
--config-out /output/config.yml \
|
||||
--env-out /output/env.sh \
|
||||
--generate-sweep \
|
||||
--sweep-out-dir /output/sweep
|
||||
```
|
||||
|
||||
The initial configuration can be deployed directly:
|
||||
|
||||
```bash
|
||||
source /output/env.sh
|
||||
vllm serve --config /output/config.yml
|
||||
```
|
||||
|
||||
Stop that manually started server before running the sweep, because
|
||||
`run_sweep.sh` starts and stops its own vLLM servers.
|
||||
|
||||
```bash
|
||||
/output/sweep/run_sweep.sh --dry-run
|
||||
/output/sweep/run_sweep.sh
|
||||
/output/sweep/recommend.py
|
||||
```
|
||||
|
||||
Inspect:
|
||||
|
||||
```bash
|
||||
cat /output/sweep/recommendation.json
|
||||
```
|
||||
|
||||
Deploy the measured result:
|
||||
|
||||
```bash
|
||||
source /output/env.sh
|
||||
vllm serve --config /output/sweep/recommended-config.yml
|
||||
```
|
||||
@@ -0,0 +1,115 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
"""CPU hardware discovery for optional vLLM Recipes runtime tuning.
|
||||
|
||||
Keep this module focused on facts exposed to the current process/container.
|
||||
Tuning decisions belong in runtime_tuning.py.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import platform
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from vllm.utils.cpu_resource_utils import (
|
||||
get_allowed_cpu_list,
|
||||
get_memory_affinity,
|
||||
get_memory_node_info,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class NumaNodeInfo:
|
||||
node_id: int
|
||||
logical_cpu_ids: tuple[int, ...]
|
||||
physical_core_count: int
|
||||
total_memory_bytes: int
|
||||
available_memory_bytes: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class HardwareInfo:
|
||||
architecture: str
|
||||
socket_count: int | None
|
||||
numa_nodes: tuple[NumaNodeInfo, ...]
|
||||
allowed_logical_cpu_count: int
|
||||
physical_core_count: int
|
||||
|
||||
@property
|
||||
def numa_node_count(self) -> int:
|
||||
return len(self.numa_nodes)
|
||||
|
||||
def to_dict(self) -> dict[str, object]:
|
||||
return asdict(self)
|
||||
|
||||
|
||||
def _read_socket_id(cpu_id: int) -> int | None:
|
||||
path = Path(f"/sys/devices/system/cpu/cpu{cpu_id}/topology/physical_package_id")
|
||||
try:
|
||||
return int(path.read_text().strip())
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def detect_hardware() -> HardwareInfo:
|
||||
"""Return CPU resources effectively available to this process.
|
||||
|
||||
vLLM's CPU helpers already honor process CPU affinity and cgroup-aware
|
||||
memory limits. We intentionally use get_memory_affinity() rather than
|
||||
get_visible_memory_node() so CPU_VISIBLE_MEMORY_NODES does not become an
|
||||
input to the detector; auto binding remains owned by the CPU backend.
|
||||
"""
|
||||
cpus = get_allowed_cpu_list()
|
||||
if not cpus:
|
||||
raise RuntimeError("vLLM reported no CPUs available to this process.")
|
||||
|
||||
cpu_numa_nodes = {cpu.numa_node for cpu in cpus if cpu.numa_node >= 0}
|
||||
memory_numa_nodes = set(get_memory_affinity())
|
||||
effective_numa_nodes = sorted(cpu_numa_nodes & memory_numa_nodes)
|
||||
if not effective_numa_nodes:
|
||||
effective_numa_nodes = sorted(cpu_numa_nodes)
|
||||
|
||||
physical_cores = {
|
||||
(cpu.numa_node, cpu.physical_core) for cpu in cpus if cpu.physical_core >= 0
|
||||
}
|
||||
|
||||
numa_nodes = []
|
||||
for node_id in effective_numa_nodes:
|
||||
node_cpus = [cpu for cpu in cpus if cpu.numa_node == node_id]
|
||||
node_physical_cores = {
|
||||
cpu.physical_core for cpu in node_cpus if cpu.physical_core >= 0
|
||||
}
|
||||
memory = get_memory_node_info(node_id)
|
||||
numa_nodes.append(
|
||||
NumaNodeInfo(
|
||||
node_id=node_id,
|
||||
logical_cpu_ids=tuple(sorted(cpu.id for cpu in node_cpus)),
|
||||
physical_core_count=len(node_physical_cores),
|
||||
total_memory_bytes=memory.total_memory,
|
||||
available_memory_bytes=memory.available_memory,
|
||||
)
|
||||
)
|
||||
|
||||
socket_ids = {
|
||||
socket_id for cpu in cpus if (socket_id := _read_socket_id(cpu.id)) is not None
|
||||
}
|
||||
|
||||
return HardwareInfo(
|
||||
architecture=platform.machine(),
|
||||
socket_count=len(socket_ids) if socket_ids else None,
|
||||
numa_nodes=tuple(numa_nodes),
|
||||
allowed_logical_cpu_count=len(cpus),
|
||||
physical_core_count=len(physical_cores),
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
print(json.dumps(detect_hardware().to_dict(), indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -57,7 +57,6 @@ try:
|
||||
except ImportError as exc:
|
||||
raise SystemExit("PyYAML is required. Install it with: pip install pyyaml") from exc
|
||||
|
||||
|
||||
DEFAULT_API_BASE = "https://recipes.vllm.ai"
|
||||
|
||||
SHORT_ALIASES = {
|
||||
@@ -117,6 +116,71 @@ def parse_args() -> argparse.Namespace:
|
||||
default="env.sh",
|
||||
help="Output shell environment file (default: env.sh)",
|
||||
)
|
||||
|
||||
tuning = p.add_argument_group(
|
||||
"optional runtime tuning",
|
||||
(
|
||||
"Refine the recipe baseline only when additional information is supplied. "
|
||||
"Runtime tuning is enabled only for recipe hardware with a registered "
|
||||
"policy (currently: xeon6)."
|
||||
),
|
||||
)
|
||||
tuning.add_argument(
|
||||
"--detect-hardware",
|
||||
action="store_true",
|
||||
help=(
|
||||
"Detect effective CPU/NUMA/memory resources and allow the selected "
|
||||
"recipe-hardware policy to override recipe runtime arguments."
|
||||
),
|
||||
)
|
||||
tuning.add_argument(
|
||||
"--input-tokens",
|
||||
type=int,
|
||||
help="Expected input-token length. Optional workload hint.",
|
||||
)
|
||||
tuning.add_argument(
|
||||
"--output-tokens",
|
||||
type=int,
|
||||
help="Expected output-token length. Optional workload hint.",
|
||||
)
|
||||
tuning.add_argument(
|
||||
"--concurrency",
|
||||
type=int,
|
||||
help="Expected maximum concurrent requests. Optional workload hint.",
|
||||
)
|
||||
tuning.add_argument(
|
||||
"--ttft-sla-ms",
|
||||
type=float,
|
||||
help="Optional time-to-first-token objective in milliseconds.",
|
||||
)
|
||||
tuning.add_argument(
|
||||
"--tpot-sla-ms",
|
||||
type=float,
|
||||
help="Optional time-per-output-token objective in milliseconds.",
|
||||
)
|
||||
tuning.add_argument(
|
||||
"--target-qps",
|
||||
type=float,
|
||||
help="Optional capacity target for future DP/capacity tuning.",
|
||||
)
|
||||
|
||||
sweep = p.add_argument_group(
|
||||
"optional performance sweep",
|
||||
("Generate benchmark files after creating one initial runtime suggestion."),
|
||||
)
|
||||
sweep.add_argument(
|
||||
"--generate-sweep",
|
||||
action="store_true",
|
||||
help=(
|
||||
"Generate an optional vllm bench sweep package around the single "
|
||||
"initial runtime suggestion."
|
||||
),
|
||||
)
|
||||
sweep.add_argument(
|
||||
"--sweep-out-dir",
|
||||
default="sweep",
|
||||
help="Output directory for optional sweep files (default: sweep).",
|
||||
)
|
||||
return p.parse_args()
|
||||
|
||||
|
||||
@@ -642,18 +706,115 @@ def main() -> int:
|
||||
|
||||
argv = recipe_argv(recipe)
|
||||
config = argv_to_config(argv)
|
||||
|
||||
tuning_requested = (
|
||||
args.detect_hardware
|
||||
or args.generate_sweep
|
||||
or any(
|
||||
value is not None
|
||||
for value in (
|
||||
args.input_tokens,
|
||||
args.output_tokens,
|
||||
args.concurrency,
|
||||
args.ttft_sla_ms,
|
||||
args.tpot_sla_ms,
|
||||
args.target_qps,
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
tuning = None
|
||||
workload = None
|
||||
sweep_writer = None
|
||||
if tuning_requested:
|
||||
# Keep plain Recipes conversion lightweight. vLLM-specific modules
|
||||
# are imported only for optional runtime tuning or sweep generation.
|
||||
from runtime_tuning import (
|
||||
WorkloadHints,
|
||||
finetune_runtime_config,
|
||||
get_runtime_tuning_policies,
|
||||
)
|
||||
|
||||
workload = WorkloadHints(
|
||||
input_tokens=args.input_tokens,
|
||||
output_tokens=args.output_tokens,
|
||||
concurrency=args.concurrency,
|
||||
ttft_sla_ms=args.ttft_sla_ms,
|
||||
tpot_sla_ms=args.tpot_sla_ms,
|
||||
target_qps=args.target_qps,
|
||||
)
|
||||
|
||||
if args.generate_sweep:
|
||||
from sweep_generation import (
|
||||
validate_sweep_workload,
|
||||
write_sweep_files,
|
||||
)
|
||||
|
||||
validate_sweep_workload(workload)
|
||||
sweep_writer = write_sweep_files
|
||||
|
||||
recipe_hardware = recipe.get("hardware")
|
||||
policies = get_runtime_tuning_policies(recipe_hardware)
|
||||
|
||||
hardware = None
|
||||
if args.detect_hardware:
|
||||
from hardware_detection import detect_hardware
|
||||
|
||||
hardware = detect_hardware()
|
||||
|
||||
tuning = finetune_runtime_config(
|
||||
config,
|
||||
hardware=hardware,
|
||||
workload=workload,
|
||||
policies=policies,
|
||||
)
|
||||
config.update(tuning.overrides)
|
||||
|
||||
write_config(args.config_out, source, recipe, config)
|
||||
write_env(args.env_out, source, recipe)
|
||||
|
||||
sweep_files: list[Path] = []
|
||||
if args.generate_sweep:
|
||||
assert workload is not None
|
||||
assert sweep_writer is not None
|
||||
sweep_files = sweep_writer(
|
||||
args.sweep_out_dir,
|
||||
config_path=args.config_out,
|
||||
env_path=args.env_out,
|
||||
config=config,
|
||||
workload=workload,
|
||||
)
|
||||
|
||||
if tuning is not None:
|
||||
if tuning.overrides:
|
||||
print("Initial runtime suggestion:")
|
||||
for key, value in tuning.overrides.items():
|
||||
print(f" {key}: {value}")
|
||||
for note in tuning.notes:
|
||||
print(f" tuning: {note}")
|
||||
except Exception as exc:
|
||||
print(f"ERROR: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
print(f"Wrote {args.config_out}")
|
||||
print(f"Wrote {args.env_out}")
|
||||
if sweep_files:
|
||||
print(f"Wrote optional sweep package under {args.sweep_out_dir}/")
|
||||
print()
|
||||
print("Run:")
|
||||
print(f" source {shlex.quote(args.env_out)}")
|
||||
print(f" vllm serve --config {shlex.quote(args.config_out)}")
|
||||
if sweep_files:
|
||||
sweep_dir = Path(args.sweep_out_dir)
|
||||
run_sweep = sweep_dir / "run_sweep.sh"
|
||||
recommend = sweep_dir / "recommend.py"
|
||||
print()
|
||||
print("Optional performance sweep:")
|
||||
print(f" {shlex.quote(str(run_sweep))} --dry-run")
|
||||
print(f" {shlex.quote(str(run_sweep))}")
|
||||
print()
|
||||
print("After the sweep:")
|
||||
print(f" {shlex.quote(str(recommend))}")
|
||||
return 0
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,314 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
"""Optional runtime refinement for configs produced from vLLM Recipes.
|
||||
|
||||
The recipe remains the baseline. Each policy may return an override only when
|
||||
enough information is available. Keep policies independent so the tuning
|
||||
algorithms can evolve without changing the converter or hardware detector.
|
||||
|
||||
The policies in this draft are intentionally simple heuristics, not benchmark-
|
||||
proven universal optima.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass, field
|
||||
from math import ceil
|
||||
from typing import Any
|
||||
|
||||
from hardware_detection import HardwareInfo
|
||||
|
||||
from vllm.config.scheduler import SchedulerConfig
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class WorkloadHints:
|
||||
input_tokens: int | None = None
|
||||
output_tokens: int | None = None
|
||||
concurrency: int | None = None
|
||||
ttft_sla_ms: float | None = None
|
||||
tpot_sla_ms: float | None = None
|
||||
target_qps: float | None = None
|
||||
|
||||
def validate(self) -> None:
|
||||
for name in ("input_tokens", "output_tokens", "concurrency"):
|
||||
value = getattr(self, name)
|
||||
if value is not None and value <= 0:
|
||||
raise ValueError(f"{name} must be greater than zero")
|
||||
|
||||
for name in ("ttft_sla_ms", "tpot_sla_ms", "target_qps"):
|
||||
value = getattr(self, name)
|
||||
if value is not None and value <= 0:
|
||||
raise ValueError(f"{name} must be greater than zero")
|
||||
|
||||
|
||||
@dataclass
|
||||
class TuningResult:
|
||||
overrides: dict[str, Any] = field(default_factory=dict)
|
||||
notes: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
Policy = Callable[
|
||||
[dict[str, Any], HardwareInfo | None, WorkloadHints, TuningResult],
|
||||
None,
|
||||
]
|
||||
|
||||
|
||||
def _record_override(
|
||||
result: TuningResult,
|
||||
key: str,
|
||||
value: Any,
|
||||
reason: str,
|
||||
) -> None:
|
||||
result.overrides[key] = value
|
||||
result.notes.append(f"{key}={value!r}: {reason}")
|
||||
|
||||
|
||||
def _get_active_sequence_count(
|
||||
config: dict[str, Any],
|
||||
workload: WorkloadHints,
|
||||
) -> int:
|
||||
"""Return the sequence count used to size one scheduler iteration."""
|
||||
if workload.concurrency is not None:
|
||||
return workload.concurrency
|
||||
|
||||
configured = config.get("max-num-seqs")
|
||||
if (
|
||||
isinstance(configured, int)
|
||||
and not isinstance(configured, bool)
|
||||
and configured > 0
|
||||
):
|
||||
return configured
|
||||
|
||||
return SchedulerConfig.DEFAULT_MAX_NUM_SEQS
|
||||
|
||||
|
||||
def _estimate_prefills_per_step(
|
||||
workload: WorkloadHints,
|
||||
active_sequences: int,
|
||||
) -> float:
|
||||
"""Estimate steady-state prompt arrivals per decode scheduler step."""
|
||||
# Reserve enough prefill budget for at least one representative prompt.
|
||||
prefills_per_step = 1.0
|
||||
|
||||
# In steady state, active_sequences / output_tokens approximates how many
|
||||
# requests finish and are replaced by new prefills per decode step.
|
||||
if workload.output_tokens is not None:
|
||||
prefills_per_step = max(
|
||||
prefills_per_step,
|
||||
active_sequences / workload.output_tokens,
|
||||
)
|
||||
|
||||
# With target QPS + TPOT, approximate how many prompts arrive during one
|
||||
# decode scheduler-step interval.
|
||||
if workload.target_qps is not None and workload.tpot_sla_ms is not None:
|
||||
prefills_per_step = max(
|
||||
prefills_per_step,
|
||||
workload.target_qps * workload.tpot_sla_ms / 1000.0,
|
||||
)
|
||||
|
||||
# A step cannot replace more requests than are active.
|
||||
return min(float(active_sequences), prefills_per_step)
|
||||
|
||||
|
||||
def _resolve_tensor_parallel_size(
|
||||
config: dict[str, Any],
|
||||
hardware: HardwareInfo | None,
|
||||
workload: WorkloadHints,
|
||||
result: TuningResult,
|
||||
) -> None:
|
||||
del config, workload
|
||||
if hardware is None or hardware.numa_node_count < 1:
|
||||
return
|
||||
|
||||
# Draft policy: use the largest power-of-two TP that does not exceed the
|
||||
# effective NUMA-node count. This deliberately avoids hard-coding unusual
|
||||
# non-power-of-two topologies into the converter.
|
||||
candidate = 1 << (hardware.numa_node_count.bit_length() - 1)
|
||||
_record_override(
|
||||
result,
|
||||
"tensor-parallel-size",
|
||||
candidate,
|
||||
"derived from the effective NUMA-node count",
|
||||
)
|
||||
|
||||
|
||||
def _resolve_gpu_memory_utilization(
|
||||
config: dict[str, Any],
|
||||
hardware: HardwareInfo | None,
|
||||
workload: WorkloadHints,
|
||||
result: TuningResult,
|
||||
) -> None:
|
||||
del workload
|
||||
if hardware is None or not hardware.numa_nodes:
|
||||
return
|
||||
|
||||
reserve_fraction = 0.10
|
||||
safe_fractions = [
|
||||
max(
|
||||
0.0,
|
||||
node.available_memory_bytes / node.total_memory_bytes - reserve_fraction,
|
||||
)
|
||||
for node in hardware.numa_nodes
|
||||
if node.total_memory_bytes > 0
|
||||
]
|
||||
if not safe_fractions:
|
||||
return
|
||||
|
||||
safe_fraction = round(min(0.90, min(safe_fractions)), 2)
|
||||
if safe_fraction <= 0:
|
||||
return
|
||||
|
||||
recipe_value = config.get("gpu-memory-utilization")
|
||||
if isinstance(recipe_value, (int, float)):
|
||||
candidate = min(float(recipe_value), safe_fraction)
|
||||
else:
|
||||
candidate = min(0.80, safe_fraction)
|
||||
|
||||
_record_override(
|
||||
result,
|
||||
"gpu-memory-utilization",
|
||||
round(candidate, 2),
|
||||
"constrained by effective per-NUMA available memory",
|
||||
)
|
||||
|
||||
|
||||
def _resolve_max_num_seqs(
|
||||
config: dict[str, Any],
|
||||
hardware: HardwareInfo | None,
|
||||
workload: WorkloadHints,
|
||||
result: TuningResult,
|
||||
) -> None:
|
||||
del config, hardware
|
||||
if workload.concurrency is None:
|
||||
return
|
||||
|
||||
_record_override(
|
||||
result,
|
||||
"max-num-seqs",
|
||||
workload.concurrency,
|
||||
"derived from the optional target concurrency",
|
||||
)
|
||||
|
||||
|
||||
def _resolve_max_num_batched_tokens(
|
||||
config: dict[str, Any],
|
||||
hardware: HardwareInfo | None,
|
||||
workload: WorkloadHints,
|
||||
result: TuningResult,
|
||||
) -> None:
|
||||
del hardware
|
||||
if workload.input_tokens is None:
|
||||
return
|
||||
|
||||
active_sequences = _get_active_sequence_count(config, workload)
|
||||
|
||||
# Decode requests consume roughly one token per active sequence in one
|
||||
# scheduler iteration. Prefills share the remaining scheduler token budget.
|
||||
decode_budget = active_sequences
|
||||
prefills_per_step = _estimate_prefills_per_step(workload, active_sequences)
|
||||
prefill_budget = ceil(workload.input_tokens * prefills_per_step)
|
||||
|
||||
# Use vLLM's scheduler defaults/constraints as floors rather than local
|
||||
# recipe-specific magic constants.
|
||||
candidate = max(
|
||||
SchedulerConfig.DEFAULT_MAX_NUM_BATCHED_TOKENS,
|
||||
active_sequences,
|
||||
decode_budget + prefill_budget,
|
||||
)
|
||||
|
||||
# Chunked prefill normally lets a prompt span scheduler iterations. If the
|
||||
# recipe explicitly disables it, vLLM requires the token budget to cover
|
||||
# max_model_len when max_model_len is known.
|
||||
if config.get("enable-chunked-prefill") is False:
|
||||
max_model_len = config.get("max-model-len")
|
||||
if (
|
||||
isinstance(max_model_len, int)
|
||||
and not isinstance(max_model_len, bool)
|
||||
and max_model_len > 0
|
||||
):
|
||||
candidate = max(candidate, max_model_len)
|
||||
|
||||
_record_override(
|
||||
result,
|
||||
"max-num-batched-tokens",
|
||||
candidate,
|
||||
(
|
||||
"workload-derived scheduler budget "
|
||||
f"(active_sequences={active_sequences}, "
|
||||
f"prefills_per_step={prefills_per_step:.2f}, "
|
||||
f"decode_budget={decode_budget}, "
|
||||
f"prefill_budget={prefill_budget})"
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _resolve_data_parallel_size(
|
||||
config: dict[str, Any],
|
||||
hardware: HardwareInfo | None,
|
||||
workload: WorkloadHints,
|
||||
result: TuningResult,
|
||||
) -> None:
|
||||
del config, hardware
|
||||
if workload.target_qps is None:
|
||||
return
|
||||
|
||||
result.notes.append(
|
||||
"data-parallel-size: target QPS supplied; kept recipe value because "
|
||||
"per-replica capacity is not known yet"
|
||||
)
|
||||
|
||||
|
||||
DEFAULT_POLICIES: tuple[Policy, ...] = (
|
||||
_resolve_tensor_parallel_size,
|
||||
_resolve_gpu_memory_utilization,
|
||||
_resolve_max_num_seqs,
|
||||
_resolve_max_num_batched_tokens,
|
||||
_resolve_data_parallel_size,
|
||||
)
|
||||
|
||||
# Select tuning behavior from the hardware requested by the Recipes rendering,
|
||||
# not from devices physically present in the host. A GPU server still exposes
|
||||
# its host CPU topology, so host inspection alone cannot determine deployment
|
||||
# intent.
|
||||
HARDWARE_TUNING_POLICIES: dict[str, tuple[Policy, ...]] = {
|
||||
"xeon6": DEFAULT_POLICIES,
|
||||
}
|
||||
|
||||
|
||||
def get_runtime_tuning_policies(
|
||||
recipe_hardware: object,
|
||||
) -> tuple[Policy, ...]:
|
||||
if not isinstance(recipe_hardware, str) or not recipe_hardware:
|
||||
raise ValueError(
|
||||
"Runtime tuning requires the resolved Recipes JSON to declare "
|
||||
"a non-empty `hardware` field."
|
||||
)
|
||||
|
||||
policies = HARDWARE_TUNING_POLICIES.get(recipe_hardware)
|
||||
if policies is None:
|
||||
supported = ", ".join(sorted(HARDWARE_TUNING_POLICIES))
|
||||
raise ValueError(
|
||||
"Runtime tuning is not supported for recipe hardware "
|
||||
f"{recipe_hardware!r}. Currently supported: {supported}."
|
||||
)
|
||||
return policies
|
||||
|
||||
|
||||
def finetune_runtime_config(
|
||||
config: dict[str, Any],
|
||||
*,
|
||||
hardware: HardwareInfo | None = None,
|
||||
workload: WorkloadHints | None = None,
|
||||
policies: tuple[Policy, ...] = DEFAULT_POLICIES,
|
||||
) -> TuningResult:
|
||||
"""Return overrides; never mutate the recipe-derived config in place."""
|
||||
hints = workload or WorkloadHints()
|
||||
hints.validate()
|
||||
|
||||
result = TuningResult()
|
||||
for policy in policies:
|
||||
policy(config, hardware, hints, result)
|
||||
return result
|
||||
@@ -0,0 +1,341 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
"""Generate optional vLLM benchmark sweep files from one initial suggestion."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import shlex
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from runtime_tuning import WorkloadHints
|
||||
|
||||
|
||||
def _positive_int(config: dict[str, Any], key: str) -> int:
|
||||
value = config.get(key)
|
||||
if not isinstance(value, int) or isinstance(value, bool) or value <= 0:
|
||||
raise ValueError(f"Sweep generation requires a positive initial {key!r} value.")
|
||||
return value
|
||||
|
||||
|
||||
def _strict_lower_power_of_two(value: int) -> int:
|
||||
if value <= 1:
|
||||
return 1
|
||||
return 1 << ((value - 1).bit_length() - 1)
|
||||
|
||||
|
||||
def _strict_upper_power_of_two(value: int) -> int:
|
||||
return 1 << value.bit_length()
|
||||
|
||||
|
||||
def validate_sweep_workload(workload: WorkloadHints) -> None:
|
||||
required = {
|
||||
"--input-tokens": workload.input_tokens,
|
||||
"--output-tokens": workload.output_tokens,
|
||||
"--concurrency": workload.concurrency,
|
||||
}
|
||||
missing = [name for name, value in required.items() if value is None]
|
||||
if missing:
|
||||
raise ValueError("--generate-sweep requires " + ", ".join(missing) + ".")
|
||||
|
||||
|
||||
def build_serve_params(config: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
"""Build a small directed sweep around the single initial suggestion."""
|
||||
initial_seqs = _positive_int(config, "max-num-seqs")
|
||||
initial_batch = _positive_int(config, "max-num-batched-tokens")
|
||||
|
||||
minimum_batch = initial_seqs
|
||||
if config.get("enable-chunked-prefill") is False:
|
||||
max_model_len = config.get("max-model-len")
|
||||
if (
|
||||
isinstance(max_model_len, int)
|
||||
and not isinstance(max_model_len, bool)
|
||||
and max_model_len > 0
|
||||
):
|
||||
minimum_batch = max(minimum_batch, max_model_len)
|
||||
|
||||
lower_seqs = max(1, (initial_seqs + 1) // 2)
|
||||
lower_batch = max(
|
||||
minimum_batch,
|
||||
_strict_lower_power_of_two(initial_batch),
|
||||
)
|
||||
higher_batch = max(
|
||||
minimum_batch,
|
||||
_strict_upper_power_of_two(initial_batch),
|
||||
)
|
||||
|
||||
candidates: list[dict[str, Any]] = []
|
||||
seen: set[tuple[int, int]] = set()
|
||||
|
||||
def add(name: str, max_num_seqs: int, max_num_batched_tokens: int) -> None:
|
||||
max_num_batched_tokens = max(
|
||||
max_num_batched_tokens,
|
||||
max_num_seqs,
|
||||
)
|
||||
signature = (max_num_seqs, max_num_batched_tokens)
|
||||
if signature in seen:
|
||||
return
|
||||
seen.add(signature)
|
||||
candidates.append(
|
||||
{
|
||||
"_benchmark_name": name,
|
||||
"max_num_seqs": max_num_seqs,
|
||||
"max_num_batched_tokens": max_num_batched_tokens,
|
||||
}
|
||||
)
|
||||
|
||||
# Keep the exact initial suggestion as the measured baseline. Additional
|
||||
# values exist only in the optional sweep package.
|
||||
add("initial", initial_seqs, initial_batch)
|
||||
add("lower_scheduler_concurrency", lower_seqs, initial_batch)
|
||||
add("lower_batch_budget", initial_seqs, lower_batch)
|
||||
add("latency_focused", lower_seqs, lower_batch)
|
||||
add("higher_batch_budget", initial_seqs, higher_batch)
|
||||
|
||||
return candidates
|
||||
|
||||
|
||||
def build_bench_params(workload: WorkloadHints) -> list[dict[str, Any]]:
|
||||
validate_sweep_workload(workload)
|
||||
assert workload.input_tokens is not None
|
||||
assert workload.output_tokens is not None
|
||||
assert workload.concurrency is not None
|
||||
|
||||
return [
|
||||
{
|
||||
"_benchmark_name": "user_workload",
|
||||
"random_input_len": workload.input_tokens,
|
||||
"random_output_len": workload.output_tokens,
|
||||
"max_concurrency": workload.concurrency,
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def _benchmark_models(config: dict[str, Any]) -> tuple[str, str]:
|
||||
model = config.get("model")
|
||||
if not isinstance(model, str) or not model:
|
||||
raise ValueError("Sweep generation requires a model in config.yml.")
|
||||
|
||||
served_model_name = config.get("served-model-name")
|
||||
if isinstance(served_model_name, str) and served_model_name:
|
||||
return served_model_name, model
|
||||
if (
|
||||
isinstance(served_model_name, list)
|
||||
and served_model_name
|
||||
and isinstance(served_model_name[0], str)
|
||||
):
|
||||
return served_model_name[0], model
|
||||
|
||||
return model, model
|
||||
|
||||
|
||||
def _relative_to(directory: Path, target: str) -> str:
|
||||
return os.path.relpath(Path(target).resolve(), directory.resolve())
|
||||
|
||||
|
||||
def _write_json(path: Path, value: object) -> None:
|
||||
path.write_text(
|
||||
json.dumps(value, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def _write_run_script(
|
||||
path: Path,
|
||||
*,
|
||||
config_rel: str,
|
||||
env_rel: str,
|
||||
request_model: str,
|
||||
tokenizer: str,
|
||||
workload: WorkloadHints,
|
||||
) -> None:
|
||||
bench_parts = [
|
||||
"vllm bench serve",
|
||||
"--backend vllm",
|
||||
f"--model {shlex.quote(request_model)}",
|
||||
f"--tokenizer {shlex.quote(tokenizer)}",
|
||||
"--dataset-name random",
|
||||
"--request-rate inf",
|
||||
"--ignore-eos",
|
||||
"--metric-percentiles 99",
|
||||
]
|
||||
|
||||
goodput_pairs: list[str] = []
|
||||
if workload.ttft_sla_ms is not None:
|
||||
goodput_pairs.append(f"ttft:{workload.ttft_sla_ms:g}")
|
||||
if workload.tpot_sla_ms is not None:
|
||||
goodput_pairs.append(f"tpot:{workload.tpot_sla_ms:g}")
|
||||
if goodput_pairs:
|
||||
bench_parts.append(
|
||||
"--goodput " + " ".join(shlex.quote(pair) for pair in goodput_pairs)
|
||||
)
|
||||
|
||||
bench_cmd = " ".join(bench_parts)
|
||||
script = f"""#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${{BASH_SOURCE[0]}}")" && pwd)"
|
||||
CONFIG_PATH="${{SCRIPT_DIR}}/{config_rel}"
|
||||
ENV_PATH="${{SCRIPT_DIR}}/{env_rel}"
|
||||
|
||||
source "${{ENV_PATH}}"
|
||||
|
||||
vllm bench sweep serve \
|
||||
--serve-cmd "vllm serve --config '${{CONFIG_PATH}}'" \
|
||||
--bench-cmd "{bench_cmd}" \
|
||||
--serve-params "${{SCRIPT_DIR}}/serve_params.json" \
|
||||
--bench-params "${{SCRIPT_DIR}}/bench_params.json" \
|
||||
--output-dir "${{SCRIPT_DIR}}/results" \
|
||||
--experiment-name runtime-tuning \
|
||||
"$@"
|
||||
"""
|
||||
path.write_text(script, encoding="utf-8")
|
||||
path.chmod(path.stat().st_mode | 0o111)
|
||||
|
||||
|
||||
def _write_recommend_script(
|
||||
path: Path,
|
||||
*,
|
||||
config_rel: str,
|
||||
env_rel: str,
|
||||
workload: WorkloadHints,
|
||||
) -> None:
|
||||
template_path = Path(__file__).with_name("sweep_recommendation.py")
|
||||
source = template_path.read_text(encoding="utf-8")
|
||||
|
||||
replacements = {
|
||||
"DEFAULT_CONFIG_PATH: str | None = None": (
|
||||
f"DEFAULT_CONFIG_PATH: str | None = {config_rel!r}"
|
||||
),
|
||||
"DEFAULT_ENV_PATH: str | None = None": (
|
||||
f"DEFAULT_ENV_PATH: str | None = {env_rel!r}"
|
||||
),
|
||||
"DEFAULT_TTFT_SLA_MS: float | None = None": (
|
||||
f"DEFAULT_TTFT_SLA_MS: float | None = {workload.ttft_sla_ms!r}"
|
||||
),
|
||||
"DEFAULT_TPOT_SLA_MS: float | None = None": (
|
||||
f"DEFAULT_TPOT_SLA_MS: float | None = {workload.tpot_sla_ms!r}"
|
||||
),
|
||||
}
|
||||
for marker, replacement in replacements.items():
|
||||
if marker not in source:
|
||||
raise ValueError(f"Recommender template marker not found: {marker}")
|
||||
source = source.replace(marker, replacement, 1)
|
||||
|
||||
path.write_text(source, encoding="utf-8")
|
||||
path.chmod(path.stat().st_mode | 0o111)
|
||||
|
||||
|
||||
def _write_guide(path: Path, workload: WorkloadHints) -> None:
|
||||
sla_lines: list[str] = []
|
||||
if workload.ttft_sla_ms is not None:
|
||||
sla_lines.append(f"- TTFT objective: `{workload.ttft_sla_ms:g} ms`")
|
||||
if workload.tpot_sla_ms is not None:
|
||||
sla_lines.append(f"- TPOT objective: `{workload.tpot_sla_ms:g} ms`")
|
||||
|
||||
sla_text = "\n".join(sla_lines)
|
||||
if sla_text:
|
||||
sla_text = (
|
||||
"\n## Supplied latency objectives\n\n"
|
||||
+ sla_text
|
||||
+ "\n\nThe generated benchmark uses these values with vLLM "
|
||||
"`--goodput`.\n"
|
||||
)
|
||||
|
||||
content = f"""# Optional Runtime Tuning Sweep
|
||||
|
||||
The generated `config.yml` is the **single initial suggestion** and can be
|
||||
deployed directly. The sweep benchmarks nearby values for:
|
||||
|
||||
- `max-num-seqs`
|
||||
- `max-num-batched-tokens`
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
./run_sweep.sh --dry-run
|
||||
./run_sweep.sh
|
||||
./recommend.py
|
||||
```
|
||||
|
||||
For a quick one-run experiment:
|
||||
|
||||
```bash
|
||||
rm -rf results/runtime-tuning
|
||||
./run_sweep.sh --num-runs 1
|
||||
./recommend.py
|
||||
```
|
||||
|
||||
Resume an interrupted sweep:
|
||||
|
||||
```bash
|
||||
./run_sweep.sh --resume
|
||||
```
|
||||
|
||||
By default, vLLM benchmarks each parameter combination three times.
|
||||
{sla_text}
|
||||
## Outputs
|
||||
|
||||
`recommend.py` writes:
|
||||
|
||||
```text
|
||||
recommended-config.yml
|
||||
recommendation.json
|
||||
```
|
||||
|
||||
With TTFT/TPOT objectives it selects highest mean request goodput across
|
||||
repeated runs. Without latency objectives it selects highest mean output-token
|
||||
throughput. Configurations with failed requests are excluded.
|
||||
|
||||
`recommended-config.yml` copies the initial configuration and changes only
|
||||
`max-num-seqs` and `max-num-batched-tokens`.
|
||||
"""
|
||||
path.write_text(content, encoding="utf-8")
|
||||
|
||||
|
||||
def write_sweep_files(
|
||||
output_dir: str,
|
||||
*,
|
||||
config_path: str,
|
||||
env_path: str,
|
||||
config: dict[str, Any],
|
||||
workload: WorkloadHints,
|
||||
) -> list[Path]:
|
||||
"""Write an optional sweep package around the initial config suggestion."""
|
||||
validate_sweep_workload(workload)
|
||||
|
||||
directory = Path(output_dir)
|
||||
directory.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
serve_params = directory / "serve_params.json"
|
||||
bench_params = directory / "bench_params.json"
|
||||
run_script = directory / "run_sweep.sh"
|
||||
recommend_script = directory / "recommend.py"
|
||||
guide = directory / "SWEEP.md"
|
||||
|
||||
config_rel = _relative_to(directory, config_path)
|
||||
env_rel = _relative_to(directory, env_path)
|
||||
|
||||
_write_json(serve_params, build_serve_params(config))
|
||||
_write_json(bench_params, build_bench_params(workload))
|
||||
request_model, tokenizer = _benchmark_models(config)
|
||||
_write_run_script(
|
||||
run_script,
|
||||
config_rel=config_rel,
|
||||
env_rel=env_rel,
|
||||
request_model=request_model,
|
||||
tokenizer=tokenizer,
|
||||
workload=workload,
|
||||
)
|
||||
_write_recommend_script(
|
||||
recommend_script,
|
||||
config_rel=config_rel,
|
||||
env_rel=env_rel,
|
||||
workload=workload,
|
||||
)
|
||||
_write_guide(guide, workload)
|
||||
|
||||
return [serve_params, bench_params, run_script, recommend_script, guide]
|
||||
Executable
+329
@@ -0,0 +1,329 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
"""Select one measured scheduler recommendation from vLLM sweep results."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
from statistics import mean
|
||||
from typing import Any
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError as exc:
|
||||
raise SystemExit("PyYAML is required. Install it with: pip install pyyaml") from exc
|
||||
|
||||
|
||||
DEFAULT_CONFIG_PATH: str | None = None
|
||||
DEFAULT_ENV_PATH: str | None = None
|
||||
DEFAULT_TTFT_SLA_MS: float | None = None
|
||||
DEFAULT_TPOT_SLA_MS: float | None = None
|
||||
|
||||
|
||||
def _number(value: object) -> float | None:
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)):
|
||||
return None
|
||||
return float(value)
|
||||
|
||||
|
||||
def _mean(rows: list[dict[str, Any]], key: str) -> float | None:
|
||||
values = [_number(row.get(key)) for row in rows]
|
||||
numbers = [value for value in values if value is not None]
|
||||
return mean(numbers) if numbers else None
|
||||
|
||||
|
||||
def _positive_int(row: dict[str, Any], key: str) -> int | None:
|
||||
value = row.get(key)
|
||||
if isinstance(value, bool) or not isinstance(value, int) or value <= 0:
|
||||
return None
|
||||
return value
|
||||
|
||||
|
||||
def _resolve(script_dir: Path, value: str | None) -> Path | None:
|
||||
if value is None:
|
||||
return None
|
||||
path = Path(value)
|
||||
return path if path.is_absolute() else script_dir / path
|
||||
|
||||
|
||||
def _load_runs(results_dir: Path) -> list[dict[str, Any]]:
|
||||
rows: list[dict[str, Any]] = []
|
||||
for path in sorted(results_dir.rglob("summary.json")):
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
if not isinstance(data, list):
|
||||
continue
|
||||
for row in data:
|
||||
if isinstance(row, dict):
|
||||
copied = dict(row)
|
||||
copied["_summary_path"] = str(path)
|
||||
rows.append(copied)
|
||||
return rows
|
||||
|
||||
|
||||
def _aggregate_candidates(
|
||||
rows: list[dict[str, Any]],
|
||||
*,
|
||||
use_goodput: bool,
|
||||
) -> list[dict[str, Any]]:
|
||||
grouped: dict[tuple[int, int], list[dict[str, Any]]] = defaultdict(list)
|
||||
|
||||
for row in rows:
|
||||
seqs = _positive_int(row, "max_num_seqs")
|
||||
batch = _positive_int(row, "max_num_batched_tokens")
|
||||
if seqs is None or batch is None:
|
||||
continue
|
||||
grouped[(seqs, batch)].append(row)
|
||||
|
||||
candidates: list[dict[str, Any]] = []
|
||||
for (seqs, batch), runs in sorted(grouped.items()):
|
||||
failed_requests = sum(int(_number(run.get("failed")) or 0) for run in runs)
|
||||
output_throughput = _mean(runs, "output_throughput")
|
||||
request_goodput = _mean(runs, "request_goodput")
|
||||
p99_ttft_ms = _mean(runs, "p99_ttft_ms")
|
||||
p99_tpot_ms = _mean(runs, "p99_tpot_ms")
|
||||
|
||||
valid = failed_requests == 0 and output_throughput is not None
|
||||
reason = None
|
||||
if failed_requests:
|
||||
valid = False
|
||||
reason = f"{failed_requests} failed request(s) across repeated runs"
|
||||
elif use_goodput and request_goodput is None:
|
||||
valid = False
|
||||
reason = "request_goodput is missing; rerun sweep with SLO goodput enabled"
|
||||
|
||||
candidates.append(
|
||||
{
|
||||
"max_num_seqs": seqs,
|
||||
"max_num_batched_tokens": batch,
|
||||
"run_count": len(runs),
|
||||
"failed_requests": failed_requests,
|
||||
"mean_request_goodput": request_goodput,
|
||||
"mean_output_throughput": output_throughput,
|
||||
"mean_p99_ttft_ms": p99_ttft_ms,
|
||||
"mean_p99_tpot_ms": p99_tpot_ms,
|
||||
"valid": valid,
|
||||
"invalid_reason": reason,
|
||||
"summary_files": sorted({str(run["_summary_path"]) for run in runs}),
|
||||
}
|
||||
)
|
||||
|
||||
return candidates
|
||||
|
||||
|
||||
def _select_candidate(
|
||||
candidates: list[dict[str, Any]],
|
||||
*,
|
||||
use_goodput: bool,
|
||||
) -> tuple[dict[str, Any], str]:
|
||||
valid = [candidate for candidate in candidates if candidate["valid"]]
|
||||
if not valid:
|
||||
raise ValueError(
|
||||
"No valid sweep configuration was found. Inspect the sweep results "
|
||||
"for failed requests or missing metrics."
|
||||
)
|
||||
|
||||
if use_goodput:
|
||||
objective = "highest_mean_request_goodput"
|
||||
winner = max(
|
||||
valid,
|
||||
key=lambda candidate: (
|
||||
candidate["mean_request_goodput"],
|
||||
candidate["mean_output_throughput"],
|
||||
-candidate["max_num_batched_tokens"],
|
||||
-candidate["max_num_seqs"],
|
||||
),
|
||||
)
|
||||
if winner["mean_request_goodput"] <= 0:
|
||||
raise ValueError(
|
||||
"No sweep configuration produced non-zero request goodput for "
|
||||
"the supplied TTFT/TPOT objectives."
|
||||
)
|
||||
else:
|
||||
objective = "highest_mean_output_throughput"
|
||||
winner = max(
|
||||
valid,
|
||||
key=lambda candidate: (
|
||||
candidate["mean_output_throughput"],
|
||||
-candidate["max_num_batched_tokens"],
|
||||
-candidate["max_num_seqs"],
|
||||
),
|
||||
)
|
||||
|
||||
return winner, objective
|
||||
|
||||
|
||||
def _load_config(path: Path) -> dict[str, Any]:
|
||||
data = yaml.safe_load(path.read_text(encoding="utf-8"))
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError(f"{path} does not contain a YAML configuration object.")
|
||||
return data
|
||||
|
||||
|
||||
def _write_config(
|
||||
path: Path,
|
||||
*,
|
||||
source_path: Path,
|
||||
config: dict[str, Any],
|
||||
objective: str,
|
||||
) -> None:
|
||||
body = yaml.safe_dump(
|
||||
config,
|
||||
sort_keys=False,
|
||||
default_flow_style=False,
|
||||
allow_unicode=True,
|
||||
)
|
||||
header = (
|
||||
"# Generated from the initial recipe config after a vLLM parameter sweep.\n"
|
||||
f"# Initial config: {source_path}\n"
|
||||
f"# Selection objective: {objective}\n"
|
||||
)
|
||||
path.write_text(header + body, encoding="utf-8")
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description=(
|
||||
"Select one recommended max-num-seqs/max-num-batched-tokens pair "
|
||||
"from generated vLLM sweep results."
|
||||
)
|
||||
)
|
||||
parser.add_argument(
|
||||
"--results-dir",
|
||||
default="results/runtime-tuning",
|
||||
help="Sweep experiment directory (default: results/runtime-tuning).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--config",
|
||||
default=DEFAULT_CONFIG_PATH,
|
||||
help="Initial config.yml used by the sweep.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--env",
|
||||
default=DEFAULT_ENV_PATH,
|
||||
help="Environment file to show in the deployment instructions.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--ttft-sla-ms",
|
||||
type=float,
|
||||
default=DEFAULT_TTFT_SLA_MS,
|
||||
help="TTFT objective used to generate request goodput.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--tpot-sla-ms",
|
||||
type=float,
|
||||
default=DEFAULT_TPOT_SLA_MS,
|
||||
help="TPOT objective used to generate request goodput.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-config",
|
||||
default="recommended-config.yml",
|
||||
help="Recommended config output path.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-json",
|
||||
default="recommendation.json",
|
||||
help="Recommendation evidence output path.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
script_dir = Path(__file__).resolve().parent
|
||||
|
||||
config_path = _resolve(script_dir, args.config)
|
||||
env_path = _resolve(script_dir, args.env)
|
||||
results_dir = _resolve(script_dir, args.results_dir)
|
||||
output_config = _resolve(script_dir, args.output_config)
|
||||
output_json = _resolve(script_dir, args.output_json)
|
||||
|
||||
if config_path is None:
|
||||
raise ValueError("--config is required.")
|
||||
assert results_dir is not None
|
||||
assert output_config is not None
|
||||
assert output_json is not None
|
||||
|
||||
if not results_dir.exists():
|
||||
raise ValueError(
|
||||
f"Sweep results were not found at {results_dir}. Run the sweep first."
|
||||
)
|
||||
|
||||
rows = _load_runs(results_dir)
|
||||
if not rows:
|
||||
raise ValueError(f"No summary.json sweep results found under {results_dir}.")
|
||||
|
||||
use_goodput = args.ttft_sla_ms is not None or args.tpot_sla_ms is not None
|
||||
candidates = _aggregate_candidates(rows, use_goodput=use_goodput)
|
||||
winner, objective = _select_candidate(candidates, use_goodput=use_goodput)
|
||||
|
||||
initial_config = _load_config(config_path)
|
||||
recommended_config = dict(initial_config)
|
||||
recommended_config["max-num-seqs"] = winner["max_num_seqs"]
|
||||
recommended_config["max-num-batched-tokens"] = winner["max_num_batched_tokens"]
|
||||
|
||||
_write_config(
|
||||
output_config,
|
||||
source_path=config_path,
|
||||
config=recommended_config,
|
||||
objective=objective,
|
||||
)
|
||||
|
||||
recommendation = {
|
||||
"selection_objective": objective,
|
||||
"slo": {
|
||||
"ttft_ms": args.ttft_sla_ms,
|
||||
"tpot_ms": args.tpot_sla_ms,
|
||||
},
|
||||
"initial": {
|
||||
"max_num_seqs": initial_config.get("max-num-seqs"),
|
||||
"max_num_batched_tokens": initial_config.get("max-num-batched-tokens"),
|
||||
},
|
||||
"recommended": {
|
||||
"max_num_seqs": winner["max_num_seqs"],
|
||||
"max_num_batched_tokens": winner["max_num_batched_tokens"],
|
||||
},
|
||||
"measured": {
|
||||
"run_count": winner["run_count"],
|
||||
"mean_request_goodput": winner["mean_request_goodput"],
|
||||
"mean_output_throughput": winner["mean_output_throughput"],
|
||||
"mean_p99_ttft_ms": winner["mean_p99_ttft_ms"],
|
||||
"mean_p99_tpot_ms": winner["mean_p99_tpot_ms"],
|
||||
},
|
||||
"candidates": candidates,
|
||||
}
|
||||
output_json.write_text(
|
||||
json.dumps(recommendation, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
print("Recommended runtime configuration")
|
||||
print()
|
||||
print(f" max-num-seqs: {winner['max_num_seqs']}")
|
||||
print(f" max-num-batched-tokens: {winner['max_num_batched_tokens']}")
|
||||
print()
|
||||
print(f"Selection objective: {objective}")
|
||||
if use_goodput:
|
||||
print(f" mean request goodput: {winner['mean_request_goodput']:.2f} req/s")
|
||||
print(f" mean output throughput: {winner['mean_output_throughput']:.2f} tok/s")
|
||||
if winner["mean_p99_ttft_ms"] is not None:
|
||||
print(f" mean P99 TTFT: {winner['mean_p99_ttft_ms']:.2f} ms")
|
||||
if winner["mean_p99_tpot_ms"] is not None:
|
||||
print(f" mean P99 TPOT: {winner['mean_p99_tpot_ms']:.2f} ms")
|
||||
print()
|
||||
print(f"Wrote {output_config}")
|
||||
print(f"Wrote {output_json}")
|
||||
print()
|
||||
print("Deploy:")
|
||||
if env_path is not None:
|
||||
print(f" source {env_path}")
|
||||
print(f" vllm serve --config {output_config}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in New Issue
Block a user