diff --git a/tools/recipes/README.md b/tools/recipes/README.md index ac4cf3d298..e1ff3e5549 100644 --- a/tools/recipes/README.md +++ b/tools/recipes/README.md @@ -1,87 +1,46 @@ # vLLM Recipes Tools -Utilities for consuming deployment configurations from [vLLM Recipes](https://recipes.vllm.ai/) and converting them into files that can be used directly with vLLM. +Convert a vLLM Recipes deployment rendering into files for `vllm serve`. -## `recipe_json_to_vllm_config.py` +## Optimized Deployment Flow -Converts a hardware-specific vLLM Recipes JSON rendering into: +```mermaid +flowchart LR + R["vLLM Recipe"] --> C["Recipe Converter"] + H["Hardware Info (optional)"] --> C + W["Workload Info (optional)"] --> C + C --> F["config.yml + env.sh"] + C -.-> S["Sweep Tuning (optional)"] + S -.-> F + F --> D["vLLM Docker Image"] + D --> E["OpenAI Endpoint"] -- `config.yml` — native configuration for `vllm serve --config` -- `env.sh` — environment variables required by the recipe + style S stroke-dasharray: 5 5 +``` -The converter uses the Recipes JSON API as the source of truth. It does not reimplement model-variant, hardware, or strategy compatibility rules. +The recipe is the baseline. Hardware and workload information can optionally +refine the initial configuration. Sweep tuning is an optional validation step. -## Install Dependency +## 1. vLLM Recipes Only + +Use the converter directly when the recipe already contains the deployment +settings you need. This path requires only PyYAML; the vLLM Python package is +not required unless optional runtime tuning or sweep generation is requested. ```bash pip install pyyaml ``` -## Recipe Selection Model +Choose whichever recipe-selection method fits the workflow: -Discovery follows the JSON links published by vLLM Recipes: - -```text -/models.json - -> /{hf_id}.json - -> recommended_command.by_hardware[{hardware}] - -> /{hf_id}/hw/{hardware}.json - -> recommended strategy - -> alternatives[{strategy}] -``` - -The per-hardware JSON is the recommended deployment rendering for that model and hardware. Alternative strategy renderings are referenced through the JSON's `alternatives` map. - -The converter follows those exact links instead of constructing strategy paths locally. This also handles the Recipes API's legacy default-hardware strategy path automatically. - -## Interactive Recipe Discovery - -If you do not know the recipe JSON URL, run the script without an input: +**Interactive discovery** — search models, then choose hardware and strategy: ```bash python3 tools/recipes/recipe_json_to_vllm_config.py ``` -The script will: - -1. Search the Recipes model index, including promoted variant model IDs. -2. Ask which model to use. -3. Show the hardware configurations available for that model. -4. Show the recommended strategy and generated alternative strategies. -5. Resolve the exact Recipes JSON endpoint. -6. Generate `config.yml` and `env.sh`. - -Example interaction: - -```text -No recipe JSON supplied; starting Recipes API discovery. -Model search (for example: llama 3.1): llama 3.1 fp8 - -Matching models: - [1] nvidia/Llama-3.1-8B-Instruct-FP8 — Llama-3.1-8B-Instruct [Meta] -Select model: 1 - -Available hardware: - [1] arc_pro_b70 - [2] b200 - [3] h100 -Select hardware: 1 - -Available strategies: - [1] single_node_tp (recommended) - [2] multi_node_tp -Select strategy: 1 - -Resolved recipe: - Model: nvidia/Llama-3.1-8B-Instruct-FP8 - Hardware: arc_pro_b70 - Strategy: single_node_tp - JSON: https://recipes.vllm.ai/nvidia/Llama-3.1-8B-Instruct-FP8/hw/arc_pro_b70.json -``` - -## Non-Interactive Recipe Discovery - -For automation or CI, specify the model and hardware directly: +**Non-interactive discovery** — provide model and hardware and use the +Recipes-recommended strategy: ```bash python3 tools/recipes/recipe_json_to_vllm_config.py \ @@ -89,68 +48,81 @@ python3 tools/recipes/recipe_json_to_vllm_config.py \ --hardware xeon6 ``` -When `--strategy` is omitted with both `--model` and `--hardware`, the converter uses the Recipes-recommended strategy from the per-hardware JSON. - -To request a generated alternative strategy explicitly: - -```bash -python3 tools/recipes/recipe_json_to_vllm_config.py \ - --model nvidia/Llama-3.1-8B-Instruct-FP8 \ - --hardware arc_pro_b70 \ - --strategy multi_node_tp -``` - -The converter does not synthesize a URL such as `.../strategies/multi_node_tp.json`. It reads the selected hardware JSON and follows the exact `alternatives["multi_node_tp"]` link published by the Recipes API. - -## Direct Recipe JSON Input - -If you already know the recipe JSON URL: +**Direct JSON input** — use a Recipes JSON URL or a local JSON file: ```bash python3 tools/recipes/recipe_json_to_vllm_config.py \ https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct/hw/xeon6.json -``` -A local JSON file is also supported: - -```bash python3 tools/recipes/recipe_json_to_vllm_config.py recipe.json ``` -When a positional JSON source is supplied, do not combine it with `--model`, `--hardware`, or `--strategy`. +All paths generate `config.yml` and `env.sh`. See +[REFERENCE.md](REFERENCE.md) for recipe discovery, strategy selection, direct +JSON input, custom output files, and deployment scope. -## Start vLLM +## 2. Hardware Information (Optional) -After generating the files: - -```bash -source env.sh -vllm serve --config config.yml -``` - -Example generated `config.yml`: - -```yaml -model: meta-llama/Llama-3.1-8B-Instruct -tensor-parallel-size: 1 -``` - -If the selected recipe does not require environment variables, `env.sh` will contain no additional exports. - -## Custom Output Files +Add `--detect-hardware` when the target host's effective CPU/NUMA/memory +resources should refine deployment-sensitive values such as +`tensor-parallel-size` and `gpu-memory-utilization`. ```bash python3 tools/recipes/recipe_json_to_vllm_config.py \ --model meta-llama/Llama-3.1-8B-Instruct \ --hardware xeon6 \ - --config-out llama31-xeon6.yml \ - --env-out llama31-xeon6-env.sh + --detect-hardware ``` -## Strategy and Deployment Scope +Hardware detection is optional and uses vLLM CPU resource utilities only when +requested. See [RUNTIME_TUNING.md](RUNTIME_TUNING.md#hardware-information). -Strategy discovery and config conversion are separate concerns. +## 3. Workload Information (Optional) -The converter can resolve any generated strategy JSON exposed by the Recipes API. It currently emits one `config.yml`, so the selected rendering must contain a single `vllm serve` `argv`. +Workload hints can refine scheduler settings for one initial deployment +suggestion. Inputs include token lengths, concurrency, and optional latency or +capacity objectives. -Single-process renderings such as `single_node_tp` can be converted directly. Multi-node, disaggregated prefill/decode, or other multi-process renderings expose fields such as `head_argv`, `worker_argv`, `prefill`, or `decode`; the converter intentionally exits instead of generating an incomplete single-process config. +```bash +python3 tools/recipes/recipe_json_to_vllm_config.py \ + --model meta-llama/Llama-3.1-8B-Instruct \ + --hardware xeon6 \ + --input-tokens 2048 \ + --output-tokens 128 \ + --concurrency 32 \ + --ttft-sla-ms 3000 \ + --tpot-sla-ms 100 +``` + +Hardware detection and workload information are independent optional inputs; +they can also be supplied together. See +[RUNTIME_TUNING.md](RUNTIME_TUNING.md#workload-information) for the supported +inputs and how runtime parameters are calculated. + +## 4. Sweep Tuning (Optional) + +Use `--generate-sweep` when the initial scheduler suggestion should be validated +with `vllm bench sweep serve`. The sweep benchmarks nearby scheduler values and +`recommend.py` produces one measured `recommended-config.yml` plus the +selection evidence in `recommendation.json`. + +```bash +python3 tools/recipes/recipe_json_to_vllm_config.py \ + --model meta-llama/Llama-3.1-8B-Instruct \ + --hardware xeon6 \ + --detect-hardware \ + --input-tokens 2048 \ + --output-tokens 128 \ + --concurrency 32 \ + --generate-sweep +``` + +See [SWEEP_TUNING.md](SWEEP_TUNING.md) for the benchmark, recommendation, and +vLLM CPU Docker-shell workflow. + +## Start vLLM + +```bash +source env.sh +vllm serve --config config.yml +``` diff --git a/tools/recipes/REFERENCE.md b/tools/recipes/REFERENCE.md new file mode 100644 index 0000000000..747695eae8 --- /dev/null +++ b/tools/recipes/REFERENCE.md @@ -0,0 +1,177 @@ +# vLLM Recipes Tools + +Utilities for consuming deployment configurations from [vLLM Recipes](https://recipes.vllm.ai/) and converting them into files that can be used directly with vLLM. + +## Optimized Deployment Flow + +The recipe provides the validated deployment baseline. Hardware and workload +information are optional inputs that can refine deployment-sensitive values +before the generated configuration is passed to vLLM. + +```mermaid +flowchart LR + R["vLLM Recipe"] --> C["Recipe Converter"] + H["Hardware Info (optional)"] --> C + W["Workload Info (optional)"] --> C + C --> F["config.yml + env.sh"] + F --> D["vLLM Docker Image"] + D --> E["OpenAI Endpoint"] +``` + +## `recipe_json_to_vllm_config.py` + +Converts a hardware-specific vLLM Recipes JSON rendering into: + +- `config.yml` — native configuration for `vllm serve --config` +- `env.sh` — environment variables required by the recipe + +The converter uses the Recipes JSON API as the source of truth. It does not reimplement model-variant, hardware, or strategy compatibility rules. + +## Install Dependency + +```bash +pip install pyyaml +``` + +## Recipe Selection Model + +Discovery follows the JSON links published by vLLM Recipes: + +```text +/models.json + -> /{hf_id}.json + -> recommended_command.by_hardware[{hardware}] + -> /{hf_id}/hw/{hardware}.json + -> recommended strategy + -> alternatives[{strategy}] +``` + +The per-hardware JSON is the recommended deployment rendering for that model and hardware. Alternative strategy renderings are referenced through the JSON's `alternatives` map. + +The converter follows those exact links instead of constructing strategy paths locally. This also handles the Recipes API's legacy default-hardware strategy path automatically. + +## Interactive Recipe Discovery + +If you do not know the recipe JSON URL, run the script without an input: + +```bash +python3 tools/recipes/recipe_json_to_vllm_config.py +``` + +The script will: + +1. Search the Recipes model index, including promoted variant model IDs. +2. Ask which model to use. +3. Show the hardware configurations available for that model. +4. Show the recommended strategy and generated alternative strategies. +5. Resolve the exact Recipes JSON endpoint. +6. Generate `config.yml` and `env.sh`. + +Example interaction: + +```text +No recipe JSON supplied; starting Recipes API discovery. +Model search (for example: llama 3.1): llama 3.1 fp8 + +Matching models: + [1] nvidia/Llama-3.1-8B-Instruct-FP8 — Llama-3.1-8B-Instruct [Meta] +Select model: 1 + +Available hardware: + [1] arc_pro_b70 + [2] b200 + [3] h100 +Select hardware: 1 + +Available strategies: + [1] single_node_tp (recommended) + [2] multi_node_tp +Select strategy: 1 + +Resolved recipe: + Model: nvidia/Llama-3.1-8B-Instruct-FP8 + Hardware: arc_pro_b70 + Strategy: single_node_tp + JSON: https://recipes.vllm.ai/nvidia/Llama-3.1-8B-Instruct-FP8/hw/arc_pro_b70.json +``` + +## Non-Interactive Recipe Discovery + +For automation or CI, specify the model and hardware directly: + +```bash +python3 tools/recipes/recipe_json_to_vllm_config.py \ + --model meta-llama/Llama-3.1-8B-Instruct \ + --hardware xeon6 +``` + +When `--strategy` is omitted with both `--model` and `--hardware`, the converter uses the Recipes-recommended strategy from the per-hardware JSON. + +To request a generated alternative strategy explicitly: + +```bash +python3 tools/recipes/recipe_json_to_vllm_config.py \ + --model nvidia/Llama-3.1-8B-Instruct-FP8 \ + --hardware arc_pro_b70 \ + --strategy multi_node_tp +``` + +The converter does not synthesize a URL such as `.../strategies/multi_node_tp.json`. It reads the selected hardware JSON and follows the exact `alternatives["multi_node_tp"]` link published by the Recipes API. + +## Direct Recipe JSON Input + +If you already know the recipe JSON URL: + +```bash +python3 tools/recipes/recipe_json_to_vllm_config.py \ + https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct/hw/xeon6.json +``` + +A local JSON file is also supported: + +```bash +python3 tools/recipes/recipe_json_to_vllm_config.py recipe.json +``` + +When a positional JSON source is supplied, do not combine it with `--model`, `--hardware`, or `--strategy`. + +## Start vLLM + +After generating the files: + +```bash +source env.sh +vllm serve --config config.yml +``` + +Example generated `config.yml`: + +```yaml +model: meta-llama/Llama-3.1-8B-Instruct +tensor-parallel-size: 1 +``` + +If the selected recipe does not require environment variables, `env.sh` will contain no additional exports. + +## Custom Output Files + +```bash +python3 tools/recipes/recipe_json_to_vllm_config.py \ + --model meta-llama/Llama-3.1-8B-Instruct \ + --hardware xeon6 \ + --config-out llama31-xeon6.yml \ + --env-out llama31-xeon6-env.sh +``` + +## Strategy and Deployment Scope + +Strategy discovery and config conversion are separate concerns. + +The converter can resolve any generated strategy JSON exposed by the Recipes API. It currently emits one `config.yml`, so the selected rendering must contain a single `vllm serve` `argv`. + +Single-process renderings such as `single_node_tp` can be converted directly. Multi-node, disaggregated prefill/decode, or other multi-process renderings expose fields such as `head_argv`, `worker_argv`, `prefill`, or `decode`; the converter intentionally exits instead of generating an incomplete single-process config. + +## Related Guides + +- [RUNTIME_TUNING.md](RUNTIME_TUNING.md) — optional hardware and workload inputs, runtime parameter policies, and calculations. +- [SWEEP_TUNING.md](SWEEP_TUNING.md) — optional benchmark sweep and measured recommendation workflow. diff --git a/tools/recipes/RUNTIME_TUNING.md b/tools/recipes/RUNTIME_TUNING.md new file mode 100644 index 0000000000..43031b7d2a --- /dev/null +++ b/tools/recipes/RUNTIME_TUNING.md @@ -0,0 +1,292 @@ +# Runtime Tuning + +The Recipes JSON remains the baseline. vLLM Recipes already provide validated +model, hardware, strategy, environment variables, and serving arguments. +Runtime tuning is optional and is intended for parameters whose best value can +depend on the actual deployment resources or expected request workload. + +## Information Sources + +Runtime tuning combines up to three inputs: + +- **vLLM Recipe** — the required baseline from the Recipes JSON API or a direct + recipe JSON file. +- **Hardware information (optional)** — detected with `--detect-hardware`, + including effective CPU/NUMA topology and memory availability. +- **Workload information (optional)** — supplied through CLI hints such as input + and output token lengths, concurrency, TTFT/TPOT objectives, and target QPS. + +If no optional hardware or workload information is supplied, the converter keeps +the normal Recipes conversion behavior. + +See [Deployment-Time Parameters](#deployment-time-parameters) for the mapping +between these inputs and individual runtime parameters. + +## Hardware Information + +Hardware detection is enabled only with `--detect-hardware`. It reports the +effective resources available to the process or container, including CPU/NUMA +topology and memory information. The current policy can use those values to +refine `tensor-parallel-size` and `gpu-memory-utilization`. + +```bash +python3 tools/recipes/recipe_json_to_vllm_config.py \ + --model meta-llama/Llama-3.1-8B-Instruct \ + --hardware xeon6 \ + --detect-hardware +``` + +Hardware detection is optional. Recipe hardware selects the tuning policy; host +hardware detection provides resource information to that policy. + +## Workload Information + +Workload information is supplied explicitly because the converter runs before a +vLLM server exists. Supported hints include: + +- `--input-tokens` +- `--output-tokens` +- `--concurrency` +- `--ttft-sla-ms` +- `--tpot-sla-ms` +- `--target-qps` + +Example: + +```bash +python3 tools/recipes/recipe_json_to_vllm_config.py \ + --model meta-llama/Llama-3.1-8B-Instruct \ + --hardware xeon6 \ + --input-tokens 2048 \ + --output-tokens 128 \ + --concurrency 32 \ + --ttft-sla-ms 3000 \ + --tpot-sla-ms 100 +``` + +`--output-tokens` is used to estimate steady-state request turnover for +`max-num-batched-tokens`. When both `--target-qps` and `--tpot-sla-ms` are +available, they are also used to estimate prompt arrival pressure per scheduler +step. `--ttft-sla-ms` is collected but is not yet converted directly into a +batch-size formula because the relationship between TTFT and scheduler budget is +model- and hardware-dependent. + +## Deployment-Time Parameters + +The parameters below may already exist in a recipe. They are candidates for +deployment-time refinement when the recipe value is missing, generic, or based +on a validation environment that differs from the user's target deployment. + +| Runtime parameter | Why it may need deployment-time refinement | Main decision input | Current policy | +| --- | --- | --- | --- | +| `tensor-parallel-size` | The effective CPU/NUMA topology available to a container or pod can differ from the system used to validate the recipe. | Hardware topology | Use the largest power-of-two TP value that does not exceed the effective NUMA-node count. | +| `gpu-memory-utilization` | Available memory can differ by machine size, container limits, and other memory use. The vLLM option name is also used by the CPU backend. | Hardware memory + recipe baseline | Calculate a conservative fraction from the most constrained NUMA node. | +| `max-num-seqs` | The useful scheduler concurrency depends on the number of requests expected to be active at the same time. | Workload concurrency | Set `max-num-seqs` to `--concurrency` when supplied. | +| `max-num-batched-tokens` | Each scheduler iteration must share its token budget between active decodes and incoming prefills. | Input/output token shape, concurrency, and optional QPS/TPOT | Calculate decode budget + expected prefill demand, with vLLM scheduler constraints as floors. | +| `data-parallel-size` | The required replica count depends on the requested throughput and the capacity of one replica. | Capacity target | Keep the recipe value today because per-replica SLA capacity is not known. | + +## How Each Runtime Parameter Is Calculated + +### `tensor-parallel-size` + +TP is derived from the effective NUMA topology reported by hardware detection: + +```text +TP = largest power of two <= effective NUMA-node count +``` + +For example, 2 effective NUMA nodes produce TP=2, while 6 nodes currently +produce TP=4. This is a topology-based starting point and avoids automatically +selecting unusual non-power-of-two TP sizes before they are validated. + +### `gpu-memory-utilization` + +For every effective NUMA memory node, the policy calculates: + +```text +node_safe_fraction = available_memory / total_memory - 0.10 +safe_fraction = min(0.90, minimum node_safe_fraction) +``` + +The 10% reserve leaves memory for model/runtime overhead outside the vLLM cache +budget. If the recipe already contains a smaller value, the policy preserves the +smaller value. If it is absent, `0.80` is used as the initial ceiling: + +```text +candidate = min(recipe value or 0.80, safe_fraction) +``` + +### `max-num-seqs` + +When `--concurrency` is supplied, it directly represents the requested maximum +number of simultaneously active requests: + +```text +max-num-seqs = concurrency +``` + +If concurrency is not supplied, the converter does not override this parameter. + +### `max-num-batched-tokens` + +The token budget is workload-derived and no longer assumes a fixed number of +parallel prefills. + +First, the policy determines the active sequence count: + +```text +active_sequences = + --concurrency + or recipe max-num-seqs + or vLLM default max_num_seqs (128) +``` + +Decode requests consume approximately one token per active sequence in a +scheduler iteration: + +```text +decode_budget = active_sequences +``` + +The policy then estimates how many new prompts need prefill work per scheduler +step: + +```text +prefills_per_step = max( + 1, + active_sequences / output_tokens, + target_qps * tpot_sla_ms / 1000 +) + +prefills_per_step = min(active_sequences, prefills_per_step) +prefill_budget = ceil(input_tokens * prefills_per_step) +``` + +The final scheduler budget is: + +```text +max-num-batched-tokens = max( + vLLM default max_num_batched_tokens (2048), + active_sequences, + decode_budget + prefill_budget +) +``` + +For example, with 2048 input tokens, 128 output tokens, and concurrency 32: + +```text +decode_budget = 32 +prefills_per_step = max(1, 32 / 128) = 1 +prefill_budget = 2048 +candidate = max(2048, 32, 32 + 2048) = 2080 +``` + +If chunked prefill is explicitly disabled and `max-model-len` is available, the +policy also ensures: + +```text +max-num-batched-tokens >= max-model-len +``` + +### `data-parallel-size` + +`--target-qps` alone is not enough to choose DP safely. A correct capacity rule +also needs measured per-replica throughput that still satisfies TTFT/TPOT: + +```text +DP = ceil(target_qps / qps_per_replica_at_SLO) +``` + +Because the converter does not have that measured capacity yet, it intentionally +keeps the recipe DP value rather than guessing. + +## Precedence + +```text +vLLM defaults + -> vLLM Recipes baseline + -> hardware refinement (optional) + -> workload / SLO refinement (optional) + -> config.yml + env.sh +``` + +## Runtime-Tuning Hardware Scope + +Runtime tuning is selected from the resolved recipe JSON's `hardware` field, +rather than by inspecting which physical devices happen to be present on the +host. This matters because a GPU server also exposes its host CPU topology. + +The current runtime-tuning policy registry contains `xeon6`. A tuning request +for unregistered recipe hardware, such as `b200`, fails before host hardware +detection: + +```text +ERROR: Runtime tuning is not supported for recipe hardware 'b200'. +Currently supported: xeon6. +``` + +Plain recipe conversion remains available for all recipe hardware. The gate only +applies when optional runtime-tuning inputs are requested. + +## vLLM CPU Docker Shell + +Run hardware detection and runtime tuning inside the target vLLM CPU container +so detected CPU, NUMA, memory, and cgroup limits match the deployment +environment. + +From the vLLM source tree: + +```bash +mkdir -p recipe-output + +docker run --rm -it \ + --entrypoint bash \ + --security-opt seccomp=unconfined \ + --cap-add SYS_NICE \ + --shm-size=4g \ + -p 8000:8000 \ + -v "$PWD/tools/recipes:/recipes:ro" \ + -v "$PWD/recipe-output:/output" \ + -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \ + -w /output \ + vllm/vllm-openai-cpu:latest-x86_64 +``` + +`/output` is writable; `/recipes` remains read-only. + +Inside the container, generate one runtime-tuned initial configuration: + +```bash +python3 /recipes/recipe_json_to_vllm_config.py \ + --model meta-llama/Llama-3.1-8B-Instruct \ + --hardware xeon6 \ + --detect-hardware \ + --input-tokens 2048 \ + --output-tokens 128 \ + --concurrency 32 \ + --ttft-sla-ms 3000 \ + --tpot-sla-ms 100 \ + --config-out /output/config.yml \ + --env-out /output/env.sh +``` + +Deploy the generated initial configuration: + +```bash +source /output/env.sh +vllm serve --config /output/config.yml +``` + +For optional benchmark-backed validation of the scheduler settings, continue +with [SWEEP_TUNING.md](SWEEP_TUNING.md). + +## Implementation + +- `hardware_detection.py` collects effective CPU/NUMA/memory information. +- `runtime_tuning.py` owns hardware-policy selection and independent parameter + tuning functions. +- `recipe_json_to_vllm_config.py` resolves the recipe, collects optional inputs, + applies the selected policy, and generates `config.yml` and `env.sh`. + +For benchmark-backed validation of the initial scheduler suggestion, see +[SWEEP_TUNING.md](SWEEP_TUNING.md). diff --git a/tools/recipes/SWEEP_TUNING.md b/tools/recipes/SWEEP_TUNING.md new file mode 100644 index 0000000000..7842c8aff5 --- /dev/null +++ b/tools/recipes/SWEEP_TUNING.md @@ -0,0 +1,124 @@ +# Sweep Tuning + +Sweep tuning is optional. The converter always creates one initial `config.yml` +first. Users can deploy it immediately or benchmark nearby scheduler settings +and generate one measured recommendation. + +## Generate + +```bash +python3 tools/recipes/recipe_json_to_vllm_config.py \ + --model meta-llama/Llama-3.1-8B-Instruct \ + --hardware xeon6 \ + --detect-hardware \ + --input-tokens 2048 \ + --output-tokens 128 \ + --concurrency 32 \ + --ttft-sla-ms 3000 \ + --tpot-sla-ms 100 \ + --generate-sweep +``` + +The optional package is: + +```text +sweep/ +├── serve_params.json +├── bench_params.json +├── run_sweep.sh +├── recommend.py +└── SWEEP.md +``` + +The sweep currently varies only `max-num-seqs` and +`max-num-batched-tokens`. + +## Run and Recommend + +```bash +sweep/run_sweep.sh --dry-run +sweep/run_sweep.sh +sweep/recommend.py +``` + +Resume an interrupted sweep with: + +```bash +sweep/run_sweep.sh --resume +``` + +`recommend.py` writes: + +```text +sweep/recommended-config.yml +sweep/recommendation.json +``` + +With TTFT/TPOT objectives, the benchmark uses vLLM `--goodput`, and the +recommender selects highest mean request goodput across repeated runs. Without +latency objectives it selects highest mean output-token throughput. Failed +benchmark configurations are excluded. + +`recommended-config.yml` changes only the two swept scheduler parameters. +`recommendation.json` records the measured evidence. + +Deploy: + +```bash +source env.sh +vllm serve --config sweep/recommended-config.yml +``` + +## vLLM CPU Docker Shell + +Use the common CPU container setup in +[RUNTIME_TUNING.md](RUNTIME_TUNING.md#vllm-cpu-docker-shell). It ensures +hardware detection runs against the same effective CPU, NUMA, memory, and +cgroup limits used by the deployment. + +Inside that container, generate the initial suggestion plus sweep package: + +```bash +python3 /recipes/recipe_json_to_vllm_config.py \ + --model meta-llama/Llama-3.1-8B-Instruct \ + --hardware xeon6 \ + --detect-hardware \ + --input-tokens 2048 \ + --output-tokens 128 \ + --concurrency 32 \ + --ttft-sla-ms 3000 \ + --tpot-sla-ms 100 \ + --config-out /output/config.yml \ + --env-out /output/env.sh \ + --generate-sweep \ + --sweep-out-dir /output/sweep +``` + +The initial configuration can be deployed directly: + +```bash +source /output/env.sh +vllm serve --config /output/config.yml +``` + +Stop that manually started server before running the sweep, because +`run_sweep.sh` starts and stops its own vLLM servers. + +```bash +/output/sweep/run_sweep.sh --dry-run +/output/sweep/run_sweep.sh +/output/sweep/recommend.py +``` + +Inspect: + +```bash +cat /output/sweep/recommendation.json +``` + +Deploy the measured result: + +```bash +source /output/env.sh +vllm serve --config /output/sweep/recommended-config.yml +``` diff --git a/tools/recipes/hardware_detection.py b/tools/recipes/hardware_detection.py new file mode 100644 index 0000000000..2b90dffac4 --- /dev/null +++ b/tools/recipes/hardware_detection.py @@ -0,0 +1,115 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""CPU hardware discovery for optional vLLM Recipes runtime tuning. + +Keep this module focused on facts exposed to the current process/container. +Tuning decisions belong in runtime_tuning.py. +""" + +from __future__ import annotations + +import json +import platform +from dataclasses import asdict, dataclass +from pathlib import Path + +from vllm.utils.cpu_resource_utils import ( + get_allowed_cpu_list, + get_memory_affinity, + get_memory_node_info, +) + + +@dataclass(frozen=True) +class NumaNodeInfo: + node_id: int + logical_cpu_ids: tuple[int, ...] + physical_core_count: int + total_memory_bytes: int + available_memory_bytes: int + + +@dataclass(frozen=True) +class HardwareInfo: + architecture: str + socket_count: int | None + numa_nodes: tuple[NumaNodeInfo, ...] + allowed_logical_cpu_count: int + physical_core_count: int + + @property + def numa_node_count(self) -> int: + return len(self.numa_nodes) + + def to_dict(self) -> dict[str, object]: + return asdict(self) + + +def _read_socket_id(cpu_id: int) -> int | None: + path = Path(f"/sys/devices/system/cpu/cpu{cpu_id}/topology/physical_package_id") + try: + return int(path.read_text().strip()) + except (OSError, ValueError): + return None + + +def detect_hardware() -> HardwareInfo: + """Return CPU resources effectively available to this process. + + vLLM's CPU helpers already honor process CPU affinity and cgroup-aware + memory limits. We intentionally use get_memory_affinity() rather than + get_visible_memory_node() so CPU_VISIBLE_MEMORY_NODES does not become an + input to the detector; auto binding remains owned by the CPU backend. + """ + cpus = get_allowed_cpu_list() + if not cpus: + raise RuntimeError("vLLM reported no CPUs available to this process.") + + cpu_numa_nodes = {cpu.numa_node for cpu in cpus if cpu.numa_node >= 0} + memory_numa_nodes = set(get_memory_affinity()) + effective_numa_nodes = sorted(cpu_numa_nodes & memory_numa_nodes) + if not effective_numa_nodes: + effective_numa_nodes = sorted(cpu_numa_nodes) + + physical_cores = { + (cpu.numa_node, cpu.physical_core) for cpu in cpus if cpu.physical_core >= 0 + } + + numa_nodes = [] + for node_id in effective_numa_nodes: + node_cpus = [cpu for cpu in cpus if cpu.numa_node == node_id] + node_physical_cores = { + cpu.physical_core for cpu in node_cpus if cpu.physical_core >= 0 + } + memory = get_memory_node_info(node_id) + numa_nodes.append( + NumaNodeInfo( + node_id=node_id, + logical_cpu_ids=tuple(sorted(cpu.id for cpu in node_cpus)), + physical_core_count=len(node_physical_cores), + total_memory_bytes=memory.total_memory, + available_memory_bytes=memory.available_memory, + ) + ) + + socket_ids = { + socket_id for cpu in cpus if (socket_id := _read_socket_id(cpu.id)) is not None + } + + return HardwareInfo( + architecture=platform.machine(), + socket_count=len(socket_ids) if socket_ids else None, + numa_nodes=tuple(numa_nodes), + allowed_logical_cpu_count=len(cpus), + physical_core_count=len(physical_cores), + ) + + +def main() -> int: + print(json.dumps(detect_hardware().to_dict(), indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/recipes/recipe_json_to_vllm_config.py b/tools/recipes/recipe_json_to_vllm_config.py index 52c109dcba..099b5a03c4 100644 --- a/tools/recipes/recipe_json_to_vllm_config.py +++ b/tools/recipes/recipe_json_to_vllm_config.py @@ -57,7 +57,6 @@ try: except ImportError as exc: raise SystemExit("PyYAML is required. Install it with: pip install pyyaml") from exc - DEFAULT_API_BASE = "https://recipes.vllm.ai" SHORT_ALIASES = { @@ -117,6 +116,71 @@ def parse_args() -> argparse.Namespace: default="env.sh", help="Output shell environment file (default: env.sh)", ) + + tuning = p.add_argument_group( + "optional runtime tuning", + ( + "Refine the recipe baseline only when additional information is supplied. " + "Runtime tuning is enabled only for recipe hardware with a registered " + "policy (currently: xeon6)." + ), + ) + tuning.add_argument( + "--detect-hardware", + action="store_true", + help=( + "Detect effective CPU/NUMA/memory resources and allow the selected " + "recipe-hardware policy to override recipe runtime arguments." + ), + ) + tuning.add_argument( + "--input-tokens", + type=int, + help="Expected input-token length. Optional workload hint.", + ) + tuning.add_argument( + "--output-tokens", + type=int, + help="Expected output-token length. Optional workload hint.", + ) + tuning.add_argument( + "--concurrency", + type=int, + help="Expected maximum concurrent requests. Optional workload hint.", + ) + tuning.add_argument( + "--ttft-sla-ms", + type=float, + help="Optional time-to-first-token objective in milliseconds.", + ) + tuning.add_argument( + "--tpot-sla-ms", + type=float, + help="Optional time-per-output-token objective in milliseconds.", + ) + tuning.add_argument( + "--target-qps", + type=float, + help="Optional capacity target for future DP/capacity tuning.", + ) + + sweep = p.add_argument_group( + "optional performance sweep", + ("Generate benchmark files after creating one initial runtime suggestion."), + ) + sweep.add_argument( + "--generate-sweep", + action="store_true", + help=( + "Generate an optional vllm bench sweep package around the single " + "initial runtime suggestion." + ), + ) + sweep.add_argument( + "--sweep-out-dir", + default="sweep", + help="Output directory for optional sweep files (default: sweep).", + ) return p.parse_args() @@ -642,18 +706,115 @@ def main() -> int: argv = recipe_argv(recipe) config = argv_to_config(argv) + + tuning_requested = ( + args.detect_hardware + or args.generate_sweep + or any( + value is not None + for value in ( + args.input_tokens, + args.output_tokens, + args.concurrency, + args.ttft_sla_ms, + args.tpot_sla_ms, + args.target_qps, + ) + ) + ) + + tuning = None + workload = None + sweep_writer = None + if tuning_requested: + # Keep plain Recipes conversion lightweight. vLLM-specific modules + # are imported only for optional runtime tuning or sweep generation. + from runtime_tuning import ( + WorkloadHints, + finetune_runtime_config, + get_runtime_tuning_policies, + ) + + workload = WorkloadHints( + input_tokens=args.input_tokens, + output_tokens=args.output_tokens, + concurrency=args.concurrency, + ttft_sla_ms=args.ttft_sla_ms, + tpot_sla_ms=args.tpot_sla_ms, + target_qps=args.target_qps, + ) + + if args.generate_sweep: + from sweep_generation import ( + validate_sweep_workload, + write_sweep_files, + ) + + validate_sweep_workload(workload) + sweep_writer = write_sweep_files + + recipe_hardware = recipe.get("hardware") + policies = get_runtime_tuning_policies(recipe_hardware) + + hardware = None + if args.detect_hardware: + from hardware_detection import detect_hardware + + hardware = detect_hardware() + + tuning = finetune_runtime_config( + config, + hardware=hardware, + workload=workload, + policies=policies, + ) + config.update(tuning.overrides) + write_config(args.config_out, source, recipe, config) write_env(args.env_out, source, recipe) + + sweep_files: list[Path] = [] + if args.generate_sweep: + assert workload is not None + assert sweep_writer is not None + sweep_files = sweep_writer( + args.sweep_out_dir, + config_path=args.config_out, + env_path=args.env_out, + config=config, + workload=workload, + ) + + if tuning is not None: + if tuning.overrides: + print("Initial runtime suggestion:") + for key, value in tuning.overrides.items(): + print(f" {key}: {value}") + for note in tuning.notes: + print(f" tuning: {note}") except Exception as exc: print(f"ERROR: {exc}", file=sys.stderr) return 1 print(f"Wrote {args.config_out}") print(f"Wrote {args.env_out}") + if sweep_files: + print(f"Wrote optional sweep package under {args.sweep_out_dir}/") print() print("Run:") print(f" source {shlex.quote(args.env_out)}") print(f" vllm serve --config {shlex.quote(args.config_out)}") + if sweep_files: + sweep_dir = Path(args.sweep_out_dir) + run_sweep = sweep_dir / "run_sweep.sh" + recommend = sweep_dir / "recommend.py" + print() + print("Optional performance sweep:") + print(f" {shlex.quote(str(run_sweep))} --dry-run") + print(f" {shlex.quote(str(run_sweep))}") + print() + print("After the sweep:") + print(f" {shlex.quote(str(recommend))}") return 0 diff --git a/tools/recipes/runtime_tuning.py b/tools/recipes/runtime_tuning.py new file mode 100644 index 0000000000..e251de89d7 --- /dev/null +++ b/tools/recipes/runtime_tuning.py @@ -0,0 +1,314 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Optional runtime refinement for configs produced from vLLM Recipes. + +The recipe remains the baseline. Each policy may return an override only when +enough information is available. Keep policies independent so the tuning +algorithms can evolve without changing the converter or hardware detector. + +The policies in this draft are intentionally simple heuristics, not benchmark- +proven universal optima. +""" + +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass, field +from math import ceil +from typing import Any + +from hardware_detection import HardwareInfo + +from vllm.config.scheduler import SchedulerConfig + + +@dataclass(frozen=True) +class WorkloadHints: + input_tokens: int | None = None + output_tokens: int | None = None + concurrency: int | None = None + ttft_sla_ms: float | None = None + tpot_sla_ms: float | None = None + target_qps: float | None = None + + def validate(self) -> None: + for name in ("input_tokens", "output_tokens", "concurrency"): + value = getattr(self, name) + if value is not None and value <= 0: + raise ValueError(f"{name} must be greater than zero") + + for name in ("ttft_sla_ms", "tpot_sla_ms", "target_qps"): + value = getattr(self, name) + if value is not None and value <= 0: + raise ValueError(f"{name} must be greater than zero") + + +@dataclass +class TuningResult: + overrides: dict[str, Any] = field(default_factory=dict) + notes: list[str] = field(default_factory=list) + + +Policy = Callable[ + [dict[str, Any], HardwareInfo | None, WorkloadHints, TuningResult], + None, +] + + +def _record_override( + result: TuningResult, + key: str, + value: Any, + reason: str, +) -> None: + result.overrides[key] = value + result.notes.append(f"{key}={value!r}: {reason}") + + +def _get_active_sequence_count( + config: dict[str, Any], + workload: WorkloadHints, +) -> int: + """Return the sequence count used to size one scheduler iteration.""" + if workload.concurrency is not None: + return workload.concurrency + + configured = config.get("max-num-seqs") + if ( + isinstance(configured, int) + and not isinstance(configured, bool) + and configured > 0 + ): + return configured + + return SchedulerConfig.DEFAULT_MAX_NUM_SEQS + + +def _estimate_prefills_per_step( + workload: WorkloadHints, + active_sequences: int, +) -> float: + """Estimate steady-state prompt arrivals per decode scheduler step.""" + # Reserve enough prefill budget for at least one representative prompt. + prefills_per_step = 1.0 + + # In steady state, active_sequences / output_tokens approximates how many + # requests finish and are replaced by new prefills per decode step. + if workload.output_tokens is not None: + prefills_per_step = max( + prefills_per_step, + active_sequences / workload.output_tokens, + ) + + # With target QPS + TPOT, approximate how many prompts arrive during one + # decode scheduler-step interval. + if workload.target_qps is not None and workload.tpot_sla_ms is not None: + prefills_per_step = max( + prefills_per_step, + workload.target_qps * workload.tpot_sla_ms / 1000.0, + ) + + # A step cannot replace more requests than are active. + return min(float(active_sequences), prefills_per_step) + + +def _resolve_tensor_parallel_size( + config: dict[str, Any], + hardware: HardwareInfo | None, + workload: WorkloadHints, + result: TuningResult, +) -> None: + del config, workload + if hardware is None or hardware.numa_node_count < 1: + return + + # Draft policy: use the largest power-of-two TP that does not exceed the + # effective NUMA-node count. This deliberately avoids hard-coding unusual + # non-power-of-two topologies into the converter. + candidate = 1 << (hardware.numa_node_count.bit_length() - 1) + _record_override( + result, + "tensor-parallel-size", + candidate, + "derived from the effective NUMA-node count", + ) + + +def _resolve_gpu_memory_utilization( + config: dict[str, Any], + hardware: HardwareInfo | None, + workload: WorkloadHints, + result: TuningResult, +) -> None: + del workload + if hardware is None or not hardware.numa_nodes: + return + + reserve_fraction = 0.10 + safe_fractions = [ + max( + 0.0, + node.available_memory_bytes / node.total_memory_bytes - reserve_fraction, + ) + for node in hardware.numa_nodes + if node.total_memory_bytes > 0 + ] + if not safe_fractions: + return + + safe_fraction = round(min(0.90, min(safe_fractions)), 2) + if safe_fraction <= 0: + return + + recipe_value = config.get("gpu-memory-utilization") + if isinstance(recipe_value, (int, float)): + candidate = min(float(recipe_value), safe_fraction) + else: + candidate = min(0.80, safe_fraction) + + _record_override( + result, + "gpu-memory-utilization", + round(candidate, 2), + "constrained by effective per-NUMA available memory", + ) + + +def _resolve_max_num_seqs( + config: dict[str, Any], + hardware: HardwareInfo | None, + workload: WorkloadHints, + result: TuningResult, +) -> None: + del config, hardware + if workload.concurrency is None: + return + + _record_override( + result, + "max-num-seqs", + workload.concurrency, + "derived from the optional target concurrency", + ) + + +def _resolve_max_num_batched_tokens( + config: dict[str, Any], + hardware: HardwareInfo | None, + workload: WorkloadHints, + result: TuningResult, +) -> None: + del hardware + if workload.input_tokens is None: + return + + active_sequences = _get_active_sequence_count(config, workload) + + # Decode requests consume roughly one token per active sequence in one + # scheduler iteration. Prefills share the remaining scheduler token budget. + decode_budget = active_sequences + prefills_per_step = _estimate_prefills_per_step(workload, active_sequences) + prefill_budget = ceil(workload.input_tokens * prefills_per_step) + + # Use vLLM's scheduler defaults/constraints as floors rather than local + # recipe-specific magic constants. + candidate = max( + SchedulerConfig.DEFAULT_MAX_NUM_BATCHED_TOKENS, + active_sequences, + decode_budget + prefill_budget, + ) + + # Chunked prefill normally lets a prompt span scheduler iterations. If the + # recipe explicitly disables it, vLLM requires the token budget to cover + # max_model_len when max_model_len is known. + if config.get("enable-chunked-prefill") is False: + max_model_len = config.get("max-model-len") + if ( + isinstance(max_model_len, int) + and not isinstance(max_model_len, bool) + and max_model_len > 0 + ): + candidate = max(candidate, max_model_len) + + _record_override( + result, + "max-num-batched-tokens", + candidate, + ( + "workload-derived scheduler budget " + f"(active_sequences={active_sequences}, " + f"prefills_per_step={prefills_per_step:.2f}, " + f"decode_budget={decode_budget}, " + f"prefill_budget={prefill_budget})" + ), + ) + + +def _resolve_data_parallel_size( + config: dict[str, Any], + hardware: HardwareInfo | None, + workload: WorkloadHints, + result: TuningResult, +) -> None: + del config, hardware + if workload.target_qps is None: + return + + result.notes.append( + "data-parallel-size: target QPS supplied; kept recipe value because " + "per-replica capacity is not known yet" + ) + + +DEFAULT_POLICIES: tuple[Policy, ...] = ( + _resolve_tensor_parallel_size, + _resolve_gpu_memory_utilization, + _resolve_max_num_seqs, + _resolve_max_num_batched_tokens, + _resolve_data_parallel_size, +) + +# Select tuning behavior from the hardware requested by the Recipes rendering, +# not from devices physically present in the host. A GPU server still exposes +# its host CPU topology, so host inspection alone cannot determine deployment +# intent. +HARDWARE_TUNING_POLICIES: dict[str, tuple[Policy, ...]] = { + "xeon6": DEFAULT_POLICIES, +} + + +def get_runtime_tuning_policies( + recipe_hardware: object, +) -> tuple[Policy, ...]: + if not isinstance(recipe_hardware, str) or not recipe_hardware: + raise ValueError( + "Runtime tuning requires the resolved Recipes JSON to declare " + "a non-empty `hardware` field." + ) + + policies = HARDWARE_TUNING_POLICIES.get(recipe_hardware) + if policies is None: + supported = ", ".join(sorted(HARDWARE_TUNING_POLICIES)) + raise ValueError( + "Runtime tuning is not supported for recipe hardware " + f"{recipe_hardware!r}. Currently supported: {supported}." + ) + return policies + + +def finetune_runtime_config( + config: dict[str, Any], + *, + hardware: HardwareInfo | None = None, + workload: WorkloadHints | None = None, + policies: tuple[Policy, ...] = DEFAULT_POLICIES, +) -> TuningResult: + """Return overrides; never mutate the recipe-derived config in place.""" + hints = workload or WorkloadHints() + hints.validate() + + result = TuningResult() + for policy in policies: + policy(config, hardware, hints, result) + return result diff --git a/tools/recipes/sweep_generation.py b/tools/recipes/sweep_generation.py new file mode 100644 index 0000000000..5758711e5e --- /dev/null +++ b/tools/recipes/sweep_generation.py @@ -0,0 +1,341 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Generate optional vLLM benchmark sweep files from one initial suggestion.""" + +from __future__ import annotations + +import json +import os +import shlex +from pathlib import Path +from typing import Any + +from runtime_tuning import WorkloadHints + + +def _positive_int(config: dict[str, Any], key: str) -> int: + value = config.get(key) + if not isinstance(value, int) or isinstance(value, bool) or value <= 0: + raise ValueError(f"Sweep generation requires a positive initial {key!r} value.") + return value + + +def _strict_lower_power_of_two(value: int) -> int: + if value <= 1: + return 1 + return 1 << ((value - 1).bit_length() - 1) + + +def _strict_upper_power_of_two(value: int) -> int: + return 1 << value.bit_length() + + +def validate_sweep_workload(workload: WorkloadHints) -> None: + required = { + "--input-tokens": workload.input_tokens, + "--output-tokens": workload.output_tokens, + "--concurrency": workload.concurrency, + } + missing = [name for name, value in required.items() if value is None] + if missing: + raise ValueError("--generate-sweep requires " + ", ".join(missing) + ".") + + +def build_serve_params(config: dict[str, Any]) -> list[dict[str, Any]]: + """Build a small directed sweep around the single initial suggestion.""" + initial_seqs = _positive_int(config, "max-num-seqs") + initial_batch = _positive_int(config, "max-num-batched-tokens") + + minimum_batch = initial_seqs + if config.get("enable-chunked-prefill") is False: + max_model_len = config.get("max-model-len") + if ( + isinstance(max_model_len, int) + and not isinstance(max_model_len, bool) + and max_model_len > 0 + ): + minimum_batch = max(minimum_batch, max_model_len) + + lower_seqs = max(1, (initial_seqs + 1) // 2) + lower_batch = max( + minimum_batch, + _strict_lower_power_of_two(initial_batch), + ) + higher_batch = max( + minimum_batch, + _strict_upper_power_of_two(initial_batch), + ) + + candidates: list[dict[str, Any]] = [] + seen: set[tuple[int, int]] = set() + + def add(name: str, max_num_seqs: int, max_num_batched_tokens: int) -> None: + max_num_batched_tokens = max( + max_num_batched_tokens, + max_num_seqs, + ) + signature = (max_num_seqs, max_num_batched_tokens) + if signature in seen: + return + seen.add(signature) + candidates.append( + { + "_benchmark_name": name, + "max_num_seqs": max_num_seqs, + "max_num_batched_tokens": max_num_batched_tokens, + } + ) + + # Keep the exact initial suggestion as the measured baseline. Additional + # values exist only in the optional sweep package. + add("initial", initial_seqs, initial_batch) + add("lower_scheduler_concurrency", lower_seqs, initial_batch) + add("lower_batch_budget", initial_seqs, lower_batch) + add("latency_focused", lower_seqs, lower_batch) + add("higher_batch_budget", initial_seqs, higher_batch) + + return candidates + + +def build_bench_params(workload: WorkloadHints) -> list[dict[str, Any]]: + validate_sweep_workload(workload) + assert workload.input_tokens is not None + assert workload.output_tokens is not None + assert workload.concurrency is not None + + return [ + { + "_benchmark_name": "user_workload", + "random_input_len": workload.input_tokens, + "random_output_len": workload.output_tokens, + "max_concurrency": workload.concurrency, + } + ] + + +def _benchmark_models(config: dict[str, Any]) -> tuple[str, str]: + model = config.get("model") + if not isinstance(model, str) or not model: + raise ValueError("Sweep generation requires a model in config.yml.") + + served_model_name = config.get("served-model-name") + if isinstance(served_model_name, str) and served_model_name: + return served_model_name, model + if ( + isinstance(served_model_name, list) + and served_model_name + and isinstance(served_model_name[0], str) + ): + return served_model_name[0], model + + return model, model + + +def _relative_to(directory: Path, target: str) -> str: + return os.path.relpath(Path(target).resolve(), directory.resolve()) + + +def _write_json(path: Path, value: object) -> None: + path.write_text( + json.dumps(value, indent=2) + "\n", + encoding="utf-8", + ) + + +def _write_run_script( + path: Path, + *, + config_rel: str, + env_rel: str, + request_model: str, + tokenizer: str, + workload: WorkloadHints, +) -> None: + bench_parts = [ + "vllm bench serve", + "--backend vllm", + f"--model {shlex.quote(request_model)}", + f"--tokenizer {shlex.quote(tokenizer)}", + "--dataset-name random", + "--request-rate inf", + "--ignore-eos", + "--metric-percentiles 99", + ] + + goodput_pairs: list[str] = [] + if workload.ttft_sla_ms is not None: + goodput_pairs.append(f"ttft:{workload.ttft_sla_ms:g}") + if workload.tpot_sla_ms is not None: + goodput_pairs.append(f"tpot:{workload.tpot_sla_ms:g}") + if goodput_pairs: + bench_parts.append( + "--goodput " + " ".join(shlex.quote(pair) for pair in goodput_pairs) + ) + + bench_cmd = " ".join(bench_parts) + script = f"""#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd -- "$(dirname -- "${{BASH_SOURCE[0]}}")" && pwd)" +CONFIG_PATH="${{SCRIPT_DIR}}/{config_rel}" +ENV_PATH="${{SCRIPT_DIR}}/{env_rel}" + +source "${{ENV_PATH}}" + +vllm bench sweep serve \ + --serve-cmd "vllm serve --config '${{CONFIG_PATH}}'" \ + --bench-cmd "{bench_cmd}" \ + --serve-params "${{SCRIPT_DIR}}/serve_params.json" \ + --bench-params "${{SCRIPT_DIR}}/bench_params.json" \ + --output-dir "${{SCRIPT_DIR}}/results" \ + --experiment-name runtime-tuning \ + "$@" +""" + path.write_text(script, encoding="utf-8") + path.chmod(path.stat().st_mode | 0o111) + + +def _write_recommend_script( + path: Path, + *, + config_rel: str, + env_rel: str, + workload: WorkloadHints, +) -> None: + template_path = Path(__file__).with_name("sweep_recommendation.py") + source = template_path.read_text(encoding="utf-8") + + replacements = { + "DEFAULT_CONFIG_PATH: str | None = None": ( + f"DEFAULT_CONFIG_PATH: str | None = {config_rel!r}" + ), + "DEFAULT_ENV_PATH: str | None = None": ( + f"DEFAULT_ENV_PATH: str | None = {env_rel!r}" + ), + "DEFAULT_TTFT_SLA_MS: float | None = None": ( + f"DEFAULT_TTFT_SLA_MS: float | None = {workload.ttft_sla_ms!r}" + ), + "DEFAULT_TPOT_SLA_MS: float | None = None": ( + f"DEFAULT_TPOT_SLA_MS: float | None = {workload.tpot_sla_ms!r}" + ), + } + for marker, replacement in replacements.items(): + if marker not in source: + raise ValueError(f"Recommender template marker not found: {marker}") + source = source.replace(marker, replacement, 1) + + path.write_text(source, encoding="utf-8") + path.chmod(path.stat().st_mode | 0o111) + + +def _write_guide(path: Path, workload: WorkloadHints) -> None: + sla_lines: list[str] = [] + if workload.ttft_sla_ms is not None: + sla_lines.append(f"- TTFT objective: `{workload.ttft_sla_ms:g} ms`") + if workload.tpot_sla_ms is not None: + sla_lines.append(f"- TPOT objective: `{workload.tpot_sla_ms:g} ms`") + + sla_text = "\n".join(sla_lines) + if sla_text: + sla_text = ( + "\n## Supplied latency objectives\n\n" + + sla_text + + "\n\nThe generated benchmark uses these values with vLLM " + "`--goodput`.\n" + ) + + content = f"""# Optional Runtime Tuning Sweep + +The generated `config.yml` is the **single initial suggestion** and can be +deployed directly. The sweep benchmarks nearby values for: + +- `max-num-seqs` +- `max-num-batched-tokens` + +## Run + +```bash +./run_sweep.sh --dry-run +./run_sweep.sh +./recommend.py +``` + +For a quick one-run experiment: + +```bash +rm -rf results/runtime-tuning +./run_sweep.sh --num-runs 1 +./recommend.py +``` + +Resume an interrupted sweep: + +```bash +./run_sweep.sh --resume +``` + +By default, vLLM benchmarks each parameter combination three times. +{sla_text} +## Outputs + +`recommend.py` writes: + +```text +recommended-config.yml +recommendation.json +``` + +With TTFT/TPOT objectives it selects highest mean request goodput across +repeated runs. Without latency objectives it selects highest mean output-token +throughput. Configurations with failed requests are excluded. + +`recommended-config.yml` copies the initial configuration and changes only +`max-num-seqs` and `max-num-batched-tokens`. +""" + path.write_text(content, encoding="utf-8") + + +def write_sweep_files( + output_dir: str, + *, + config_path: str, + env_path: str, + config: dict[str, Any], + workload: WorkloadHints, +) -> list[Path]: + """Write an optional sweep package around the initial config suggestion.""" + validate_sweep_workload(workload) + + directory = Path(output_dir) + directory.mkdir(parents=True, exist_ok=True) + + serve_params = directory / "serve_params.json" + bench_params = directory / "bench_params.json" + run_script = directory / "run_sweep.sh" + recommend_script = directory / "recommend.py" + guide = directory / "SWEEP.md" + + config_rel = _relative_to(directory, config_path) + env_rel = _relative_to(directory, env_path) + + _write_json(serve_params, build_serve_params(config)) + _write_json(bench_params, build_bench_params(workload)) + request_model, tokenizer = _benchmark_models(config) + _write_run_script( + run_script, + config_rel=config_rel, + env_rel=env_rel, + request_model=request_model, + tokenizer=tokenizer, + workload=workload, + ) + _write_recommend_script( + recommend_script, + config_rel=config_rel, + env_rel=env_rel, + workload=workload, + ) + _write_guide(guide, workload) + + return [serve_params, bench_params, run_script, recommend_script, guide] diff --git a/tools/recipes/sweep_recommendation.py b/tools/recipes/sweep_recommendation.py new file mode 100755 index 0000000000..a49b02849a --- /dev/null +++ b/tools/recipes/sweep_recommendation.py @@ -0,0 +1,329 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +"""Select one measured scheduler recommendation from vLLM sweep results.""" + +from __future__ import annotations + +import argparse +import json +from collections import defaultdict +from pathlib import Path +from statistics import mean +from typing import Any + +try: + import yaml +except ImportError as exc: + raise SystemExit("PyYAML is required. Install it with: pip install pyyaml") from exc + + +DEFAULT_CONFIG_PATH: str | None = None +DEFAULT_ENV_PATH: str | None = None +DEFAULT_TTFT_SLA_MS: float | None = None +DEFAULT_TPOT_SLA_MS: float | None = None + + +def _number(value: object) -> float | None: + if isinstance(value, bool) or not isinstance(value, (int, float)): + return None + return float(value) + + +def _mean(rows: list[dict[str, Any]], key: str) -> float | None: + values = [_number(row.get(key)) for row in rows] + numbers = [value for value in values if value is not None] + return mean(numbers) if numbers else None + + +def _positive_int(row: dict[str, Any], key: str) -> int | None: + value = row.get(key) + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + return None + return value + + +def _resolve(script_dir: Path, value: str | None) -> Path | None: + if value is None: + return None + path = Path(value) + return path if path.is_absolute() else script_dir / path + + +def _load_runs(results_dir: Path) -> list[dict[str, Any]]: + rows: list[dict[str, Any]] = [] + for path in sorted(results_dir.rglob("summary.json")): + data = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(data, list): + continue + for row in data: + if isinstance(row, dict): + copied = dict(row) + copied["_summary_path"] = str(path) + rows.append(copied) + return rows + + +def _aggregate_candidates( + rows: list[dict[str, Any]], + *, + use_goodput: bool, +) -> list[dict[str, Any]]: + grouped: dict[tuple[int, int], list[dict[str, Any]]] = defaultdict(list) + + for row in rows: + seqs = _positive_int(row, "max_num_seqs") + batch = _positive_int(row, "max_num_batched_tokens") + if seqs is None or batch is None: + continue + grouped[(seqs, batch)].append(row) + + candidates: list[dict[str, Any]] = [] + for (seqs, batch), runs in sorted(grouped.items()): + failed_requests = sum(int(_number(run.get("failed")) or 0) for run in runs) + output_throughput = _mean(runs, "output_throughput") + request_goodput = _mean(runs, "request_goodput") + p99_ttft_ms = _mean(runs, "p99_ttft_ms") + p99_tpot_ms = _mean(runs, "p99_tpot_ms") + + valid = failed_requests == 0 and output_throughput is not None + reason = None + if failed_requests: + valid = False + reason = f"{failed_requests} failed request(s) across repeated runs" + elif use_goodput and request_goodput is None: + valid = False + reason = "request_goodput is missing; rerun sweep with SLO goodput enabled" + + candidates.append( + { + "max_num_seqs": seqs, + "max_num_batched_tokens": batch, + "run_count": len(runs), + "failed_requests": failed_requests, + "mean_request_goodput": request_goodput, + "mean_output_throughput": output_throughput, + "mean_p99_ttft_ms": p99_ttft_ms, + "mean_p99_tpot_ms": p99_tpot_ms, + "valid": valid, + "invalid_reason": reason, + "summary_files": sorted({str(run["_summary_path"]) for run in runs}), + } + ) + + return candidates + + +def _select_candidate( + candidates: list[dict[str, Any]], + *, + use_goodput: bool, +) -> tuple[dict[str, Any], str]: + valid = [candidate for candidate in candidates if candidate["valid"]] + if not valid: + raise ValueError( + "No valid sweep configuration was found. Inspect the sweep results " + "for failed requests or missing metrics." + ) + + if use_goodput: + objective = "highest_mean_request_goodput" + winner = max( + valid, + key=lambda candidate: ( + candidate["mean_request_goodput"], + candidate["mean_output_throughput"], + -candidate["max_num_batched_tokens"], + -candidate["max_num_seqs"], + ), + ) + if winner["mean_request_goodput"] <= 0: + raise ValueError( + "No sweep configuration produced non-zero request goodput for " + "the supplied TTFT/TPOT objectives." + ) + else: + objective = "highest_mean_output_throughput" + winner = max( + valid, + key=lambda candidate: ( + candidate["mean_output_throughput"], + -candidate["max_num_batched_tokens"], + -candidate["max_num_seqs"], + ), + ) + + return winner, objective + + +def _load_config(path: Path) -> dict[str, Any]: + data = yaml.safe_load(path.read_text(encoding="utf-8")) + if not isinstance(data, dict): + raise ValueError(f"{path} does not contain a YAML configuration object.") + return data + + +def _write_config( + path: Path, + *, + source_path: Path, + config: dict[str, Any], + objective: str, +) -> None: + body = yaml.safe_dump( + config, + sort_keys=False, + default_flow_style=False, + allow_unicode=True, + ) + header = ( + "# Generated from the initial recipe config after a vLLM parameter sweep.\n" + f"# Initial config: {source_path}\n" + f"# Selection objective: {objective}\n" + ) + path.write_text(header + body, encoding="utf-8") + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description=( + "Select one recommended max-num-seqs/max-num-batched-tokens pair " + "from generated vLLM sweep results." + ) + ) + parser.add_argument( + "--results-dir", + default="results/runtime-tuning", + help="Sweep experiment directory (default: results/runtime-tuning).", + ) + parser.add_argument( + "--config", + default=DEFAULT_CONFIG_PATH, + help="Initial config.yml used by the sweep.", + ) + parser.add_argument( + "--env", + default=DEFAULT_ENV_PATH, + help="Environment file to show in the deployment instructions.", + ) + parser.add_argument( + "--ttft-sla-ms", + type=float, + default=DEFAULT_TTFT_SLA_MS, + help="TTFT objective used to generate request goodput.", + ) + parser.add_argument( + "--tpot-sla-ms", + type=float, + default=DEFAULT_TPOT_SLA_MS, + help="TPOT objective used to generate request goodput.", + ) + parser.add_argument( + "--output-config", + default="recommended-config.yml", + help="Recommended config output path.", + ) + parser.add_argument( + "--output-json", + default="recommendation.json", + help="Recommendation evidence output path.", + ) + return parser.parse_args() + + +def main() -> int: + args = parse_args() + script_dir = Path(__file__).resolve().parent + + config_path = _resolve(script_dir, args.config) + env_path = _resolve(script_dir, args.env) + results_dir = _resolve(script_dir, args.results_dir) + output_config = _resolve(script_dir, args.output_config) + output_json = _resolve(script_dir, args.output_json) + + if config_path is None: + raise ValueError("--config is required.") + assert results_dir is not None + assert output_config is not None + assert output_json is not None + + if not results_dir.exists(): + raise ValueError( + f"Sweep results were not found at {results_dir}. Run the sweep first." + ) + + rows = _load_runs(results_dir) + if not rows: + raise ValueError(f"No summary.json sweep results found under {results_dir}.") + + use_goodput = args.ttft_sla_ms is not None or args.tpot_sla_ms is not None + candidates = _aggregate_candidates(rows, use_goodput=use_goodput) + winner, objective = _select_candidate(candidates, use_goodput=use_goodput) + + initial_config = _load_config(config_path) + recommended_config = dict(initial_config) + recommended_config["max-num-seqs"] = winner["max_num_seqs"] + recommended_config["max-num-batched-tokens"] = winner["max_num_batched_tokens"] + + _write_config( + output_config, + source_path=config_path, + config=recommended_config, + objective=objective, + ) + + recommendation = { + "selection_objective": objective, + "slo": { + "ttft_ms": args.ttft_sla_ms, + "tpot_ms": args.tpot_sla_ms, + }, + "initial": { + "max_num_seqs": initial_config.get("max-num-seqs"), + "max_num_batched_tokens": initial_config.get("max-num-batched-tokens"), + }, + "recommended": { + "max_num_seqs": winner["max_num_seqs"], + "max_num_batched_tokens": winner["max_num_batched_tokens"], + }, + "measured": { + "run_count": winner["run_count"], + "mean_request_goodput": winner["mean_request_goodput"], + "mean_output_throughput": winner["mean_output_throughput"], + "mean_p99_ttft_ms": winner["mean_p99_ttft_ms"], + "mean_p99_tpot_ms": winner["mean_p99_tpot_ms"], + }, + "candidates": candidates, + } + output_json.write_text( + json.dumps(recommendation, indent=2) + "\n", + encoding="utf-8", + ) + + print("Recommended runtime configuration") + print() + print(f" max-num-seqs: {winner['max_num_seqs']}") + print(f" max-num-batched-tokens: {winner['max_num_batched_tokens']}") + print() + print(f"Selection objective: {objective}") + if use_goodput: + print(f" mean request goodput: {winner['mean_request_goodput']:.2f} req/s") + print(f" mean output throughput: {winner['mean_output_throughput']:.2f} tok/s") + if winner["mean_p99_ttft_ms"] is not None: + print(f" mean P99 TTFT: {winner['mean_p99_ttft_ms']:.2f} ms") + if winner["mean_p99_tpot_ms"] is not None: + print(f" mean P99 TPOT: {winner['mean_p99_tpot_ms']:.2f} ms") + print() + print(f"Wrote {output_config}") + print(f"Wrote {output_json}") + print() + print("Deploy:") + if env_path is not None: + print(f" source {env_path}") + print(f" vllm serve --config {output_config}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())