|
|
|
@@ -3,30 +3,51 @@
|
|
|
|
|
|
|
|
|
|
## Introduction
|
|
|
|
|
|
|
|
|
|
This directory contains the performance benchmarking CI for vllm.
|
|
|
|
|
The goal is to help developers know the impact of their PRs on the performance of vllm.
|
|
|
|
|
This directory contains two sets of benchmark for vllm.
|
|
|
|
|
- Performance benchmark: benchmark vllm's performance under various workload, for **developers** to gain clarity on whether their PR improves/degrades vllm's performance
|
|
|
|
|
- Nightly benchmark: compare vllm's performance against alternatives (tgi, trt-llm and lmdeploy), for **the public** to know when to choose vllm.
|
|
|
|
|
|
|
|
|
|
This benchmark will be *triggered* upon:
|
|
|
|
|
- A PR being merged into vllm.
|
|
|
|
|
- Every commit for those PRs with `perf-benchmarks` label.
|
|
|
|
|
|
|
|
|
|
**Benchmarking Coverage**: latency, throughput and fix-qps serving on A100 (the support for more GPUs is comming later), with different models.
|
|
|
|
|
See [vLLM performance dashboard](https://perf.vllm.ai) for the latest performance benchmark results and [vLLM GitHub README](https://github.com/vllm-project/vllm/blob/main/README.md) for latest nightly benchmark results.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Performance benchmark quick overview
|
|
|
|
|
|
|
|
|
|
**Benchmarking Coverage**: latency, throughput and fix-qps serving on A100 (the support for FP8 benchmark on H100 is coming!), with different models.
|
|
|
|
|
|
|
|
|
|
**Benchmarking Duration**: about 1hr.
|
|
|
|
|
|
|
|
|
|
**For benchmarking developers**: please try your best to constraint the duration of benchmarking to less than 1.5 hr so that it won't take forever to run.
|
|
|
|
|
**For benchmarking developers**: please try your best to constraint the duration of benchmarking to about 1 hr so that it won't take forever to run.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Configuring the workload
|
|
|
|
|
## Nightly benchmark quick overview
|
|
|
|
|
|
|
|
|
|
The benchmarking workload contains three parts:
|
|
|
|
|
- Latency tests in `latency-tests.json`.
|
|
|
|
|
- Throughput tests in `throughput-tests.json`.
|
|
|
|
|
- Serving tests in `serving-tests.json`.
|
|
|
|
|
**Benchmarking Coverage**: Fix-qps serving on A100 (the support for FP8 benchmark on H100 is coming!) on Llama-3 8B, 70B and Mixtral 8x7B.
|
|
|
|
|
|
|
|
|
|
See [descriptions.md](tests/descriptions.md) for detailed descriptions.
|
|
|
|
|
**Benchmarking engines**: vllm, TGI, trt-llm and lmdeploy.
|
|
|
|
|
|
|
|
|
|
### Latency test
|
|
|
|
|
**Benchmarking Duration**: about 3.5hrs.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Trigger the benchmark
|
|
|
|
|
|
|
|
|
|
Performance benchmark will be triggered when:
|
|
|
|
|
- A PR being merged into vllm.
|
|
|
|
|
- Every commit for those PRs with `perf-benchmarks` label.
|
|
|
|
|
|
|
|
|
|
Nightly benchmark will be triggered when:
|
|
|
|
|
- Every commit for those PRs with `nightly-benchmarks` label.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Performance benchmark details
|
|
|
|
|
|
|
|
|
|
See [descriptions.md](tests/descriptions.md) for detailed descriptions, and use `tests/latency-tests.json`, `tests/throughput-tests.json`, `tests/serving-tests.json` to configure the test cases.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### Latency test
|
|
|
|
|
|
|
|
|
|
Here is an example of one test inside `latency-tests.json`:
|
|
|
|
|
|
|
|
|
@@ -54,12 +75,12 @@ Note that the performance numbers are highly sensitive to the value of the param
|
|
|
|
|
WARNING: The benchmarking script will save json results by itself, so please do not configure `--output-json` parameter in the json file.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Throughput test
|
|
|
|
|
#### Throughput test
|
|
|
|
|
The tests are specified in `throughput-tests.json`. The syntax is similar to `latency-tests.json`, except for that the parameters will be fed forward to `benchmark_throughput.py`.
|
|
|
|
|
|
|
|
|
|
The number of this test is also stable -- a slight change on the value of this number might vary the performance numbers by a lot.
|
|
|
|
|
|
|
|
|
|
### Serving test
|
|
|
|
|
#### Serving test
|
|
|
|
|
We test the throughput by using `benchmark_serving.py` with request rate = inf to cover the online serving overhead. The corresponding parameters are in `serving-tests.json`, and here is an example:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
@@ -96,9 +117,36 @@ The number of this test is less stable compared to the delay and latency benchma
|
|
|
|
|
|
|
|
|
|
WARNING: The benchmarking script will save json results by itself, so please do not configure `--save-results` or other results-saving-related parameters in `serving-tests.json`.
|
|
|
|
|
|
|
|
|
|
## Visualizing the results
|
|
|
|
|
#### Visualizing the results
|
|
|
|
|
The `convert-results-json-to-markdown.py` helps you put the benchmarking results inside a markdown table, by formatting [descriptions.md](tests/descriptions.md) with real benchmarking results.
|
|
|
|
|
You can find the result presented as a table inside the `buildkite/performance-benchmark` job page.
|
|
|
|
|
If you do not see the table, please wait till the benchmark finish running.
|
|
|
|
|
The json version of the table (together with the json version of the benchmark) will be also attached to the markdown file.
|
|
|
|
|
The raw benchmarking results (in the format of json files) are in the `Artifacts` tab of the benchmarking.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Nightly test details
|
|
|
|
|
|
|
|
|
|
See [nightly-descriptions.md](nightly-descriptions.md) for the detailed description on test workload, models and docker containers of benchmarking other llm engines.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### Workflow
|
|
|
|
|
|
|
|
|
|
- The [nightly-pipeline.yaml](nightly-pipeline.yaml) specifies the docker containers for different LLM serving engines.
|
|
|
|
|
- Inside each container, we run [run-nightly-suite.sh](run-nightly-suite.sh), which will probe the serving engine of the current container.
|
|
|
|
|
- The `run-nightly-suite.sh` will redirect the request to `tests/run-[llm serving engine name]-nightly.sh`, which parses the workload described in [nightly-tests.json](tests/nightly-tests.json) and performs the benchmark.
|
|
|
|
|
- At last, we run [scripts/plot-nightly-results.py](scripts/plot-nightly-results.py) to collect and plot the final benchmarking results, and update the results to buildkite.
|
|
|
|
|
|
|
|
|
|
#### Nightly tests
|
|
|
|
|
|
|
|
|
|
In [nightly-tests.json](tests/nightly-tests.json), we include the command line arguments for benchmarking commands, together with the benchmarking test cases. The format is highly similar to performance benchmark.
|
|
|
|
|
|
|
|
|
|
#### Docker containers
|
|
|
|
|
|
|
|
|
|
The docker containers for benchmarking are specified in `nightly-pipeline.yaml`.
|
|
|
|
|
|
|
|
|
|
WARNING: the docker versions are HARD-CODED and SHOULD BE ALIGNED WITH `nightly-descriptions.md`. The docker versions need to be hard-coded as there are several version-specific bug fixes inside `tests/run-[llm serving engine name]-nightly.sh`.
|
|
|
|
|
|
|
|
|
|
WARNING: populating `trt-llm` to latest version is not easy, as it requires updating several protobuf files in [tensorrt-demo](https://github.com/neuralmagic/tensorrt-demo.git).
|
|
|
|
|
|
|
|
|
|