[Frontend][Rust] Add mm-processor benchmark for Rust frontend (#51922)
Signed-off-by: Karthik Gangula <gangula-karthik@users.noreply.github.com> Signed-off-by: gangula-karthik <gkarthik923@gmail.com> Signed-off-by: Isotr0py <Isotr0py@outlook.com> Co-authored-by: Karthik Gangula <gangula-karthik@users.noreply.github.com> Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com> Co-authored-by: Isotr0py <Isotr0py@outlook.com>
This commit is contained in:
co-authored by
Karthik Gangula
factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Isotr0py
parent
0961bbae28
commit
f84325c48c
@@ -23,7 +23,7 @@ cargo test -p vllm-bench -- --ignored
|
||||
|
||||
## Architecture
|
||||
|
||||
- `src/main.rs` — Entry point, mimalloc, tokio runtime, mode dispatch (compare/sweep/multi-run/multi-turn/single)
|
||||
- `src/main.rs` — Entry point, mimalloc, tokio runtime, mode dispatch (compare/sweep/multi-run/multi-turn/single, `mm-processor` subcommand)
|
||||
- `src/cli.rs` — clap derive CLI args (~50+ flags)
|
||||
- `src/config.rs` — Validated config from CLI; `GoodputConfig`, `RampUpConfig`, sampling param merging
|
||||
- `src/error.rs` — `BenchError` enum (Http, Json, Tokenizer, Config, EndpointTimeout, Backend, Io)
|
||||
@@ -32,6 +32,7 @@ cargo test -p vllm-bench -- --ignored
|
||||
- `src/sweep.rs` — Concurrency/rate parameter sweep (`--sweep-max-concurrency`, `--sweep-request-rate`)
|
||||
- `src/multi_run.rs` — N-run aggregation with mean/std/min/max/CV (`--num-runs`)
|
||||
- `src/compare.rs` — Side-by-side diff of two result JSON files (`--compare`)
|
||||
- `src/mm_processor.rs` — `vllm-bench mm-processor` subcommand: offline multimodal preprocessing latency benchmark against a managed headless engine (mirrors `vllm bench mm-processor`); per-stage timings come from `vllm_tracing::timing::RequestTimingLayer` (via `vllm_chat::mm_timing_layer`)
|
||||
- `src/tokenizer.rs` — `TokenizerKind` enum: Local(HuggingFace), Tiktoken, OR Server-side `/tokenize`+`/detokenize` fallback
|
||||
- `src/tiktoken.rs` — Tiktoken BPE loader (`.tiktoken`/`.model` files; built-in encodings o200k_base/cl100k_base; pat_str extraction from Python source)
|
||||
- `src/hub.rs` — `HubRepo`: thin wrapper over hf-hub's async (reqwest/native-tls) API — file download and repo listing, optional revision; the sync ureq backend is unusable here because it pulls rustls, which `rust/deny.toml` bans
|
||||
|
||||
@@ -36,6 +36,11 @@ tracing.workspace = true
|
||||
url.workspace = true
|
||||
uuid.workspace = true
|
||||
vllm-build-info.workspace = true
|
||||
vllm-chat.workspace = true
|
||||
vllm-engine-core-client.workspace = true
|
||||
vllm-llm.workspace = true
|
||||
vllm-managed-engine.workspace = true
|
||||
vllm-text.workspace = true
|
||||
vllm-tracing.workspace = true
|
||||
|
||||
[lints]
|
||||
|
||||
@@ -18,6 +18,7 @@ vllm-bench --backend vllm --base-url http://127.0.0.1:8000 \
|
||||
- **Beyond a single run** — concurrency/rate **sweeps**, **multi-run** stats, **multi-turn** conversations, **LoRA** multi-adapter, and result **comparison**.
|
||||
- **Steady-state metrics** — throughput/latency measured over the saturated plateau, excluding ramp-up and drain.
|
||||
- **Parity** — JSON output schema and timing semantics match Python `vllm bench serve` exactly.
|
||||
- **Multimodal preprocessing** — `mm-processor` subcommand measures per-stage VLM preprocessing latency against a managed headless engine (mirrors `vllm bench mm-processor`).
|
||||
|
||||
### Performance vs. Python
|
||||
|
||||
@@ -404,6 +405,32 @@ vllm-bench \
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b>Multimodal preprocessing (offline)</b></summary>
|
||||
|
||||
The `mm-processor` subcommand benchmarks the multimodal preprocessing pipeline
|
||||
itself (media fetch → per-modality preprocess → prompt expansion), mirroring
|
||||
`vllm bench mm-processor`. It spawns a managed headless Python engine (real
|
||||
weights, real encoder) — no serving endpoint needed — and reports per-stage
|
||||
timing (mean/median/std/percentiles) plus end-to-end latency.
|
||||
|
||||
```bash
|
||||
vllm-bench mm-processor \
|
||||
--model Qwen/Qwen2.5-VL-7B-Instruct \
|
||||
--num-prompts 100 --num-warmups 5 \
|
||||
--random-input-len 512 --random-output-len 128 \
|
||||
--random-mm-base-items-per-request 1 \
|
||||
--random-mm-limit-mm-per-prompt '{"image": 1, "video": 0}' \
|
||||
--random-mm-bucket-config '{(1024, 800, 1): 1.0}' \
|
||||
--metric-percentiles "50,90,99" \
|
||||
--output-json mm_processor_stats.json
|
||||
```
|
||||
|
||||
`--max-concurrency` defaults to `1`, matching Python's serial driver; raise it
|
||||
to measure a concurrent serving-style workload.
|
||||
|
||||
</details>
|
||||
|
||||
## Supported Backends
|
||||
|
||||
### Generation
|
||||
@@ -705,6 +732,36 @@ With `--multi-turn`, `--num-prompts` controls the number of **conversations**, n
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary><b><code>mm-processor</code> subcommand</b></summary>
|
||||
|
||||
Offline multimodal preprocessing latency benchmark (mirrors `vllm bench mm-processor`). Spawns a managed headless Python engine and drives the full chat preprocessing pipeline.
|
||||
|
||||
| Flag | Default | Description |
|
||||
| ------ | --------- | ------------- |
|
||||
| `--model` | (required) | Model to serve and benchmark (HF id or local path) |
|
||||
| `--num-prompts` | `10` | Prompts to process, excluding warmups |
|
||||
| `--num-warmups` | `1` | Warmup prompts processed and discarded before timing |
|
||||
| `--metric-percentiles` | `99` | Comma-separated percentiles to report |
|
||||
| `--output-json` | — | Path to write aggregate stats as JSON (same schema as Python) |
|
||||
| `--random-input-len` | `1024` | Text input length per prompt |
|
||||
| `--random-output-len` | `128` | Expected output length per prompt |
|
||||
| `--random-prefix-len` | `0` | Prefix token length per prompt |
|
||||
| `--random-range-ratio` | `0.0` | Input/output length range ratio (float or `{"input": i, "output": o}`) |
|
||||
| `--random-mm-base-items-per-request` | `1` | Base number of multimodal items per request |
|
||||
| `--random-mm-num-mm-items-range-ratio` | `0.0` | Range ratio (in [0, 1]) for the number of mm items per request |
|
||||
| `--random-mm-limit-mm-per-prompt` | `{"image": 255, "video": 1}` | Per-modality item caps |
|
||||
| `--random-mm-bucket-config` | `{(256,256,1): 0.5, (720,1280,1): 0.5}` | Bucket config |
|
||||
| `--seed` | `0` | Seed for dataset generation |
|
||||
| `--request-id-prefix` | `mm-proc-` | Request-id prefix for per-request timing keys |
|
||||
| `--max-concurrency` | `1` | Maximum concurrent requests (1 matches Python's serial driver) |
|
||||
| `--trust-remote-code` | `false` | Trust remote code for the tokenizer and the managed engine |
|
||||
| `--chat-template` | — | Chat-template override (inline text or path) |
|
||||
|
||||
Managed headless-engine options (e.g. `--python`, `--data-parallel-address`, `--data-parallel-rpc-port`, `--data-parallel-size`, `--max-model-len`, `--python-args`) are shared with other managed-engine commands; see `vllm-bench mm-processor --help`.
|
||||
|
||||
</details>
|
||||
|
||||
## Tokenizer Support
|
||||
|
||||
Tokenizers are loaded with a three-tier fallback chain:
|
||||
@@ -734,10 +791,11 @@ Use `--append-result` to append multiple runs to the same file in JSONL format.
|
||||
|
||||
```text
|
||||
src/
|
||||
├── main.rs # Entry point, mimalloc, tokio runtime, mode dispatch
|
||||
├── main.rs # Entry point, mimalloc, tokio runtime, mode/subcommand dispatch
|
||||
├── cli.rs # clap CLI argument definitions
|
||||
├── config.rs # Validated config, goodput/ramp-up parsing
|
||||
├── benchmark.rs # Core orchestrator (schedule, spawn, collect, verify, profile)
|
||||
├── mm_processor.rs # Offline mm preprocessing benchmark (mm-processor subcommand)
|
||||
├── multi_turn.rs # Multi-turn conversation orchestrator (channel workers)
|
||||
├── compare.rs # Result diff (--compare file_a.json file_b.json)
|
||||
├── sweep.rs # Parameter sweep (--sweep-max-concurrency, --sweep-request-rate)
|
||||
|
||||
@@ -10,6 +10,7 @@ mod datasets;
|
||||
mod error;
|
||||
mod hub;
|
||||
mod metrics;
|
||||
mod mm_processor;
|
||||
mod multi_run;
|
||||
mod multi_turn;
|
||||
mod output;
|
||||
@@ -25,6 +26,7 @@ pub use cli::{
|
||||
BackendKind, BenchServeArgs, DatasetName, LoraAssignment, RampUpStrategy, SpeedBenchConfig,
|
||||
};
|
||||
use config::BenchConfig;
|
||||
pub use mm_processor::{MmProcessorArgs, run_mm_processor};
|
||||
|
||||
/// Prepare process-wide resources for a benchmark run.
|
||||
pub fn prepare_process() {
|
||||
|
||||
@@ -6,22 +6,30 @@
|
||||
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
|
||||
|
||||
use anyhow::Context;
|
||||
use clap::Parser;
|
||||
use clap::{Parser, Subcommand};
|
||||
|
||||
#[derive(Parser)]
|
||||
#[command(
|
||||
name = "vllm-bench",
|
||||
about = "Benchmark online serving throughput",
|
||||
about = "Benchmark online serving throughput and offline multimodal preprocessing",
|
||||
version = vllm_build_info::VERSION
|
||||
)]
|
||||
struct Cli {
|
||||
#[command(flatten)]
|
||||
args: vllm_bench::BenchServeArgs,
|
||||
|
||||
/// Optional subcommand; absent by default for the online serving benchmark.
|
||||
#[command(subcommand)]
|
||||
command: Option<Command>,
|
||||
}
|
||||
|
||||
#[derive(Subcommand)]
|
||||
enum Command {
|
||||
/// Offline multimodal preprocessing latency benchmark.
|
||||
MmProcessor(vllm_bench::MmProcessorArgs),
|
||||
}
|
||||
|
||||
fn main() -> anyhow::Result<()> {
|
||||
vllm_tracing::init_tracing("Bench");
|
||||
|
||||
let cli = Cli::parse();
|
||||
vllm_bench::prepare_process();
|
||||
|
||||
@@ -30,5 +38,15 @@ fn main() -> anyhow::Result<()> {
|
||||
.build()
|
||||
.context("Failed to build tokio runtime")?;
|
||||
|
||||
runtime.block_on(vllm_bench::run(cli.args))
|
||||
match cli.command {
|
||||
Some(Command::MmProcessor(args)) => {
|
||||
let (timing_layer, timing_stats) = vllm_chat::mm_timing_layer();
|
||||
vllm_tracing::init_tracing_with("Bench", timing_layer);
|
||||
runtime.block_on(vllm_bench::run_mm_processor(args, timing_stats))
|
||||
}
|
||||
None => {
|
||||
vllm_tracing::init_tracing("Bench");
|
||||
runtime.block_on(vllm_bench::run(cli.args))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -216,20 +216,20 @@ pub fn calculate_metrics(
|
||||
(metrics, actual_output_lens)
|
||||
}
|
||||
|
||||
fn mean(data: &[f64]) -> f64 {
|
||||
pub(crate) fn mean(data: &[f64]) -> f64 {
|
||||
if data.is_empty() {
|
||||
return 0.0;
|
||||
}
|
||||
data.iter().sum::<f64>() / data.len() as f64
|
||||
}
|
||||
|
||||
fn sort_clone(data: &[f64]) -> Vec<f64> {
|
||||
pub(crate) fn sort_clone(data: &[f64]) -> Vec<f64> {
|
||||
let mut sorted = data.to_vec();
|
||||
sorted.sort_by(|a, b| a.partial_cmp(b).unwrap_or(std::cmp::Ordering::Equal));
|
||||
sorted
|
||||
}
|
||||
|
||||
fn median_sorted(sorted: &[f64]) -> f64 {
|
||||
pub(crate) fn median_sorted(sorted: &[f64]) -> f64 {
|
||||
if sorted.is_empty() {
|
||||
return 0.0;
|
||||
}
|
||||
@@ -241,7 +241,7 @@ fn median_sorted(sorted: &[f64]) -> f64 {
|
||||
}
|
||||
}
|
||||
|
||||
fn std_dev(data: &[f64]) -> f64 {
|
||||
pub(crate) fn std_dev(data: &[f64]) -> f64 {
|
||||
if data.is_empty() {
|
||||
return 0.0;
|
||||
}
|
||||
@@ -250,7 +250,7 @@ fn std_dev(data: &[f64]) -> f64 {
|
||||
variance.sqrt()
|
||||
}
|
||||
|
||||
fn percentile_sorted(sorted_data: &[f64], p: f64) -> f64 {
|
||||
pub(crate) fn percentile_sorted(sorted_data: &[f64], p: f64) -> f64 {
|
||||
if sorted_data.is_empty() {
|
||||
return 0.0;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,572 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
//! Multimodal preprocessing benchmark (`vllm-bench mm-processor`), mirroring
|
||||
//! `vllm bench mm-processor`.
|
||||
|
||||
use std::collections::{BTreeMap, HashMap};
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use anyhow::{Context, Result};
|
||||
use clap::Args;
|
||||
use futures::future::join_all;
|
||||
use serde::Serialize;
|
||||
use tokio::sync::Semaphore;
|
||||
|
||||
use vllm_chat::{
|
||||
ChatContent, ChatContentPart, ChatLlm, ChatMessage, ChatRequest, LoadModelBackendsOptions,
|
||||
MultiModalTimingStats, load_model_backends,
|
||||
};
|
||||
use vllm_engine_core_client::{EngineCoreClient, EngineCoreClientConfig, TransportMode};
|
||||
use vllm_llm::Llm;
|
||||
use vllm_managed_engine::cli::ManagedEngineArgs;
|
||||
use vllm_managed_engine::{ManagedEngineConfig, ManagedEngineHandle};
|
||||
use vllm_text::TextLlm;
|
||||
|
||||
use crate::config::RangeRatio;
|
||||
use crate::datasets::SampleRequest;
|
||||
use crate::datasets::random_mm::{
|
||||
MmBucketKey, MmLimitPerPrompt, generate_random_mm_dataset, parse_bucket_config,
|
||||
parse_limit_mm_per_prompt,
|
||||
};
|
||||
use crate::error::BenchError;
|
||||
use crate::metrics::calculator::{mean, median_sorted, percentile_sorted, sort_clone, std_dev};
|
||||
use crate::tokenizer::{TokenizerKind, load_tokenizer};
|
||||
|
||||
/// Seconds -> milliseconds.
|
||||
const SEC_TO_MS: f64 = 1000.0;
|
||||
|
||||
/// Preferred column order; other stages appended alphabetically.
|
||||
const STAGE_ORDER: &[&str] = &[
|
||||
"preprocessor_total_ms",
|
||||
"media_fetch_ms",
|
||||
"preprocess_image_ms",
|
||||
"preprocess_video_ms",
|
||||
"preprocess_audio_ms",
|
||||
"prompt_expansion_ms",
|
||||
];
|
||||
|
||||
/// Time to wait for managed engines to register.
|
||||
const ENGINE_READY_TIMEOUT: Duration = Duration::from_secs(600);
|
||||
|
||||
/// Maximum time to wait for the managed engine to drain on shutdown.
|
||||
const ENGINE_SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(120);
|
||||
|
||||
#[derive(Args, Debug, Clone)]
|
||||
pub struct MmProcessorArgs {
|
||||
/// Model to serve and benchmark (Hugging Face id or local path).
|
||||
#[arg(long)]
|
||||
pub model: String,
|
||||
|
||||
/// Number of prompts to process, excluding warmups.
|
||||
#[arg(long, default_value_t = 10)]
|
||||
pub num_prompts: usize,
|
||||
|
||||
/// Number of warmup prompts processed and discarded before timing.
|
||||
#[arg(long, default_value_t = 1)]
|
||||
pub num_warmups: usize,
|
||||
|
||||
/// Comma-separated percentiles to report, e.g. "50,90,99".
|
||||
#[arg(long, default_value = "99")]
|
||||
pub metric_percentiles: String,
|
||||
|
||||
/// Path to write the aggregate stats as JSON.
|
||||
#[arg(long)]
|
||||
pub output_json: Option<String>,
|
||||
|
||||
/// Text input length per prompt (before range scaling).
|
||||
#[arg(long, default_value_t = 1024)]
|
||||
pub random_input_len: usize,
|
||||
|
||||
/// Expected output length per prompt.
|
||||
#[arg(long, default_value_t = 128)]
|
||||
pub random_output_len: usize,
|
||||
|
||||
/// Prefix token length per prompt.
|
||||
#[arg(long, default_value_t = 0)]
|
||||
pub random_prefix_len: usize,
|
||||
|
||||
/// Input/output length range ratio: a float or `{"input": i, "output": o}`.
|
||||
#[arg(long, default_value = "0.0")]
|
||||
pub random_range_ratio: String,
|
||||
|
||||
/// Base number of multimodal items per request.
|
||||
#[arg(long, default_value_t = 1)]
|
||||
pub random_mm_base_items_per_request: usize,
|
||||
|
||||
/// Range ratio (in [0, 1]) for the number of mm items per request.
|
||||
#[arg(long, default_value_t = 0.0)]
|
||||
pub random_mm_num_mm_items_range_ratio: f64,
|
||||
|
||||
/// Per-modality item caps, e.g. `{"image": 255, "video": 1}`.
|
||||
#[arg(long, default_value = "{\"image\": 255, \"video\": 1}")]
|
||||
pub random_mm_limit_mm_per_prompt: String,
|
||||
|
||||
/// Bucket config, e.g. `{(256,256,1): 0.5, (720,1280,1): 0.5}`.
|
||||
#[arg(long, default_value = "{(256,256,1): 0.5, (720,1280,1): 0.5}")]
|
||||
pub random_mm_bucket_config: String,
|
||||
|
||||
/// Seed for dataset generation.
|
||||
#[arg(long)]
|
||||
pub seed: Option<u64>,
|
||||
|
||||
/// Request-id prefix for per-request timing keys.
|
||||
#[arg(long, default_value = "mm-proc-")]
|
||||
pub request_id_prefix: String,
|
||||
|
||||
/// Maximum concurrent requests (default 1, matching Python's serial driver).
|
||||
#[arg(long, default_value_t = 1)]
|
||||
pub max_concurrency: usize,
|
||||
|
||||
/// Trust remote code for the tokenizer and the managed engine.
|
||||
#[arg(long, default_value_t = false)]
|
||||
pub trust_remote_code: bool,
|
||||
|
||||
/// Optional chat-template override (inline text or path).
|
||||
#[arg(long)]
|
||||
pub chat_template: Option<String>,
|
||||
|
||||
/// Managed headless-engine options.
|
||||
#[command(flatten)]
|
||||
pub engine: ManagedEngineArgs,
|
||||
}
|
||||
|
||||
/// Entry point for `vllm-bench mm-processor`.
|
||||
pub async fn run_mm_processor(args: MmProcessorArgs, timing: MultiModalTimingStats) -> Result<()> {
|
||||
if args.engine.data_parallel_size_local == Some(0) {
|
||||
anyhow::bail!(
|
||||
"`--data-parallel-size-local 0` is not supported; the benchmark requires \
|
||||
a local managed engine"
|
||||
);
|
||||
}
|
||||
|
||||
// Spawn the managed headless Python engine (real weights, real encoder).
|
||||
let handshake_port = args
|
||||
.engine
|
||||
.resolve_handshake_port()
|
||||
.context("failed to allocate handshake port")?;
|
||||
let mut python_args = args.engine.python_args.clone();
|
||||
if args.trust_remote_code {
|
||||
python_args.push("--trust-remote-code".to_string());
|
||||
}
|
||||
let engine_config = ManagedEngineConfig {
|
||||
python: args.engine.python.clone(),
|
||||
model: args.model.clone(),
|
||||
handshake_host: args.engine.handshake_host.clone(),
|
||||
handshake_port,
|
||||
data_parallel_size: args.engine.data_parallel_size,
|
||||
python_args,
|
||||
};
|
||||
eprintln!(
|
||||
"Spawning managed headless engine: python={} model={} data_parallel_size={}",
|
||||
engine_config.python, engine_config.model, engine_config.data_parallel_size
|
||||
);
|
||||
let engine = ManagedEngineHandle::spawn(engine_config)
|
||||
.await
|
||||
.context("failed to start managed Python headless engine")?;
|
||||
|
||||
let result = run_mm_processor_with_engine(args, handshake_port, &timing).await;
|
||||
let shutdown_result = engine
|
||||
.shutdown(ENGINE_SHUTDOWN_TIMEOUT)
|
||||
.await
|
||||
.context("failed to shut down managed engine");
|
||||
result?;
|
||||
shutdown_result?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn run_mm_processor_with_engine(
|
||||
args: MmProcessorArgs,
|
||||
handshake_port: u16,
|
||||
timing: &MultiModalTimingStats,
|
||||
) -> Result<()> {
|
||||
let range_ratio =
|
||||
RangeRatio::parse(&args.random_range_ratio).context("invalid --random-range-ratio")?;
|
||||
let limit: MmLimitPerPrompt = parse_limit_mm_per_prompt(&args.random_mm_limit_mm_per_prompt)
|
||||
.context("invalid --random-mm-limit-mm-per-prompt")?;
|
||||
let buckets: Vec<(MmBucketKey, f64)> = parse_bucket_config(&args.random_mm_bucket_config)
|
||||
.context("invalid --random-mm-bucket-config")?;
|
||||
let percentiles = parse_percentiles(&args.metric_percentiles)?;
|
||||
let seed = args.seed.unwrap_or(0);
|
||||
|
||||
// Offline tokenizer: local Hugging Face tokenizer or built-in tiktoken.
|
||||
let tokenizer = load_tokenizer(&args.model, args.trust_remote_code, None)
|
||||
.await
|
||||
.context("failed to load tokenizer")?;
|
||||
|
||||
// Frontend backends: model configs + multimodal processor metadata.
|
||||
let backends = load_model_backends(
|
||||
&args.model,
|
||||
LoadModelBackendsOptions {
|
||||
chat_template: args.chat_template.clone(),
|
||||
..LoadModelBackendsOptions::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.context("failed to load model backends")?;
|
||||
|
||||
if backends.chat_backend.multimodal_model_info().is_none() {
|
||||
anyhow::bail!(
|
||||
"model `{}` has no multimodal processor registered",
|
||||
args.model
|
||||
);
|
||||
}
|
||||
|
||||
// Connect to the engine over the same transport the serving frontend uses.
|
||||
let client = EngineCoreClient::connect(EngineCoreClientConfig {
|
||||
transport_mode: TransportMode::HandshakeOwner {
|
||||
handshake_address: format!("tcp://{}:{}", args.engine.handshake_host, handshake_port),
|
||||
advertised_host: args.engine.handshake_host.clone(),
|
||||
engine_count: args
|
||||
.engine
|
||||
.data_parallel_size_local
|
||||
.unwrap_or(args.engine.data_parallel_size),
|
||||
ready_timeout: ENGINE_READY_TIMEOUT,
|
||||
local_input_address: None,
|
||||
local_output_address: None,
|
||||
},
|
||||
coordinator_mode: None,
|
||||
model_name: args.model.clone(),
|
||||
client_index: 0,
|
||||
})
|
||||
.await
|
||||
.context("failed to connect to engine core")?;
|
||||
|
||||
let chat = ChatLlm::new(
|
||||
TextLlm::new(Llm::new(client), backends.text_backend),
|
||||
backends.chat_backend,
|
||||
);
|
||||
|
||||
let semaphore = Arc::new(Semaphore::new(args.max_concurrency));
|
||||
|
||||
// Warmup: generate a separate dataset, process it, then drain and discard
|
||||
// the collected timings so cold caches do not skew the results.
|
||||
if args.num_warmups > 0 {
|
||||
let warmup = generate_dataset(
|
||||
&tokenizer,
|
||||
args.num_warmups,
|
||||
seed.wrapping_add(1),
|
||||
&args,
|
||||
range_ratio,
|
||||
&limit,
|
||||
&buckets,
|
||||
)?;
|
||||
eprintln!("Processing {} warmup requests...", warmup.len());
|
||||
run_batch(&chat, &warmup, &semaphore).await;
|
||||
let _ = timing.stat();
|
||||
}
|
||||
|
||||
let samples = generate_dataset(
|
||||
&tokenizer,
|
||||
args.num_prompts,
|
||||
seed,
|
||||
&args,
|
||||
range_ratio,
|
||||
&limit,
|
||||
&buckets,
|
||||
)?;
|
||||
eprintln!("Processing {} requests...", samples.len());
|
||||
let start_time = Instant::now();
|
||||
let outcomes = run_batch(&chat, &samples, &semaphore).await;
|
||||
let total_time = start_time.elapsed();
|
||||
|
||||
let completed = outcomes.iter().filter(|outcome| outcome.is_ok()).count();
|
||||
let failed = outcomes.len() - completed;
|
||||
let e2el_times: Vec<f64> = outcomes
|
||||
.iter()
|
||||
.filter_map(|outcome| outcome.as_ref().ok().map(|elapsed| elapsed.as_secs_f64()))
|
||||
.collect();
|
||||
|
||||
let per_request = timing.stat();
|
||||
let stats = aggregate_stats(per_request, &percentiles);
|
||||
|
||||
eprintln!(
|
||||
"Processed {} requests (+{} warmup) for model `{}`",
|
||||
samples.len(),
|
||||
args.num_warmups,
|
||||
args.model
|
||||
);
|
||||
print_report(&stats, &percentiles);
|
||||
print_e2el_summary(completed, failed, total_time, &e2el_times, &percentiles);
|
||||
|
||||
if let Some(path) = &args.output_json {
|
||||
write_result_json(path, completed, failed, &e2el_times, &percentiles, stats)?;
|
||||
eprintln!("Wrote stats JSON to {path}");
|
||||
}
|
||||
|
||||
chat.shutdown().await.context("failed to shut down chat facade")?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Submit every sample through the chat pipeline and drive each stream to
|
||||
/// completion, returning per-request wall-clock latencies.
|
||||
async fn run_batch(
|
||||
chat: &ChatLlm,
|
||||
samples: &[SampleRequest],
|
||||
semaphore: &Arc<Semaphore>,
|
||||
) -> Vec<Result<Duration>> {
|
||||
let submissions = samples.iter().map(|sample| {
|
||||
let request = build_chat_request(sample);
|
||||
let semaphore = semaphore.clone();
|
||||
async move {
|
||||
let _permit = semaphore.acquire().await.context("semaphore closed")?;
|
||||
let request = request?;
|
||||
let start = Instant::now();
|
||||
chat.chat(request)
|
||||
.await
|
||||
.context("failed to submit request")?
|
||||
.collect_message()
|
||||
.await
|
||||
.context("request failed")?;
|
||||
Ok(start.elapsed())
|
||||
}
|
||||
});
|
||||
join_all(submissions).await
|
||||
}
|
||||
|
||||
fn generate_dataset(
|
||||
tokenizer: &TokenizerKind,
|
||||
num_requests: usize,
|
||||
seed: u64,
|
||||
args: &MmProcessorArgs,
|
||||
range_ratio: RangeRatio,
|
||||
limit: &MmLimitPerPrompt,
|
||||
buckets: &[(MmBucketKey, f64)],
|
||||
) -> Result<Vec<SampleRequest>> {
|
||||
generate_random_mm_dataset(
|
||||
tokenizer,
|
||||
num_requests,
|
||||
args.random_input_len,
|
||||
args.random_output_len,
|
||||
args.random_prefix_len,
|
||||
range_ratio,
|
||||
seed,
|
||||
&args.request_id_prefix,
|
||||
args.random_mm_base_items_per_request,
|
||||
args.random_mm_num_mm_items_range_ratio,
|
||||
limit,
|
||||
buckets,
|
||||
false,
|
||||
)
|
||||
.map_err(|e| anyhow::anyhow!("{e}"))
|
||||
}
|
||||
|
||||
/// Build a chat request from a sample (text prompt + `image_url` parts).
|
||||
fn build_chat_request(sample: &SampleRequest) -> Result<ChatRequest> {
|
||||
let mut request = ChatRequest::for_test();
|
||||
request.request_id = sample.request_id.clone().unwrap_or_else(|| "mm-proc".to_string());
|
||||
request.sampling_params.temperature = Some(0.0);
|
||||
request.sampling_params.max_tokens = Some(u32::try_from(sample.expected_output_len)?);
|
||||
|
||||
let mut parts = vec![ChatContentPart::text(sample.prompt.as_ref())];
|
||||
if let Some(mm_items) = &sample.multi_modal_content {
|
||||
for fragment in mm_items.iter() {
|
||||
let url = extract_image_url(fragment)?;
|
||||
parts.push(ChatContentPart::image_url(url));
|
||||
}
|
||||
}
|
||||
request.messages = vec![ChatMessage::user(ChatContent::from(parts))];
|
||||
Ok(request)
|
||||
}
|
||||
|
||||
/// Extract the `image_url.url` string from a pre-serialized content fragment
|
||||
/// (`{"type":"image_url","image_url":{"url":"data:image/jpeg;base64,..."}}`).
|
||||
fn extract_image_url(fragment: &str) -> Result<String> {
|
||||
let value: serde_json::Value = serde_json::from_str(fragment)
|
||||
.with_context(|| format!("invalid multimodal fragment: {fragment}"))?;
|
||||
let url = value
|
||||
.get("image_url")
|
||||
.and_then(|v| v.get("url"))
|
||||
.and_then(|v| v.as_str())
|
||||
.ok_or_else(|| BenchError::Config("fragment missing image_url.url".to_string()))?;
|
||||
Ok(url.to_string())
|
||||
}
|
||||
|
||||
fn parse_percentiles(raw: &str) -> Result<Vec<f64>> {
|
||||
let mut values = Vec::new();
|
||||
for part in raw.split(',') {
|
||||
let part = part.trim();
|
||||
if part.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let value: f64 = part.parse().with_context(|| format!("invalid percentile `{part}`"))?;
|
||||
if !(0.0..=100.0).contains(&value) {
|
||||
anyhow::bail!("percentile `{part}` out of range [0, 100]");
|
||||
}
|
||||
values.push(value);
|
||||
}
|
||||
Ok(values)
|
||||
}
|
||||
|
||||
/// Aggregate per-request stage timings into `{stage_ms: {statistic: ms}}`,
|
||||
/// matching the Python `mm_processor_stats` schema orientation.
|
||||
fn aggregate_stats(
|
||||
per_request: HashMap<String, HashMap<String, f64>>,
|
||||
percentiles: &[f64],
|
||||
) -> BTreeMap<String, BTreeMap<String, f64>> {
|
||||
let mut stage_values: BTreeMap<String, Vec<f64>> = BTreeMap::new();
|
||||
for stage_map in per_request.into_values() {
|
||||
for (stage, seconds) in stage_map {
|
||||
stage_values.entry(stage).or_default().push(seconds);
|
||||
}
|
||||
}
|
||||
|
||||
stage_values
|
||||
.iter()
|
||||
.map(|(stage, values)| {
|
||||
let sorted = sort_clone(values);
|
||||
let mut stat_map = BTreeMap::new();
|
||||
stat_map.insert("mean".to_string(), mean(values) * SEC_TO_MS);
|
||||
stat_map.insert("median".to_string(), median_sorted(&sorted) * SEC_TO_MS);
|
||||
stat_map.insert("std".to_string(), std_dev(values) * SEC_TO_MS);
|
||||
for p in percentiles {
|
||||
stat_map.insert(format!("p{p}"), percentile_sorted(&sorted, *p) * SEC_TO_MS);
|
||||
}
|
||||
(stage.replace("_secs", "_ms"), stat_map)
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Print a right-aligned plain-text table (pandas `to_string(index=False)` style).
|
||||
fn print_table(rows: &[Vec<String>]) {
|
||||
let ncols = rows.first().map_or(0, Vec::len);
|
||||
let mut widths = vec![0usize; ncols];
|
||||
for row in rows {
|
||||
for (i, cell) in row.iter().enumerate() {
|
||||
widths[i] = widths[i].max(cell.len());
|
||||
}
|
||||
}
|
||||
for row in rows {
|
||||
let line = row
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(i, cell)| format!("{cell:>width$}", width = widths[i]))
|
||||
.collect::<Vec<_>>()
|
||||
.join(" ");
|
||||
println!("{line}");
|
||||
}
|
||||
}
|
||||
|
||||
/// Print a plain-text table with one row per stage and one column per
|
||||
/// statistic (mirrors the Python pandas report layout).
|
||||
fn print_report(stats: &BTreeMap<String, BTreeMap<String, f64>>, percentiles: &[f64]) {
|
||||
// Column order: mean/median/std then the requested percentiles, like Python.
|
||||
let mut columns: Vec<(String, String)> =
|
||||
[("mean", "Mean"), ("median", "Median"), ("std", "Std")]
|
||||
.into_iter()
|
||||
.map(|(key, label)| (key.to_string(), label.to_string()))
|
||||
.collect();
|
||||
columns.extend(percentiles.iter().map(|p| (format!("p{p}"), format!("P{p:?}"))));
|
||||
for key in stats.values().flat_map(|map| map.keys()) {
|
||||
if !columns.iter().any(|(k, _)| k == key) {
|
||||
columns.push((key.clone(), key.clone()));
|
||||
}
|
||||
}
|
||||
|
||||
let mut stages: Vec<&String> = stats.keys().collect();
|
||||
stages.sort_by_key(|stage| {
|
||||
STAGE_ORDER
|
||||
.iter()
|
||||
.position(|s| s == stage)
|
||||
.map(|i| (i as isize, (*stage).clone()))
|
||||
.unwrap_or((isize::MAX, (*stage).clone()))
|
||||
});
|
||||
|
||||
let mut rows: Vec<Vec<String>> = vec![
|
||||
std::iter::once("Stage".to_string())
|
||||
.chain(columns.iter().map(|(_, label)| label.clone()))
|
||||
.collect(),
|
||||
];
|
||||
for stage in &stages {
|
||||
let mut row = vec![(*stage).clone()];
|
||||
for (key, _) in &columns {
|
||||
row.push(
|
||||
stats[*stage]
|
||||
.get(key)
|
||||
.map(|v| format!("{v:.2}"))
|
||||
.unwrap_or_else(|| "-".to_string()),
|
||||
);
|
||||
}
|
||||
rows.push(row);
|
||||
}
|
||||
println!("MM Processor Metrics:");
|
||||
print_table(&rows);
|
||||
}
|
||||
|
||||
/// Print the end-to-end latency summary (mirrors the Python report layout).
|
||||
fn print_e2el_summary(
|
||||
completed: usize,
|
||||
failed: usize,
|
||||
total_time: Duration,
|
||||
e2el_times_secs: &[f64],
|
||||
percentiles: &[f64],
|
||||
) {
|
||||
println!("Completed: {completed}, Failed: {failed}");
|
||||
if e2el_times_secs.is_empty() {
|
||||
println!("End-to-End Latency (ms): no completed requests");
|
||||
return;
|
||||
}
|
||||
|
||||
let e2el_ms: Vec<f64> = e2el_times_secs.iter().map(|secs| secs * SEC_TO_MS).collect();
|
||||
let sorted = sort_clone(&e2el_ms);
|
||||
let mut rows = vec![
|
||||
vec!["Metric".to_string(), "Value (ms)".to_string()],
|
||||
vec!["Mean".to_string(), format!("{:.2}", mean(&e2el_ms))],
|
||||
vec![
|
||||
"Median".to_string(),
|
||||
format!("{:.2}", median_sorted(&sorted)),
|
||||
],
|
||||
vec!["Std".to_string(), format!("{:.2}", std_dev(&e2el_ms))],
|
||||
];
|
||||
for p in percentiles {
|
||||
rows.push(vec![
|
||||
format!("P{p:?}"),
|
||||
format!("{:.2}", percentile_sorted(&sorted, *p)),
|
||||
]);
|
||||
}
|
||||
println!("End-to-End Latency (ms):");
|
||||
print_table(&rows);
|
||||
let throughput = completed as f64 / total_time.as_secs_f64();
|
||||
println!("Request throughput: {throughput:.3} req/s");
|
||||
}
|
||||
|
||||
/// JSON result schema mirroring `vllm bench mm-processor`.
|
||||
#[derive(Serialize)]
|
||||
struct BenchmarkResultJson {
|
||||
completed: usize,
|
||||
failed: usize,
|
||||
mean_e2el_ms: f64,
|
||||
median_e2el_ms: f64,
|
||||
std_e2el_ms: f64,
|
||||
percentiles_e2el_ms: Vec<(f64, f64)>,
|
||||
/// `{stage_ms: {statistic: ms}}`, keyed by stage like the Python schema.
|
||||
mm_processor_stats: BTreeMap<String, BTreeMap<String, f64>>,
|
||||
}
|
||||
|
||||
fn write_result_json(
|
||||
path: &str,
|
||||
completed: usize,
|
||||
failed: usize,
|
||||
e2el_times_secs: &[f64],
|
||||
percentiles: &[f64],
|
||||
mm_processor_stats: BTreeMap<String, BTreeMap<String, f64>>,
|
||||
) -> Result<()> {
|
||||
let e2el_ms: Vec<f64> = e2el_times_secs.iter().map(|secs| secs * SEC_TO_MS).collect();
|
||||
let sorted = sort_clone(&e2el_ms);
|
||||
let result = BenchmarkResultJson {
|
||||
completed,
|
||||
failed,
|
||||
mean_e2el_ms: mean(&e2el_ms),
|
||||
median_e2el_ms: median_sorted(&sorted),
|
||||
std_e2el_ms: std_dev(&e2el_ms),
|
||||
percentiles_e2el_ms: percentiles
|
||||
.iter()
|
||||
.map(|p| (*p, percentile_sorted(&sorted, *p)))
|
||||
.collect(),
|
||||
mm_processor_stats,
|
||||
};
|
||||
let json = serde_json::to_string_pretty(&result).context("failed to serialize result")?;
|
||||
std::fs::write(path, format!("{json}\n")).with_context(|| format!("failed to write {path}"))?;
|
||||
Ok(())
|
||||
}
|
||||
Reference in New Issue
Block a user