[Pooling] Fix Rust pooling endpoint for benchmark (#53352)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
This commit is contained in:
@@ -419,7 +419,7 @@ vllm-bench \
|
||||
| --------- | ------------- | ------------- |
|
||||
| `openai-embeddings` | `/v1/embeddings` | Text embedding (accepts text or token IDs) |
|
||||
| `openai-embeddings-chat` | `/v1/embeddings` | Chat-format embedding (supports multimodal content) |
|
||||
| `vllm-pooling` | `/v1/pooling` | vLLM native pooling endpoint |
|
||||
| `vllm-pooling` | `/pooling` | vLLM native pooling endpoint |
|
||||
| `vllm-rerank` | `/v1/rerank` | vLLM reranking (query from prompt, documents via `--extra-body`) |
|
||||
|
||||
Pooling backends are non-streaming and report E2EL (end-to-end latency) only. Use `--dataset-name sharegpt`, `sonnet`, or `hf` for text-based embedding/rerank benchmarks, or `random` for token-ID-based embedding benchmarks.
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
//! - `openai-embeddings`: Standard OpenAI `/v1/embeddings` with text input
|
||||
//! - `openai-embeddings-chat`: OpenAI `/v1/embeddings` with chat message format (supports
|
||||
//! multimodal)
|
||||
//! - `vllm-pooling`: vLLM `/v1/pooling` endpoint
|
||||
//! - `vllm-pooling`: vLLM `/pooling` endpoint
|
||||
//! - `vllm-rerank`: vLLM `/v1/rerank` endpoint (query + documents)
|
||||
|
||||
use std::time::Instant;
|
||||
|
||||
@@ -682,7 +682,7 @@ impl BenchServeArgs {
|
||||
BackendKind::OpenaiEmbeddings | BackendKind::OpenaiEmbeddingsChat => {
|
||||
"/v1/embeddings".to_string()
|
||||
}
|
||||
BackendKind::VllmPooling => "/v1/pooling".to_string(),
|
||||
BackendKind::VllmPooling => "/pooling".to_string(),
|
||||
BackendKind::VllmRerank => "/v1/rerank".to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user