95 Commits
Author SHA1 Message Date
Tyrone bafb263664 [Bugfix][Benchmarks] Name the real cause when the /tokenize alignment probe fails (#59888) 2026-10-17 14:47:52 -04:00
HarshilandCyrus Leung c54dca96bd [Bugfix][Benchmark] Fix probe latency and duration in bench serve (#60511)
Signed-off-by: QHarshil <harshil_c@hotmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-15 05:50:30 +00:00
sungbin1015andFlora Feng 2ddde19e8e [Frontend] Support min_p in the Responses API (#48084)
Signed-off-by: sungbin1015 <sbin@solbox.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-08 20:38:17 +00:00
Harry MellorandClaude Opus 5.5 21d9758459 Remove Python 3.10 support now that it is EOL (#60402)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 02:30:52 -07:00
Wentao Ye 3e182185aa [Deprecation] Deprecate sonet dataset as scheduled (#60129)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-10-06 08:15:12 -04:00
czhu-cohereandaoshen02 6fddc9b21e [Profiler] Add per-session profiling controls to Python and Rust frontends (#57875)
Signed-off-by: czhu <conway.zhu@cohere.com>
Signed-off-by: Conway Zhu <conway.zhu@cohere.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-10-06 05:21:19 +00:00
af7f9488c2 [Benchmark] Add Responses API backend to vllm bench serve (#54628)
Signed-off-by: QHarshil <harshil_c@hotmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-09-28 05:04:11 +00:00
Harry MellorandClaude Opus 5.5 d636456b1f [Frontend] Remove the slow tokenizer mode (#58545)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-24 08:48:28 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
Taneem Ibrahimandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 8e08cef46e [Bugfix] Redact credentials from benchmark logs (#56662)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-14 18:58:33 -04:00
Taneem Ibrahim 3284af6bf1 [Bugfix] Reject 0 or non-positive max concurrency (#54887)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
2026-09-04 15:59:29 -04:00
maithilijoshi20maithilijoshi20mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
9e905f7450 [Bugfix] Account for client queue time in serve benchmarks (#54136)
Signed-off-by: maithilijoshi20 <maithilijoshi2003@gmail.com>
Signed-off-by: maithilijoshi20 <97733343+maithilijoshi20@users.noreply.github.com>
Signed-off-by: maithilijoshi20 <maithilij2003@gmail.com>
Co-authored-by: maithilijoshi20 <maithilijoshi2003@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-01 03:40:51 -07:00
Taneem Ibrahimandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 704f12aa1e [Pooling] Report input throughput for batched requests (#53213)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-22 09:01:40 -04:00
Stan WozniakandThomas Parnell 63550515b4 [Bugfix] Correct prompt lengths for timed_traces benchmark (#45423)
Signed-off-by: Stanislaw Wozniak <stw@zurich.ibm.com>
Co-authored-by: Thomas Parnell <tpa@zurich.ibm.com>
2026-08-13 21:51:24 +00:00
Guan-Ming Chiu 63e78ce365 [Benchmark] Add probe requests to vllm bench serve (#49611)
Signed-off-by: Guan-Ming (Wesley) Chiu <105915352+guan404ming@users.noreply.github.com>
2026-08-01 22:10:56 +08:00
mgazzandKevin H. Luu a454a1dd25 [Bugfix][Benchmarks] Restore --skip-tokenizer-init with custom dataset (#49180)
Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-07-24 04:30:19 -07:00
Martin HickeyandHarry Mellor 275b43183c [MyPy] Fix mypy for vllm/benchmarks (#39896)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-06-23 15:22:29 +00:00
eb28452b10 [Model] Add DiffusionGemma Support (#45163)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Martin Kukla <martin.kukla@cantab.net>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Dipika Sikka <dsikka@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
Co-authored-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
Co-authored-by: Alec Kohlhoff <134344302+aleckohlhoff@users.noreply.github.com>
Co-authored-by: Porras Huang <20535584+porrashuang@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: scoootscooob <167050519+scoootscooob@users.noreply.github.com>
2026-06-11 22:17:35 -07:00
Harry Mellor 3cc9fecd58 Deprecated 1st generation Qwen and QwenVL models (#45131)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-06-10 14:55:33 +00:00
89c6a41001 [Bench] Add BFCL dataset for vllm bench serve tool-calling workloads (#42457)
Signed-off-by: Li Zhang <lzhanga@amazon.com>
Co-authored-by: Li Zhang <lzhanga@amazon.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-06-09 23:59:18 -07:00
akii96 ac3409d162 [Benchmark] Auto-detect and correct client/server tokenizer mismatch for random dataset (#44708) 2026-06-08 06:10:20 -07:00
Albert Cheng e0081ef8cf [Benchmark] Enable reasoning-model (thinking) benchmarking via --chat-template-kwargs for client-rendered datasets (#44244)
Signed-off-by: Albert Cheng <albertching0112@gmail.com>
2026-06-02 22:49:51 -07:00
Animesh Trivedi bfb9ebc211 [Feature] Add support for timed trace replay in vllm bench serve to replay Moonshot and Alibaba workload traces (#39795)
Signed-off-by: Animesh Trivedi <Animesh.Trivedi@ibm.com>
2026-05-28 03:31:34 -07:00
noobHappylife 06a60d3dd0 Fix spec decode benchmark metrics (#41916)
Signed-off-by: noobhappylife <aratar1991@hotmail.com>
2026-05-07 09:23:21 -07:00
Sophie du Couédicandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> ff2c2bd80a [Docs]Add documentation for bench serve visualization arguments (#40539)
Signed-off-by: Sophie du Couédic <sop@zurich.ibm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-04-23 15:48:29 -07:00
Talor Abramovichandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 7b1e0b07d0 [Bugfix] Fix dataset name and path argument validation bug in vllm bench serve (#40288)
Signed-off-by: talora <talora@nvidia.com>
Signed-off-by: Talor Abramovich <talor19@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-04-21 06:14:28 -07:00
Matthias Gehre 1e4748c66a [Bugfix] Fix vllm bench serve to count multimodal tokens in "total input tokens" (#38654)
Signed-off-by: Matthias Gehre <matthias.gehre@amd.com>
2026-04-14 11:00:40 +00:00
RonaldBXu c9d838fc33 Adding deterministic lora benchmarking to vLLM Bench (#36057)
Signed-off-by: Ubuntu <ubuntu@ip-172-31-43-201.ap-northeast-1.compute.internal>
Signed-off-by: Ronald Xu <ronaldxu@amazon.com>
2026-03-18 16:02:03 +00:00
Wentao Ye be292b7c14 [Bug] Fix pooling model benchmark script (#36300)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-03-09 11:17:45 -04:00
Cyrus Leung 7196348157 [Bugfix] Fix Qwen-VL tokenizer implementation (#36140)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-03-05 08:07:19 -08:00
Sophie du CouédicandCyrus Leung 02acd16861 [Benchmarks] Plot benchmark timeline and requests statistics (#35220)
Signed-off-by: Sophie du Couédic <sop@zurich.ibm.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
2026-02-26 02:17:43 -08:00
Cyrus Leung 3bcd494ef4 [Redo] Add --trust-remote-code to dataset bench args (#34251)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-02-11 11:10:12 +08:00
mgazz 599e4335a4 Support benchmarking of Geospatial models (#33922)
Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>
2026-02-10 07:04:16 -08:00
Fan YangandFan Yang a1946570d8 add --insecure arg to the vllm bench to skip TLS (#34026)
Signed-off-by: Fan Yang <yan9fan@meta.com>
Co-authored-by: Fan Yang <yan9fan@meta.com>
2026-02-10 22:23:52 +08:00
Cyrus Leung 998e2d91f8 Revert #34208 (#34216) 2026-02-09 23:59:04 -08:00
Cyrus Leung f69b903b4c [Bugfix] Add --trust-remote-code to dataset bench args (#34208)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-02-09 22:37:50 -08:00
Ekagra RanjanandNicolò Lucchesi 1d5922fade [ASR] Fix audio benchmark and add RTFx metric (#32300)
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com>
2026-02-09 10:02:37 +00:00
Nick Hill 098b2d66fe [Benchmark] Don't default to temperature==0 in vllm bench serve (#32723)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-01-22 10:03:15 +00:00
Kebe 1bf1a34b19 [bench] add start_times field to vllm bench serve json result (#32667)
Signed-off-by: Kebe <mail@kebe7jun.com>
2026-01-22 07:10:14 +00:00
Nicolò Lucchesi 22970c1626 [Misc] Disable default --ready-check-timeout-sec extra call in vllm bench (#30975)
Signed-off-by: NickLucche <nlucches@redhat.com>
2026-01-12 01:58:21 -08:00
d49899732e [Spec Decode][UX] Add acceptance stats to vllm bench serve report (#31739)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com>
2026-01-06 21:21:42 +00:00
Elvir Crnčević dba95378a6 Report error log after vllm bench serve (#31808)
Signed-off-by: Elvir Crncevic <elvircrn@gmail.com>
2026-01-06 20:24:19 +00:00
Ning Xie 3b8f31b362 [benchmark] use model card root instead of id (#31329)
Signed-off-by: Andy Xie <andy.xning@gmail.com>
2025-12-26 10:55:56 +08:00
Ming Yang 3bb9561928 Revert "[bench] Support common prefix len config (for decode-only bench)" (#31240)
Signed-off-by: Ming Yang <minos.future@gmail.com>
2025-12-23 21:17:23 -08:00
Michael Goin 519ef9a911 [UX] Make vllm bench serve discover model by default and use --input-len (#30816)
Signed-off-by: mgoin <mgoin64@gmail.com>
2025-12-17 01:55:30 -08:00
Vensen add4b0ca44 [Bugfix][benchmarks] Fix input token calculation for rerank benchmark metrics (#30596)
Signed-off-by: vensen <vensenmu@gmail.com>
2025-12-14 14:57:15 +00:00
Matthew Bonanni 794a7875ee [Misc] Consistent case for vllm bench serve results (#30403)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
2025-12-10 09:44:02 -08:00
Benjamin Chislettgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Harry Mellor
e858bfe051 [Cleanup] Refactor profiling env vars into a CLI config (#29912)
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2025-12-09 13:29:33 -05:00
Rohan Potdar 40a046cd82 [Bugfix]: Fix TokenizerLike interface (#30009)
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
2025-12-05 20:56:40 -08:00
Ming Yang f16356fe36 [bench] Support common prefix len config (for decode-only bench) (#29934)
Signed-off-by: Ming Yang <minos.future@gmail.com>
2025-12-05 10:26:52 +00:00