f2376cdc4d
[Fast Start] Preload the FlashInfer autotune table on the weight cache daemon ( #60085 )
...
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-10-10 15:48:06 +00:00
Misha Goin and Claude Opus 5.5
29e4d2d42e
[Frontend][Build] Add vllm download-kernels for FlashInfer precompiled kernels ( #58765 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Signed-off-by: Misha Goin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 21:22:35 -07:00
f226af0e11
[Bugfix] Eliminate frontend ZMQ port TOCTOU with inherited listeners ( #54113 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 06:25:27 +00:00
9d2a6f52b2
[Rust Frontend] Expose gRPC port in Python vllm serve ( #59659 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Signed-off-by: Alec Flowers <aflowers@nvidia.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-10-02 08:14:37 +00:00
e9be5323f0
[Perf] Integrate flash-maxsim Triton kernels for late-interaction scoring ( #40337 )
...
Signed-off-by: roi.pony <roi.pony@ibm.com >
Co-authored-by: roi.pony <roi.pony@ibm.com >
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
2026-10-01 14:06:43 -04:00
af5b4857e1
[Fast Start] Support PP ( #55477 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Xun Sun <UNIDY2002@outlook.com >
2026-09-29 09:37:43 +00:00
Xun Sun and Isotr0py
e7900156e1
[Fast Start] Add /health endpoint for the weight cache daemon ( #58552 )
...
Signed-off-by: Xun Sun <UNIDY2002@outlook.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-09-27 17:48:54 +00:00
Mark McLoughlin and AI Assistant
45d56c0baf
[Core] Model console logging as CLI configuration ( #57205 )
...
Add `--logging-config` CLI argument which can be supplied as
JSON or using dotted arguments. The `--log-level` argument
is provided for convenience, and `--log-config-file` is deprecated
in favor of `--logging-config.pylogging_config_file`.
Signed-off-by: Mark McLoughlin <markmc@redhat.com >
Co-authored-by: AI Assistant <noreply@openai.com >
2026-09-25 11:36:59 +01:00
Isotr0py
24928a4d62
[UX][Frontend] Introduce vllm preload cli for fast restart ( #56680 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
2026-09-25 06:56:39 +00:00
13058faa49
[Frontend] Add reusable TP1 initialized-engine snapshots ( #51360 )
...
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com >
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
Co-authored-by: elehayym <52448798+Yuzu23@users.noreply.github.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-21 23:18:49 +00:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
Misha Goin
0384e72693
[UX] Add thinking support to vllm chat ( #57045 )
2026-09-16 09:54:10 +01:00
ALEX LILUZ and Jiangyun Zhu
5be911ee87
[Bugfix] Load reasoning parser plugins before headless engine config ( #53124 )
...
Signed-off-by: alexliluz <49665315+alexliluz@users.noreply.github.com >
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com >
2026-09-15 18:31:25 +08:00
wang.yuqi
67111973ee
[Frontend] Move grpc_server to launchers. ( #56746 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-09-14 05:03:08 +00:00
Cenab Batu Bora and Codex
db723d245a
[Bugfix] Honor explicit empty and zero CLI arguments ( #55710 )
...
Signed-off-by: Cenab Batu Bora <40250311+cenab@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-10 15:07:42 +00:00
zvier
8c34a37232
[Refactor] Extract maybe_run_omni() and defer CLI imports past omni early-return ( #56103 )
...
Signed-off-by: Jeff Zvier <zvier20@gmail.com >
2026-09-10 03:21:58 +00:00
wang.yuqi
735daf8ef6
[Frontend] Move cli_args.py and dp_supervisor.py out openai folder ( #53659 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-08-25 07:02:32 +00:00
wang.yuqi
9cef631f30
[Frontend] Move run_batch.py out openai folder ( #53500 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-08-24 15:04:10 +00:00
wang.yuqi
eac636a7fa
[Frontend] Move api_server.py out openai folder ( #52131 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-08-19 09:43:46 +00:00
Connor Carpenter and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
fa722b9f01
[Rust Frontend][gRPC] Add explicit data-parallel rank routing ( #51178 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-10 15:05:24 -07:00
7b9f2dad89
[Frontend] Watch frontend processes during engine startup ( #43417 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-06 11:17:24 -07:00
Ning Xie
726ef437a1
[chore] delete useless code ( #49424 )
...
Signed-off-by: Andy Xie <andy.xning@gmail.com >
2026-07-31 11:54:08 -07:00
Bugen Zhao
01661cc57f
[Rust][Benchmark] Make vllm bench serve Rust delegation opt-in ( #50081 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-28 16:21:12 +00:00
Harry Mellor
3e74c60b9c
[Docs] Use gen-files for generated docs content ( #49587 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-25 14:43:21 +00:00
Bugen Zhao and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
c01618fdc8
[Rust][Benchmark] Integrate vllm-bench to vllm-rs & vllm CLI ( #48930 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-20 09:31:25 +00:00
Bugen Zhao
ab3b6d97aa
[Frontend] Limit SO_REUSEPORT to multi-worker serving ( #47529 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-04 01:26:24 +00:00
Benjamin Chislett
455f25aa13
[CLI] Add flag to print TTFT and TPS in vllm chat ( #46775 )
...
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com >
2026-06-26 22:15:10 -07:00
Martin Hickey and Harry Mellor
275b43183c
[MyPy] Fix mypy for vllm/benchmarks ( #39896 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-06-23 15:22:29 +00:00
Bugen Zhao and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
e2c58570ea
[Rust Frontend] Support hybrid/external DP LB in Python supervised bootstrap ( #45805 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-06-17 07:32:40 +00:00
wang.yuqi
d01d0b4646
[Frontend] Consolidate online serving utils. ( #44479 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-04 06:49:31 +00:00
Vadim Gimpelson
5d126dd155
[Bugfix] Exclude Ray DP from #42585 's deferred port allocation ( #43864 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
2026-05-28 15:55:14 +00:00
Vadim Gimpelson and gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
812e7e7364
[Bugfix][V1] Fix TOCTOU race causing intermittent EADDRINUSE on multi-API-server DP startup ( #42585 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
Signed-off-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-05-26 14:06:00 -07:00
0ddd7dd656
[Frontend] DP Supervisor ( #40841 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com >
Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: robertgshaw2-redhat <robertgshaw2@gmail.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-05-21 20:33:16 -07:00
Nick Hill and Bugen Zhao
f2ace1d57d
[Frontend][RFC] Rust front-end integration ( #40848 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
2026-05-21 12:24:48 +08:00
Alex Brooks
5e11b40365
[Frontend] Delegate to vLLM Omni When --omni Passed ( #40744 )
...
Signed-off-by: Alex Brooks <albrooks@redhat.com >
2026-04-24 12:30:00 -04:00
wang.yuqi
3975eb6de6
Revert "[Startup] Parallelize torch/transformers import + weight prefetch + forkserver prewarm" ( #40438 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-04-21 08:47:18 +00:00
Simon Mo
8256833fe6
[Startup] Parallelize torch/transformers import + weight prefetch + forkserver prewarm ( #40331 )
...
Signed-off-by: simon-mo <simon@inferact.ai >
2026-04-21 10:49:32 +08:00
Michael Goin
1948d0c467
[UX] Defer some imports on CLI paths to save ~2s ( #40056 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
2026-04-16 19:48:37 -07:00
Nick Hill
c8d98f81f6
[Core] Simplify API server handshake ( #39364 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-04-09 18:56:15 +08:00
Chang Su
acf7292bf2
[Misc] Move --grpc CLI argument into make_arg_parser ( #38570 )
...
Signed-off-by: Chang Su <chang.s.su@oracle.com >
2026-03-31 03:24:05 -07:00
fangyuchu and Nick Hill
b5e608258e
[Refactor] Unify engine process monitoring in engine manager and add Ray backend support ( #35862 )
...
Signed-off-by: fangyuchu <fangyuchu@qq.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-03-30 10:16:09 -07:00
Nick Hill
72cad44d3c
[Frontend] Move APIServerProcessManager target server fn ( #38115 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-03-25 18:14:41 +00:00
Sage
1c0aabdeb0
[Bugfix] Suppress spurious CPU KV cache warning in launch render ( #37911 )
...
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com >
2026-03-24 12:36:18 +00:00
Brandon Pelfrey and Nick Hill
80b70884eb
Add tensor IPC transfer mechanism for multimodal data ( #32104 )
...
Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com >
Signed-off-by: Brandon Pelfrey <brandonpelfrey@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-03-21 20:10:20 +00:00
Sage Moore
7c0cf3bcd0
Cap the number of API servers to 1 when using Elastic EP. ( #37466 )
...
Signed-off-by: Sage Moore <sage@neuralmagic.com >
2026-03-19 10:42:57 -04:00
Sage
0fefd00e6c
[Bugfix] Fix render server crash for quantized models on CPU-only hosts ( #37215 )
...
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com >
2026-03-16 18:59:01 +00:00
7afe0faab1
[Frontend][Core] Re-add shutdown timeout - allowing in-flight requests to finish ( #36666 )
...
Signed-off-by: Mark McLoughlin <markmc@redhat.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-03-13 12:10:06 -07:00
Nick Hill
36735fd772
[BugFix] Fix multiple/duplicate stdout prefixes ( #36822 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-03-12 12:23:21 +08:00
Mark McLoughlin
234860399b
[Frontend][Core] Revert "Add shutdown timeout" ( #34730 and #36270 ) ( #36628 )
...
Signed-off-by: Mark McLoughlin <markmc@redhat.com >
2026-03-10 06:20:41 -07:00
