iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
11,384 Commits 3 Branches 0 Tags
5bdd15527770ef39cc4c3cdca008fb4f9cf8a15f
Commit Graph
411 Commits
Author SHA1 Message Date
Nick Hill 7765e5ba75 [BugFix] Fix PP performance and PP kv connector output regression (#28768) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-17 14:08:50 -08:00
RonaldNick HillBenjamin Chislett
 		d8874c61a5 [Core] Async Scheduling X Spec Decoding Compatibility (#24799) 
Signed-off-by: Ronald1995 <ronaldautomobile@163.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
 2025-11-17 12:16:20 -08:00
Lucia FangandNick Hill b316ac6589 [V1] Support MP Executor for multi node distributed inference (#23691) 
Signed-off-by: Lu Fang <fanglu@fb.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: Lucia Fang <fanglu@fb.com>
Signed-off-by: Lucia Fang <116399278+luccafong@users.noreply.github.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-11-16 09:01:21 +00:00
tingtinggithub cb15ee28db Allow Gemma3 to take image embeddings (#28483) 
Signed-off-by: tingtinggithub <streamttt@gmail.com>
 2025-11-15 04:18:08 -08:00
Zhuohan Li dd6ac1c2bb [RL] [V1] Remove unused device argument from reset_kv_cache (#28766) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
 2025-11-14 23:59:42 -08:00
Cyrus Leung e2741f6cbc [Chore] Rename SchedulerConfig.chunked_prefill_enabled (#28735) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-14 18:39:57 +00:00
Nicolò LucchesiandHarry Mellor 6f1e7f7226 [DisaggEverything] Tokens in<>out /generate endpoint (#24261) 
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-11-14 09:58:01 -07:00
Wentao Ye e64011f29a [CI] Bug: Fix ci entrypoint pooling (#28684) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2025-11-13 14:19:35 -08:00
elvischenvandNick Hill 5d6ce2b960 [Perf] Support stream interval for reducing host overhead (#27869) 
Signed-off-by: elvischenv <219235043+elvischenv@users.noreply.github.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-11-13 13:21:25 -05:00
Yannick Schnidergemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Michael GoinRoger Wang
 		119c4927b3 [Bugfix] Fix validate model input for decoder models (#27099) 
Signed-off-by: Yannick Schnider <yannick.schnider1@ibm.com>
Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2025-11-13 10:18:47 -08:00
tjandy98 4504e8029b [Bugfix] Prevent crash on empty grammar string (#28210) 
Signed-off-by: tjandy98 <3953059+tjandy98@users.noreply.github.com>
 2025-11-13 06:42:29 +00:00
Jialin Ouyang a1d3866dda [n-gen] DO NOT repeatedly return finished child requests (#28591) 
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2025-11-13 03:36:07 +00:00
Wei Wei 478ee511de [Misc]Fix typo in llm_engine.py (#28584) 
Signed-off-by: Wei Wei <wwei6@meta.com>
 2025-11-12 12:59:43 -08:00
Jialin Ouyang 4228be7959 [Perf] Use np.ndarray instead of list[list[int]] to reduce GC overhead (#28245) 
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2025-11-11 10:28:47 -08:00
Jialin Ouyang b30372cbd0 [Perf] Move gc.freeze logic from EngineCoreProc to EngineCore for better coverage (#27896) 
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2025-11-10 15:34:18 -08:00
Wei Wei bf6a3d0ff5 [Misc] Add more scoping for improved trace (#28329) 
Signed-off-by: Wei Wei <wwei6@meta.com>
 2025-11-10 21:03:21 +00:00
Mark McLoughlin 6f7de33bed [Metrics] Refactor LoRA state tracking (#26801) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-11-10 16:34:36 +08:00
Nick Hill da786e339e [Core] Rework handling of async scheduling config (#28250) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-07 20:01:23 +00:00
Jialin Ouyang ccd98b59c1 [Perf] Introduce FlattenLogprobs to store logprobs results to reduce GC overhead (#28171) 
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2025-11-07 00:27:12 -08:00
Snehlata e15601789b [Feature]: Add corrupted request metric to V1 metrics system. (#27306) 
Signed-off-by: atalhens <sneh.lata@nutanix.com>
 2025-11-05 13:45:29 -08:00
Zhuohan Li 300a265978 [Core] Enable StatLogger in LLMEngine (#28020) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
 2025-11-04 04:13:35 -08:00
wangxiyuanandHarry Mellor 30a14b034f [V0 deprecation] Remove VLLM_USE_V1 usage in platform and v1 module (#27798) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-11-01 10:17:45 +00:00
Nick Hill 0cdbe7b744 [Core] Async scheduling + structured outputs compatibility (#26866) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-01 00:35:04 +00:00
GuanLuoandNicolò Lucchesi d6517be3cd [Bugfix] Missing NIXL metadata for handshake initialization if instance spans multi-node (#26338) 
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2025-10-31 10:16:00 -07:00
Ilya MarkovandTyler Michael Smith 60f76baa66 [Misc] Replace CUDA_VISIBLE_DEVICES in DP with torch.cuda.set_device for device selection on cuda-like devices (#27564) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2025-10-30 11:41:44 -04:00
Wentao Ye c01f6e525f [CI] Fix mypy for vllm/v1/core and vllm/v1/engine (#27108) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2025-10-30 11:32:17 +00:00
Braulio Dumba 1da3309ace [Core] Exposing engine sleep & wake_up state as prometheus metrics (#24176) 
Signed-off-by: Braulio Dumba <Braulio.Dumba@ibm.com>
 2025-10-29 09:32:01 -07:00
22quinnandZhuohan Li 2abbd351ef [Core] Enable async scheduling for external_launcher mode (#27394) 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
 2025-10-28 13:52:47 +00:00
Cyrus Leung f58d9b6404 [Misc] Separate out utils.counter and move utils.Device to engine (#27588) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-28 12:20:46 +00:00
Cyrus Leung 6ebffafbb6 [Misc] Clean up more utils (#27567) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-27 15:30:38 +00:00
Yeshwanth N 71b1c8b667 [Chore]:Extract math and argparse utilities to separate modules (#27188) 
Signed-off-by: Yeshwanth Surya <yeshsurya@gmail.com>
Signed-off-by: Yeshwanth N <yeshsurya@gmail.com>
Signed-off-by: yeshsurya <yeshsurya@gmail.com>
 2025-10-26 04:03:32 -07:00
Zhuohan Li 56ed7609a9 Revert "[Misc] Remove use of CUDA_VISIBLE_DEVICES for device selectio… (#27502) 2025-10-25 05:31:43 +00:00
Wentao Ye 52efc34ebf [Log] Optimize Startup Log (#26740) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2025-10-24 19:27:04 -04:00
KebeandRui Qiao 51dd14ac2b [Bugfix][DP] Fix creating too many DP Placement Groups (#26880) 
Signed-off-by: Kebe <mail@kebe7jun.com>
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Co-authored-by: Rui Qiao <ruisearch42@gmail.com>
 2025-10-23 20:16:51 +00:00
Ilya MarkovandTyler Michael Smith 237cf6d32a [Misc] Remove use of CUDA_VISIBLE_DEVICES for device selection (fix DP slow startup time &c) (#26709) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2025-10-23 20:58:39 +08:00
wang.yuqi 3729ed00ba [Model] Add num_cached_tokens for PoolingRequestOutput (#27378) 
Signed-off-by: wang.yuqi <noooop@126.com>
 2025-10-23 14:03:42 +08:00
Giancarlo DelfinandNick Hill 6644796bf4 [V1][spec decode] return logprobs for spec decoding (#26060) 
Signed-off-by: Giancarlo Delfin <gdelfin@meta.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-10-22 22:59:59 -07:00
dongbo910220andCyrus Leung a0003b56b0 [Chore] Separate out system utilities from vllm.utils (#27201) 
Signed-off-by: dongbo910220 <1275604947@qq.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2025-10-22 20:25:25 +00:00
Nicolò Lucchesi 4dfdb821c8 [P/D] Dynamic kv_output_aggregator collect size (#26734) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-10-22 18:07:58 +02:00
Nick Hill 647214f3d5 [V0 Deprecation] Remove V0 executors (#27142) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-10-21 11:09:37 -07:00
Cyrus Leung d31f7844f8 [Misc] Move utils to avoid conflicts with stdlib, and move tests (#27169) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-19 05:20:55 -07:00
iAmir97andiAmir97 7a6c8c3fa1 [Chore] Separate out vllm.utils.network_utils (#27164) 
Signed-off-by: iAmir97 <Amir.balwel@embeddedllm.com>
Co-authored-by: iAmir97 <Amir.balwel@embeddedllm.com>
 2025-10-19 03:06:32 -07:00
dongbo910220 8a297115e2 [Chore] Separate out hashing utilities from vllm.utils (#27151) 
Signed-off-by: dongbo910220 <1275604947@qq.com>
 2025-10-19 11:09:38 +08:00
Tova Movshovitz 83e760c57d [V1][Metrics][Plugin] Add plugin support for custom StatLoggerBase implementations (#22456) 
Signed-off-by: tovam <tovam@pliops.com>
 2025-10-18 15:12:46 -07:00
Patrick von Platengemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Rui Qiao
 		b038d9c40c [Data-parallel] Allow DP>1 for world_size > num_gpus on node (8) (#26367) 
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Rui Qiao <ruisearch42@gmail.com>
 2025-10-17 08:24:42 -07:00
Nick Hill fe3b9372ad [Core] Change execute_model_with_error_logging() to be a ctx manager (#27060) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-10-17 11:45:32 +08:00
Cyrus Leung 4d4d6bad19 [Chore] Separate out vllm.utils.importlib (#27022) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-17 00:48:59 +00:00
Cyrus Leung d2740fafbf [Chore] Separate out vllm.utils.collections (#26990) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-16 08:35:35 +00:00
Cyrus Leung f6cdc9a02f [Chore] Rename utils submodules (#26920) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-16 03:58:13 +00:00
Cyrus Leung 828523ad8e [Chore] Separate out vllm.utils.async_utils (#26913) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-15 15:33:00 +00:00