iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
10,937 Commits 3 Branches 0 Tags
2c19d96777939dd3473eabfacbe1948a3ea0b4be
Commit Graph
168 Commits
Author SHA1 Message Date
Yue ZhangandNick Hill 685c99ee77 [KV offload] Offloading connector async scheduling support (#27648) 
Signed-off-by: KevinCheung2259 <2651309292@qq.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-11-01 21:08:56 +00:00
Nick Hill 0cdbe7b744 [Core] Async scheduling + structured outputs compatibility (#26866) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-01 00:35:04 +00:00
GuanLuoandNicolò Lucchesi d6517be3cd [Bugfix] Missing NIXL metadata for handshake initialization if instance spans multi-node (#26338) 
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: GuanLuo <41310872+GuanLuo@users.noreply.github.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2025-10-31 10:16:00 -07:00
Ilya MarkovandTyler Michael Smith 60f76baa66 [Misc] Replace CUDA_VISIBLE_DEVICES in DP with torch.cuda.set_device for device selection on cuda-like devices (#27564) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2025-10-30 11:41:44 -04:00
Nick Hill 2ce5c5d3d6 [BugFix] Handle unscheduled requests properly when async scheduling (#27756) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-10-29 21:04:25 -07:00
Nick Hill d4aa144343 [BugFix] Fix handling of resumed reqs in SharedStorageConnector (#27719) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-10-29 20:16:52 +00:00
Nicolò LucchesiandMark McLoughlin accb8fab07 [KVConnector] Add metrics to Prometheus-Grafana dashboard (#26811) 
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
 2025-10-29 18:44:49 +00:00
Wentao Ye 5522fb274b [Chore] Optimize P2PNCCLEngine http_address (#27488) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2025-10-30 00:05:09 +08:00
ShaotingandSamuel Shen a4a4f0f617 [KV Connector] Update lmcache connector with latest compatibility (#27681) 
Signed-off-by: Samuel Shen <slshen@uchicago.edu>
Co-authored-by: Samuel Shen <slshen@uchicago.edu>
 2025-10-29 05:38:37 +00:00
Samuel ShenandSamuel Shen 05e034f085 [nit]: Fix import for the lmcache integration (#27600) 
Signed-off-by: Samuel Shen <slshen@uchicago.edu>
Co-authored-by: Samuel Shen <slshen@uchicago.edu>
 2025-10-28 14:40:55 +00:00
Cyrus Leung 7c2bdb83dc [Misc] Clean up utils (#27552) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-27 09:05:40 +00:00
Kuntai Du b853540388 [Core][Hybrid allocator + kv connector 1/n] Enable hybrid allocator + KV cache connector (#25712) 
Signed-off-by: KuntaiDu <kuntai@uchicago.edu>
Signed-off-by: Kuntai Du <kuntai@uchicago.edu>
 2025-10-24 23:34:18 -07:00
Zhuohan Li 56ed7609a9 Revert "[Misc] Remove use of CUDA_VISIBLE_DEVICES for device selectio… (#27502) 2025-10-25 05:31:43 +00:00
Yihua Cheng 83f478bb19 [KVConnector] Migrate the LMCache integration code to be vLLM native (#25542) 
Signed-off-by: ApostaC <yihua98@uchicago.edu>
 2025-10-25 00:23:53 +00:00
kourosh hakhamaneshiandNicolò Lucchesi 7e1d697b56 [Bugfix] Fix MultiConnector stats reconstruction across process boundaries (#27366) 
Signed-off-by: Kourosh Hakhamaneshi <Kourosh@anyscale.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2025-10-24 17:08:05 +00:00
Chendi.Xue 699d62e6cf [NIXL][BUGFIX] delay done_recving queue cleanup to bottom of get_finished (#27297) 
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
 2025-10-24 17:01:41 +00:00
Ilya MarkovandTyler Michael Smith 237cf6d32a [Misc] Remove use of CUDA_VISIBLE_DEVICES for device selection (fix DP slow startup time &c) (#26709) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2025-10-23 20:58:39 +08:00
Nicolò Lucchesi 4dfdb821c8 [P/D] Dynamic kv_output_aggregator collect size (#26734) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-10-22 18:07:58 +02:00
Chendi.XueandNicolò Lucchesi 7c4767f1eb [NIXL] use Host buffer to support TP_ratio > 1 for XPU (#27140) 
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com>
 2025-10-22 15:28:13 +00:00
Mark McLoughlin 4ca13a8667 [NIXL] Terminate handshake listener thread in shutdown (#26404) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-10-22 16:59:53 +02:00
Tyler Michael Smith 6c2eef5a5d [P/D] KVConnector for decode benchmarking (#25986) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2025-10-21 16:30:47 -07:00
David Whyte-Gray ddeec11ba9 [Bugfix][P/D] Reduce num_threads used by nixl ucx backend (#27196) 
Signed-off-by: David Whyte-Gray <40244437+dagrayvid@users.noreply.github.com>
 2025-10-21 13:41:52 -04:00
Nicolò Lucchesi 72f431e709 [Nixl] Minor refactor to handshake related metadata (#26410) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-10-21 09:07:47 +02:00
iAmir97andiAmir97 7a6c8c3fa1 [Chore] Separate out vllm.utils.network_utils (#27164) 
Signed-off-by: iAmir97 <Amir.balwel@embeddedllm.com>
Co-authored-by: iAmir97 <Amir.balwel@embeddedllm.com>
 2025-10-19 03:06:32 -07:00
Isotr0py 6ac5e06f7c [Chore] Clean up pytorch helper functions in vllm.utils (#26908) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: isotr0py <2037008807@qq.com>
 2025-10-18 09:48:22 -07:00
Mark McLoughlin 4a510ab487 [NIXL] Improve request_finished() debug logs (#25665) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-10-16 15:55:17 +02:00
Chendi.Xue bfad142e25 [BUGFIX][NIXL] quick fix for 'assert self.connector_worker is not None' in get_kv_connector_stats (#26851) 
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
 2025-10-15 02:33:25 +00:00
Chendi.Xue 7e6edb1469 [NIXL][HeteroTP] Enable KV transfer from HND prefill to NHD decode (#26556) 
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
 2025-10-14 09:46:05 +00:00
Wentao YeandHarry Mellor 314285d4f2 [CI] Fix mypy for vllm/distributed (#26593) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-13 16:02:24 -04:00
Will Eaton 53c9a7cee2 [P/D] [NixlConnector] kv load recovery integration (#26171) 
Signed-off-by: Will Eaton <weaton@redhat.com>
 2025-10-13 08:48:04 -07:00
Harry Mellor 8fcaaf6a16 Update Optional[x] -> x | None and Union[x, y] to x | y (#26633) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-12 09:51:31 -07:00
Mark McLoughlin 784c231151 [NIXL] Ignore abort on already-finished request (#25067) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-10-10 12:21:56 +02:00
Matthew Bonanni 76879cc160 [Attention] Implement universal BACKEND_MAP (#25900) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2025-10-08 12:00:25 -07:00
Harry Mellor 4e256cadc2 Remove all references to yapf as it's no longer used (#26251) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-05 09:18:11 -07:00
Harry Mellor d6953beb91 Convert formatting to use ruff instead of yapf + isort (#26247) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-05 07:06:22 -07:00
Nicolò Lucchesi 2a6dc67eb5 [Bugfix] Fix _reqs_to_process leak on abort (#26012) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-10-04 11:39:31 +00:00
Nicolò Lucchesi 48f309029a [NIXL][Misc] Expose metrics from NIXL for logging to CLI (#25388) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-10-03 10:47:59 +00:00
Matthew Bonanni 2aaa423842 [Attention] Move Backend enum into registry (#25893) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2025-10-02 20:32:24 -07:00
David Ben-DavidandDavid Ben-David 9a9f48dff7 [V1] [P/D] Add Support for KV Load Failure Recovery (#19330) 
Signed-off-by: David Ben-David <davidb@pliops.com>
Co-authored-by: David Ben-David <davidb@pliops.com>
 2025-09-30 14:57:08 -07:00
Or Ozeri cfd302db9b OffloadingConnector: Fix GPU block tracking bug (#25856) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2025-09-30 19:53:04 +00:00
Nicolò LucchesiandChen Zhang 80608ba5af [NIXL] Add support for MLA caches with different latent dim (#25902) 
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
 2025-09-30 12:18:29 +00:00
Chenxi YangandChenxi Yang d0d138bc55 [Nixl][P/D] Add cuda2cpu support (HD->DH transfer) (#24690) 
Signed-off-by: Chenxi Yang <cxyang@fb.com>
Co-authored-by: Chenxi Yang <cxyang@fb.com>
 2025-09-29 14:31:51 +00:00
+2
Robert ShawSage MooreRussell Bryantrentianyue-jkIsotr0pyChenheli HuaWentao YeMichael GoinTyler Michael SmithNicolò LucchesiRoger WangRobert Shaw
 		9b44a7d926 [P/D] NIXL Updates (#25844) 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
Signed-off-by: simon-mo <simon.mo@hey.com>
Signed-off-by: rentianyue-jk <rentianyue-jk@360shuke.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Chenheli Hua <huachenheli@outlook.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Sage Moore <sage@neuralmagic.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: rentianyue-jk <rentianyue-jk@360shuke.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Chenheli Hua <huachenheli@outlook.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
 2025-09-29 04:46:30 +00:00
Nicolò Lucchesi da63274d9f [Bugfix][NIXL] Fix Async Scheduler timeout issue (#25808) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-09-27 15:17:35 -04:00
Matthew Bonanni 3468f17ebe [V0 deprecation] Remove _VLLM_V1 suffixes from attention backend names (#25489) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni001@gmail.com>
 2025-09-25 17:37:50 +00:00
Thomas Parnell 969b4da3a6 [V0 Deprecation] Remove placeholder attn (#25510) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
 2025-09-23 22:12:14 +00:00
ChaunceyandMark McLoughlin f05a4f0e34 [P/D] Support NIXL connector to disconnect during a clean shutdown (#24423) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
 2025-09-23 16:08:02 +02:00
Chendi.Xue 5774b0a1da [NIXL][OOT platform] support nixl_connector with oot platform and other nixl_backend (#25121) 
Signed-off-by: Chendi Xue <Chendi.Xue@intel.com>
 2025-09-23 04:17:42 +00:00
Csrayzandivyilike c10101a3eb [Bugfix] Fix several issues with p2p xPyD in GET type (#23993) 
Signed-off-by: Csrayz <jover@cmbchina.com>
Signed-off-by: ivyilike <pww123@cmbchina.com>
Co-authored-by: ivyilike <pww123@cmbchina.com>
 2025-09-22 14:53:13 +00:00
Woosuk Kwon bc6e542d9f Remove V0 attention backends (#25351) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-09-21 16:03:28 -07:00