iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
14,386 Commits 3 Branches 0 Tags
fa6a6be51978bd4b49ba0da17039e60f96dc5b13
Commit Graph
597 Commits
Author SHA1 Message Date
Nicolò Lucchesi 758df5afe7 [NIXL][Metrics] Track nixl_num_kv_expired_reqs metric in Prometheus (#32340) 
Add a new metric to track the number of requests that had their KV blocks
expire. The scenario is particularly important to surface and track as it is a
vital indicator of the health of the deployment.

Currently we're resorting to track these failures through unstructured log
parsing (which is, among other thing, error string dependent); current main:

> Releasing expired KV blocks for request cmpl-071d which were retrieved by 0 decode worker(s) within 0 seconds.

Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-01-19 12:28:27 +00:00
Deming 5480c6b1fa [Doc] Correct comment for _jobs dict in OffloadingConnectorWorker (#32556) 2026-01-18 12:46:00 -08:00
bnellnm 327a02d8db [MoE Refactor] Separate Router into OO Classes (#30623) 
Signed-off-by: Bill Nell <bnell@redhat.com>
 2026-01-18 11:40:49 -05:00
Ilya MarkovandTyler Michael Smith c9a533079c [EPLB][BugFix]Possible deadlock fix (#32418) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-01-16 09:11:01 -05:00
kzwrime edadca109c [Bugfix] Add CpuCommunicator.dispatch and combine to fix DP+MoE inference (#31867) 
Signed-off-by: kunzh <zhikun.wu@outlook.com>
 2026-01-15 04:50:48 +00:00
Angela Yi 7933638051 [misc] Remove is_torch_equal_or_newer(2.4) cases (#32296) 
Signed-off-by: angelayi <yiangela7@gmail.com>
 2026-01-13 23:22:07 -08:00
Sage Moore 6beef12b9b [EPLB][Cleanup] Remove is_async_enabled from EplbModelState (#32050) 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
 2026-01-13 18:19:03 +00:00
Matthew BonanniandLuka Govedič 2263d44b68 [4/N][Attention] Move MLA common to model_executor (#32060) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
 2026-01-13 09:08:45 -08:00
Mathis Felardos 4f3676e726 nixl_connector: export UCX_MEM_MMAP_HOOK_MODE=none to avoid a UCX memory leak (#32181) 
Signed-off-by: Mathis Felardos <mathis@mistral.ai>
 2026-01-13 16:21:10 +00:00
Martin Hickeyandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 510265472c [BugFix] [KVConnector] Fix KV events for LMCache connector (#32169) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-01-13 15:50:34 +00:00
Nicolò Lucchesi f8bd8394e3 [NIXL][Bugfix] Failure logging overhaul + early metadata free on failure (#32031) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-01-12 20:38:49 +00:00
Lucas Kabela ad8818bb5e [Misc][BE] Type coverage for vllm/compilation [3/3] (#31748) 
Signed-off-by: Lucas Kabela <lucaskabela@meta.com>
 2026-01-12 19:24:38 +00:00
Ilya Markov 1eb61ab34b [Refactor] EPLB rebalance algo to NumPy (#30697) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
 2026-01-12 18:13:23 +00:00
Nicolò Lucchesi 5b68107411 [Misc][PD] Fix get_attn_backend usage in transfer connectors (#31988) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-01-12 18:10:05 +01:00
Or Ozeri 9cddbdba6d OffloadingConnector: Add cpu_bytes_to_use configuration (#24498) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-01-12 15:00:43 +00:00
dtcandNicolò Lucchesi 0565f1fdec [P/D] Refactor mooncake connector sender thread using async coroutines (#31573) 
Signed-off-by: Tianchen Ding <dtcccc@linux.alibaba.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com>
 2026-01-12 12:35:35 +00:00
Or Ozeri 4c16ba617f [KVConnector] OffloadingConnector: Fix bug in handling of preemptions (#29870) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-01-11 08:05:36 +00:00
Or Ozeri 2a4dbe24ea [BugFix] Wait for compute before offloading KV to CPU (#31341) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-01-10 22:25:08 +00:00
Lucas Kabela aaf4b70aae [Misc][BE] Type coverage for vllm/compilation [2/3] (#31744) 2026-01-09 18:30:38 -05:00
Chendi.Xue 94578127a4 [NIXL] refine decoder side post process for heterogeneous BlockSize and kv_layout (#30275) 2026-01-09 21:22:19 +00:00
Matthew Bonanni 2612ba9285 [1/N][Attention] Restructure attention: move files (#31916) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-01-09 13:10:24 -08:00
inkcherry 4505849b30 [ROCm][PD] add moriio kv connector. (#29304) 
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
 2026-01-09 14:01:57 +00:00
Bofeng XueandBofeng BF1 Xue 55212c1404 fix: remove duplicate engine_id check in nixl_connector (#31948) 
Signed-off-by: Bofeng BF1 Xue <xuebf1@Lenovo.com>
Co-authored-by: Bofeng BF1 Xue <xuebf1@Lenovo.com>
 2026-01-09 12:13:17 +00:00
Lucas WilkinsonandLuka Govedič 6cdf015c3c [Misc] Fix Current vLLM config is not set. warnings, assert to avoid issues in the future (#31747) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
 2026-01-08 15:20:49 -08:00
Ilya MarkovandTyler Michael Smith 6170d47d22 [EPLB] Optimize EPLB with numpy (#29499) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2026-01-07 15:21:35 -05:00
Kfir Toledo b89443b8d9 [KVConnector]: Enable Cross-layers KV cache layout for MultiConnector (#30761) 
Signed-off-by: Kfir Toledo <kfir.toledo@ibm.com>
 2026-01-07 16:59:43 +00:00
BlankRandWentao Ye 0790f07695 [Misc] Improve error messages for unsupported types and parameters (#30593) 
Signed-off-by: BlankR <hjyblanche@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-01-07 09:00:16 +00:00
weiyu e7596371a4 [Refactor][TPU] Remove torch_xla path and use tpu-inference (#30808) 
Signed-off-by: Wei-Yu Lin <weiyulin@google.com>
Signed-off-by: weiyu <62784299+weiyu0824@users.noreply.github.com>
 2026-01-07 16:07:16 +08:00
tjp_zjuandCyrus Leung 55caa6051d refactor: find_loaded_library (#31866) 
Signed-off-by: tjp_zju <tanjianpingzju1990@gmail.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2026-01-07 06:42:20 +00:00
Yihua Cheng 5b833be49e [1/2][lmcache connector] clean up lmcache multi-process adapter (#31838) 
Signed-off-by: ApostaC <yihua98@uchicago.edu>
 2026-01-07 02:02:42 +00:00
kzwrime 21156ff199 [Bugfix] Add missing extra_tensors arg to DeviceCommunicatorBase.disp… (#31644) 
Signed-off-by: kunzh <zhikun.wu@outlook.com>
 2026-01-06 01:26:09 +08:00
Yihua Cheng b471aad41f [KVconnector][LMCache] remove the import of legacy LMCache code (#31704) 
Signed-off-by: ApostaC <yihua98@uchicago.edu>
 2026-01-05 10:11:01 +00:00
Tyler Michael Smith fe8a9fbd2e [Bugfix] Fix EPLB state logging error (#31455) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-01-05 04:06:28 +00:00
Nick Hill bd877162eb [BugFix] Support online dense model DP without overhead (#30739) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: njhill <nickhill123@gmail.com>
 2026-01-02 23:36:38 +08:00
SageandSage 39512aba72 [Prefix Cache] Include lora_name in BlockStored event for deterministic KV-cache reconstruction (#27577) 
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
Co-authored-by: Sage <80211083+sagiahrac@users.noreply.github.com>
 2025-12-30 00:17:16 +00:00
qli88 0f35429a0c [CI]Test Group 'NixlConnector PD accuracy tests' is fixed (#31460) 
Signed-off-by: qli88 <qiang.li2@amd.com>
 2025-12-29 23:48:56 +00:00
chunxiaozheng b12cb38398 implements register kv caches in lmcache connector (#31397) 
Signed-off-by: idellzheng <idellzheng@tencent.com>
 2025-12-29 11:13:42 -08:00
Cyrus Leung aa3868ecfe [Chore] Remove unused noqas (#31263) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-24 05:38:46 -08:00
Chao Lei 8b59753cdb [P/D] Mooncake connector support more protocols (#30133) 
Signed-off-by: LCAIZJ <leichao139636@163.com>
 2025-12-24 10:24:07 +08:00
Chendi.XueandNicolò Lucchesi 6ca74bc11a [NIXL][BUG FIX] Fix both failing issue and accuracy issue with nixl + host_buffer on CUDA (#30419) 
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
Signed-off-by: Chendi.Xue <chendi.xue@intel.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2025-12-18 22:10:02 +00:00
wz1qqxandwz1qqx 889f8bb250 [BugFix]Reclaim resources to prevent memory leaks when use LMCacheMPConnector (#30745) 
Signed-off-by: wz1qqx <ziqi.wang@novita.ai>
Co-authored-by: wz1qqx <ziqi.wang@novita.ai>
 2025-12-18 19:09:51 +00:00
wangxiyuan a85724bd6e [Platform] Let EPD work with non-cuda platform (#30225) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2025-12-18 06:45:29 +00:00
Yihua Cheng ec965569d9 [KV connector][LMCache] Only record the cuda event when there are request to store/load (#30814) 
Signed-off-by: ApostaC <yihua98@uchicago.edu>
 2025-12-18 05:31:34 +00:00
Nicolò Lucchesi bc3700e0cd [NIXL] Support P tensor-parallel-size > D tensor-parallel-size (#27274) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-12-18 11:53:30 +08:00
Roger WangandSun Kim f5f51e5931 [Core][MM] Optimize encoder cache manager by operating with embeddings only (#30475) 
Signed-off-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Sun Kim <sunytokki@gmail.com>
 2025-12-16 14:18:17 -08:00
jiahanc 254a7f8fd6 [Perf] Do FP4 quant before All gather on flashinfer trtllmgen MOE (#30014) 
Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
 2025-12-16 13:01:48 -08:00
Chendi.XueandMark McLoughlin ae2e503dda [NIXL][BUG FIX] Fix a bug for PD with host_buffer after merging 29665 (#30420) 
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
 2025-12-14 15:38:28 +00:00
Micah WilliamsonWeiQing ChenCyrus Leung
 		97f2f160fd [ROCm][CI] Add "Qwen3-Next-80B-A3B-Instruct MTP Async EPLB Accuracy Test" Back Into AMD CI (#30590) 
Signed-off-by: David Chen <530634352@qq.com>
Signed-off-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com>
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
Co-authored-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2025-12-14 06:56:26 +00:00
Qier LiandQier Li dc7fb5bebe [Bug][KVConnector][Metrics] Remove a vacuous assertion breaking external-launcher (#30577) 
Co-authored-by: Qier Li <qier@fb.com>
 2025-12-14 01:23:08 +00:00
Nick Hill 947dfda9c2 [LMCache] Relax lmcache version requirement (#30425) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-12-11 18:18:47 -09:00