|
728c4c8a06
|
[Hardware][Intel GPU] Add Intel GPU(XPU) inference backend (#3814)
Co-authored-by: Jiang Li <jiang1.li@intel.com>
Co-authored-by: Abhilash Majumder <abhilash.majumder@intel.com>
Co-authored-by: Abhilash Majumder <30946547+abhilash1910@users.noreply.github.com>
|
2024-06-17 11:01:25 -07:00 |
|
|
Cyrus Leung
|
0e9164b40a
|
[mypy] Enable type checking for test directory (#5017)
|
2024-06-15 04:45:31 +00:00 |
|
|
Antoni Baum
|
50eed24d25
|
Add cuda_device_count_stateless (#5473)
|
2024-06-13 16:06:49 -07:00 |
|
|
Woosuk Kwon
|
1a8bfd92d5
|
[Hardware] Initial TPU integration (#5292)
|
2024-06-12 11:53:03 -07:00 |
|
|
bnellnm
|
5467ac3196
|
[Kernel][Misc] Use TORCH_LIBRARY instead of PYBIND11_MODULE for custom ops (#5047)
|
2024-06-09 16:23:30 -04:00 |
|
|
youkaichao
|
388596c914
|
[Misc][Utils] allow get_open_port to be called for multiple times (#5333)
|
2024-06-06 22:15:11 -07:00 |
|
|
functionxu123andxuhao
|
a377f0bd5e
|
[Misc]: optimize eager mode host time (#4196)
Co-authored-by: xuhao <xuhao@cambricon.com>
|
2024-05-31 13:14:50 +08:00 |
|
|
Cyrus Leung
|
eecd864388
|
[Bugfix][CI/Build] Fix test and improve code for merge_async_iterators (#5096)
|
2024-05-29 16:02:25 -07:00 |
|
|
youkaichao
|
5bd3c65072
|
[Core][Optimization] remove vllm-nccl (#5091)
|
2024-05-29 05:13:52 +00:00 |
|
|
Cyrus LeungandRoger Wang
|
5ae5ed1e60
|
[Core] Consolidate prompt arguments to LLM engines (#4328)
Co-authored-by: Roger Wang <ywang@roblox.com>
|
2024-05-28 13:29:31 -07:00 |
|
|
Cody Yu
|
a3a73ab069
|
[Misc] Load FP8 kv-cache scaling factors from checkpoints (#4893)
The 2nd PR for #4532.
This PR supports loading FP8 kv-cache scaling factors from a FP8 checkpoint (with .kv_scale parameter).
|
2024-05-22 13:28:20 -07:00 |
|
|
Wenwei Zhang
|
546a97ef69
|
[Misc]: allow user to specify port in distributed setting (#4914)
|
2024-05-20 17:45:06 +00:00 |
|
|
Cody Yu
|
c833101740
|
[Kernel] Refactor FP8 kv-cache with NVIDIA float8_e4m3 support (#4535)
|
2024-05-09 18:04:17 -06:00 |
|
|
zhaoyang-star
|
0650e5935b
|
Disable cuda version check in vllm-openai image (#4530)
|
2024-05-05 16:58:55 -07:00 |
|
|
Lily LiuandLiuXiaoxuanPKU
|
43c413ec57
|
[Kernel] Use flashinfer for decoding (#4353)
Co-authored-by: LiuXiaoxuanPKU <llilyliupku@gmail.com>
|
2024-05-03 15:51:27 -07:00 |
|
|
youkaichao
|
5b8a7c1cb0
|
[Misc] centralize all usage of environment variables (#4548)
|
2024-05-02 11:13:25 -07:00 |
|
|
Ruoyu Qin
|
dfea173148
|
[Bugfix] Abort requests when the connection to /v1/completions is interrupted (#4363)
|
2024-04-27 09:48:37 -07:00 |
|
|
SangBin ChoandDanny Guinther
|
a88081bf76
|
[CI] Disable non-lazy string operation on logging (#4326)
Co-authored-by: Danny Guinther <dguinther@neuralmagic.com>
|
2024-04-26 00:16:58 -07:00 |
|
|
Nick Hill
|
efffb63f58
|
[Core] Move function tracing setup to util function (#4352)
|
2024-04-25 16:45:12 -07:00 |
|
|
youkaichao
|
c1b4e4157c
|
[Core][Distributed] use absolute path for library file (#4271)
|
2024-04-22 17:21:48 -07:00 |
|
|
youkaichaoandSimon Mo
|
8a7a3e4436
|
[Core] add an option to log every function call to for debugging hang/crash in distributed inference (#4079)
Co-authored-by: Simon Mo <simon.mo@hey.com>
|
2024-04-18 16:15:12 -07:00 |
|
|
youkaichao
|
6dc1fc9cfe
|
[Core] nccl integrity check and test (#4155)
[Core] Add integrity check during initialization; add test for it (#4155)
|
2024-04-17 22:28:52 -07:00 |
|
|
youkaichao
|
8438e0569e
|
[Core] RayWorkerVllm --> WorkerWrapper to reduce duplication (#4024)
[Core] replace narrow-usage RayWorkerVllm to general WorkerWrapper to reduce code duplication (#4024)
|
2024-04-17 08:34:33 +00:00 |
|
|
SangBin Cho
|
37e84a403d
|
[Typing] Fix Sequence type GenericAlias only available after Python 3.9. (#4092)
|
2024-04-15 14:47:31 -07:00 |
|
|
SangBin Cho
|
09473ee41c
|
[mypy] Add mypy type annotation part 1 (#4006)
|
2024-04-12 14:35:50 -07:00 |
|
|
Cyrus Leung
|
7fd3949a0b
|
[Frontend][Core] Move merge_async_iterators to utils (#4026)
|
2024-04-12 05:30:54 +00:00 |
|
|
Michael FeilandRoger Wang
|
c2b4a1bce9
|
[Doc] Add typing hints / mypy types cleanup (#3816)
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
|
2024-04-11 17:17:21 -07:00 |
|
|
bigPYJ1151
|
8afca50889
|
[Hardware][Intel] Isolate CPUModelRunner and ModelRunner for better maintenance (#3824)
|
2024-04-11 11:56:49 -07:00 |
|
|
Kunshang Ji
|
e9da5a40c6
|
[Misc] Add indirection layer for custom ops (#3913)
|
2024-04-10 20:26:07 -07:00 |
|
|
zhaotyerandtianyi_zhao
|
c2e00af523
|
[Bugfix] fix utils.py/merge_dict func TypeError: 'type' object is not subscriptable (#3955)
Co-authored-by: tianyi_zhao <tianyi.zhao@transwarp.io>
|
2024-04-10 04:49:11 +00:00 |
|
|
+2
|
2ff767b513
|
Enable scaled FP8 (e4m3fn) KV cache on ROCm (AMD GPU) (#3290)
Co-authored-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Co-authored-by: HaiShaw <hixiao@gmail.com>
Co-authored-by: AdrianAbeyta <Adrian.Abeyta@amd.com>
Co-authored-by: Matthew Wong <Matthew.Wong2@amd.com>
Co-authored-by: root <root@gt-pla-u18-08.pla.dcgpu>
Co-authored-by: mawong-amd <156021403+mawong-amd@users.noreply.github.com>
Co-authored-by: ttbachyinsda <ttbachyinsda@outlook.com>
Co-authored-by: guofangze <guofangze@kuaishou.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: jacobthebanana <50071502+jacobthebanana@users.noreply.github.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
|
2024-04-03 14:15:55 -07:00 |
|
|
SangBin Cho
|
3dcb3e8b98
|
[3/N] Refactor scheduler for chunked prefill scheduling (#3550)
|
2024-04-03 14:13:49 -07:00 |
|
|
Nick Hill
|
c9b506dad4
|
[BugFix] Use different mechanism to get vllm version in is_cpu() (#3804)
|
2024-04-02 23:06:25 -07:00 |
|
|
|
0e3f06fe9c
|
[Hardware][Intel] Add CPU inference backend (#3634)
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: Yuan Zhou <yuan.zhou@intel.com>
|
2024-04-01 22:07:30 -07:00 |
|
|
Cade Daniel
|
14ccd94c89
|
[Core][Bugfix]Refactor block manager for better testability (#3492)
|
2024-03-27 23:59:28 -07:00 |
|
|
Nick Hill
|
0dc72273b8
|
[BugFix] Fix ipv4 address parsing regression (#3645)
|
2024-03-26 14:39:44 -07:00 |
|
|
liiliiliil
|
a979d9771e
|
[Bugfix] Fix ipv6 address parsing bug (#3641)
|
2024-03-26 11:58:20 -07:00 |
|
|
xwjiang2010
|
64172a976c
|
[Feature] Add vision language model support. (#3042)
|
2024-03-25 14:16:30 -07:00 |
|
|
SangBin Cho
|
01bfb22b41
|
[CI] Try introducing isort. (#3495)
|
2024-03-25 07:59:47 -07:00 |
|
|
Zhuohan Li
|
e90fc21f2e
|
[Hardware][Neuron] Refactor neuron support (#3471)
|
2024-03-22 01:22:17 +00:00 |
|
|
Nick Hill
|
4ad521d8b5
|
[Core] Add generic typing to LRUCache (#3511)
|
2024-03-20 00:36:09 -07:00 |
|
|
Simon Mo
|
20478c4d3a
|
Use lru_cache for some environment detection utils (#3508)
|
2024-03-19 21:34:15 +00:00 |
|
|
Simon Mo
|
cc63d03fbb
|
Revert "[Core] Cache some utils" (#3507)
|
2024-03-19 13:22:58 -07:00 |
|
|
Antoni Baum
|
b37cdce2b1
|
[Core] Cache some utils (#3474)
|
2024-03-18 17:14:26 -07:00 |
|
|
youkaichaoandSimon Mo
|
b522c4476f
|
[Misc] add HOST_IP env var (#3419)
Co-authored-by: Simon Mo <simon.mo@hey.com>
|
2024-03-14 21:32:52 -07:00 |
|
|
Zhuohan Li
|
2f8844ba08
|
Re-enable the 80 char line width limit (#3305)
|
2024-03-10 19:49:14 -07:00 |
|
|
Michael Goin
|
385da2dae2
|
Measure model memory usage (#3120)
|
2024-03-07 11:42:42 -08:00 |
|
|
ttbachyinsdaandguofangze
|
76e8a70476
|
[Minor fix] The domain dns.google may cause a socket.gaierror exception (#3176)
Co-authored-by: guofangze <guofangze@kuaishou.com>
|
2024-03-04 19:17:12 +00:00 |
|
|
Liangfu Chen
|
3b7178cfa4
|
[Neuron] Support inference with transformers-neuronx (#2569)
|
2024-02-28 09:34:34 -08:00 |
|
|
Jingru
|
2410e320b3
|
fix get_ip error in pure ipv6 environment (#2931)
|
2024-02-26 19:22:16 -08:00 |
|