Commit Graph - vLLM - iHore Software Repository

iHore/vLLM

Fork 0

dfeb2ecc3a [Misc] Include matched stop string/token in responses (#2976) Nick HillandSahil Suneja 2024-03-25 17:31:32 -07:00
3a243095e5 Optimize _get_ranks in Sampler (#3623) Antoni Baum 2024-03-25 16:03:02 -07:00
64172a976c [Feature] Add vision language model support. (#3042) xwjiang2010 2024-03-25 14:16:30 -07:00
f408d05c52 hotfix isort on logprobs ranks pr (#3622) Simon Mo 2024-03-25 11:55:46 -07:00
0b4997e05c [Bugfix] API stream returning two stops (#3450) Dylan HawkandDylan Hawk 2024-03-25 10:14:34 -07:00
c13ad1b7bd feat: implement the min_tokens sampling parameter (#3124) Travis JohnsonandNick Hill 2024-03-25 11:14:26 -06:00
819924e749 [Core] Adding token ranks along with logprobs (#3516) Swapnil ParekhandSwapnil Parekh 2024-03-25 13:13:10 -04:00
01bfb22b41 [CI] Try introducing isort. (#3495) SangBin Cho 2024-03-25 23:59:47 +09:00
e67c295b0c [Bugfix] fix automatic prefix args and add log info (#3608) TianYu GUO 2024-03-25 20:35:22 +08:00
925f3332ca [Core] Refactor Attention Take 2 (#3462) Woosuk Kwon 2024-03-24 21:39:33 -07:00
b0dfa91dd7 [Model] Add starcoder2 awq support (#3569) 少年 2024-03-25 12:07:36 +08:00
56a8652f33 [Bugfix] store lock file in tmp directory (#3578)" (#3599) Woosuk Kwonandyoukaichao 2024-03-24 20:06:50 -07:00
6d93d35308 [BugFix] tensor.get_device() -> tensor.device (#3604) Kunshang Ji 2024-03-25 10:01:13 +08:00
837e185142 [CI/Build] fix flaky test (#3602) youkaichao 2024-03-24 17:43:05 -07:00
42bc386129 [CI/Build] respect the common environment variable MAX_JOBS (#3600) youkaichao 2024-03-24 17:04:00 -07:00
8b268a46a7 [CI] typo fix: is_hip --> is_hip() (#3595) youkaichao 2024-03-24 16:03:06 -07:00
41deac4a3d [BugFix] 1D query fix for MoE models (#3597) Nick Hill 2024-03-24 16:00:16 -07:00
af9e53496f [BugFix] Fix Falcon tied embeddings (#3590) Woosuk Kwonand44670 2024-03-24 06:34:01 -07:00
f8a12ecc7f [Misc] Bump transformers version (#3592) Roger Wang 2024-03-24 06:32:45 -07:00
3c5ab9b811 [Misc] Fix BLOOM copyright notice (#3591) Woosuk Kwon 2024-03-23 23:30:56 -07:00
743a0b7402 [Bugfix] use SoftLockFile instead of LockFile (#3578) kota-iizuka 2024-03-24 03:43:11 +09:00
bfdb1ba5c3 [Core] Improve detokenization performance for prefill (#3469) Antoni BaumandMeloYang 2024-03-22 13:44:12 -07:00
cf2f084d56 Dynamic scheduler delay to improve ITL performance (#3279) Thomas ParnellandJan van Lunteren 2024-03-22 20:28:14 +01:00
f721096d48 [BugFix] Some fixes for custom allreduce kernels (#2760) Hanzhi Zhou 2024-03-21 23:02:58 -07:00
e90fc21f2e [Hardware][Neuron] Refactor neuron support (#3471) Zhuohan Li 2024-03-21 18:22:17 -07:00
ea5f14e6ff [Bugfix][Model] Fix Qwen2 (#3554) Roy 2024-03-22 08:18:58 +08:00
b7050ca7df [BugFix] gemma loading after quantization or LoRA. (#3553) Taemin Lee 2024-03-22 05:16:57 +09:00
c188ecb080 [Misc] Bump up transformers to v4.39.0 & Remove StarCoder2Config (#3551)
Woosuk Kwon Roy Roger Meier
2024-03-21 07:58:12 -07:00
865732342b [Misc][Log] Add log for tokenizer length not equal to vocabulary size (#3500) Roy 2024-03-21 18:07:48 +08:00
4c07dd28c0 [🚀 Ready to be merged] Added support for Jais models (#3183) Lalit Pradhan 2024-03-21 13:45:24 +04:00
3bbff9e5ab Fix 1D query issue from _prune_hidden_states (#3539) SangBin Cho 2024-03-21 17:49:06 +09:00
6ebd02bdef [PREFIX CACHING FOLLOW UP] OrderedDict-based evictor (#3431)
ElizaWszola rsnm2 Luka
2024-03-21 07:20:04 +01:00
523e30ea0c [BugFix] Hot fix in setup.py for neuron build (#3537) Zhuohan Li 2024-03-20 17:59:52 -07:00
f1c0fc3919 Migrate logits computation and gather to model_runner (#3233) Roy 2024-03-21 07:25:01 +08:00
6e435de766 [1/n][Chunked Prefill] Refactor input query shapes (#3236) SangBin Cho 2024-03-21 06:46:05 +09:00
426ec4ec67 [1/n] Triton sampling kernel (#3186) Antoni BaumandRoger Wang 2024-03-20 14:45:08 -07:00
80e254834d [Bugfix] Fix ROCm support in CMakeLists.txt (#3534) James Whedbee 2024-03-20 16:05:03 -05:00
ba8ae1d84f Check for _is_cuda() in compute_num_jobs (#3481) bnellnm 2024-03-20 13:06:56 -04:00
84eaa68425 Abort when nvcc command is not found in the PATH (#3527) Allen.Dou 2024-03-21 00:28:29 +08:00
5ee14494e4 [Misc] Remove cache stream and cache events (#3461) Woosuk Kwon 2024-03-20 00:38:53 -07:00
4ad521d8b5 [Core] Add generic typing to LRUCache (#3511) Nick Hill 2024-03-20 00:36:09 -07:00
9474e89ba4 [PREFIX CACHING FOLLOW UP] A bunch of fixes to block allocator performance when automatic prefix caching is disabled (#3357) ElizaWszolaandZhuohan Li 2024-03-20 08:11:11 +01:00
20478c4d3a Use lru_cache for some environment detection utils (#3508) Simon Mo 2024-03-19 14:34:15 -07:00
63e8b28a99 [Doc] minor fix of spelling in amd-installation.rst (#3506) Jim Burtoft 2024-03-19 16:32:30 -04:00
cc63d03fbb Revert "[Core] Cache some utils" (#3507) Simon Mo 2024-03-19 13:22:58 -07:00
2a60c9bd17 [Doc] minor fix to neuron-installation.rst (#3505) Jim Burtoft 2024-03-19 16:21:35 -04:00
c614cfee58 Update dockerfile with ModelScope support (#3429) ifsheldon 2024-03-20 01:54:59 +08:00
7341c77d69 [BugFix] Avoid initializing CUDA too early (#3487) Nick Hill 2024-03-18 23:05:20 -07:00
ef65dcfa6f [Doc] Add docs about OpenAI compatible server (#3288) Simon Mo 2024-03-18 22:05:34 -07:00
6a9c583e73 [Core] print error before deadlock (#3459) youkaichao 2024-03-18 21:06:23 -07:00
b37cdce2b1 [Core] Cache some utils (#3474) Antoni Baum 2024-03-18 17:14:26 -07:00
b30880a762 [Misc] Update README for the Third vLLM Meetup (#3479) Zhuohan Li 2024-03-18 15:58:38 -07:00
49eedea373 [Core] Zero-copy asdict for InputMetadata (#3475) Antoni Baum 2024-03-18 15:56:40 -07:00
9fdf3de346 Cmake based build system (#2830) bnellnm 2024-03-18 18:38:33 -04:00
c0c17d4896 [Misc] Fix PR Template (#3478) Zhuohan Li 2024-03-18 15:00:31 -07:00
097aa0ea22 [CI/Build] Fix Bad Import In Test (#3473) Robert Shaw 2024-03-18 15:28:00 -05:00
482b0adf1b [Testing] Add test_config.py to CI (#3437) Cade Daniel 2024-03-18 12:48:45 -07:00
8c654c045f CI: Add ROCm Docker Build (#2886) Simon Mo 2024-03-18 12:33:47 -07:00
9101d832e6 [Bugfix] Make moe_align_block_size AMD-compatible (#3470) Woosuk Kwon 2024-03-18 11:26:24 -07:00
93348d9458 [CI] Shard tests for LoRA and Kernels to speed up (#3445) Simon Mo 2024-03-17 14:56:30 -07:00
abfc4f3387 [Misc] Use dataclass for InputMetadata (#3452) Woosuk Kwonandyoukaichao 2024-03-17 03:02:46 -07:00
6b78837b29 Fix setup.py neuron-ls issue (#2671) Simon Mo 2024-03-16 16:00:25 -07:00
120157fd2a Support arbitrary json_object in OpenAI and Context Free Grammar (#3211) Simon Mo 2024-03-16 13:35:27 -07:00
8e67598aa6 [Misc] fix line length for entire codebase (#3444) Simon Mo 2024-03-16 00:36:29 -07:00
ad50bf4b25 fix lint simon-mo 2024-03-15 22:23:38 -07:00
cf6ff18246 Fix Baichuan chat template (#3340) Dinghow Yang 2024-03-16 12:02:12 +08:00
14e3f9a1b2 Replace lstrip() with removeprefix() to fix Ruff linter warning (#2958) Ronen Schaffer 2024-03-16 06:01:30 +02:00
3123f15138 Fixes the incorrect argument in the prefix-prefill test cases (#3246) Tao He 2024-03-16 11:58:10 +08:00
413366e9a2 [Misc] PR templates (#3413) youkaichaoandZhuohan Li 2024-03-15 18:25:51 -07:00
10585e035e Removed Extraneous Print Message From OAI Server (#3440) Robert Shaw 2024-03-15 19:35:36 -05:00
fb96c1e98c Asynchronous tokenization (#2879) Antoni Baum 2024-03-15 16:37:01 -07:00
8fa7357f2d fix document error for value and v_vec illustration (#3421) laneeee 2024-03-16 07:06:09 +08:00
a7af4538ca Fix issue templates (#3436) Harry Mellor 2024-03-15 21:26:00 +00:00
604f235937 [Misc] add error message in non linux platform (#3438) youkaichao 2024-03-15 14:21:37 -07:00
14b8ae02e7 Fixes the misuse/mixuse of time.time()/time.monotonic() (#3220) Tao Heandsimon-mo 2024-03-16 02:25:43 +08:00
03d37f2441 [Fix] Add args for mTLS support (#3430) Dan Clarkanddeclark1 2024-03-15 09:56:13 -07:00
a7c871680e Fix tie_word_embeddings for Qwen2. (#3344) Yang Fan 2024-03-16 00:36:53 +08:00
429284dc37 Fix dist.broadcast stall without group argument (#3408) Junda Chen 2024-03-14 23:25:05 -07:00
253a98078a Add chat templates for ChatGLM (#3418) Dinghow Yang 2024-03-15 14:19:22 +08:00
21539e6856 Add chat templates for Falcon (#3420) Dinghow Yang 2024-03-15 14:19:02 +08:00
b522c4476f [Misc] add HOST_IP env var (#3419) youkaichaoandSimon Mo 2024-03-14 21:32:52 -07:00
78b6c4845a Dynamically configure shared memory size for moe_align_block_size_kernel (#3376) akhoroshev 2024-03-15 04:18:07 +03:00
b983ba35bd fix marlin config repr (#3414) Enrique Shockwave 2024-03-14 23:26:19 +00:00
54be8a0be2 Fix assertion failure in Qwen 1.5 with prefix caching enabled (#3373) 陈序andCade Daniel 2024-03-15 04:56:57 +08:00
dfc77408bd [issue templates] add some issue templates (#3412) youkaichao 2024-03-14 13:16:00 -07:00
c17ca8ef18 Add args for mTLS support (#3410) Dan ClarkandDaniel Clark 2024-03-14 13:11:45 -07:00
06ec486794 Install flash_attn in Docker image (#3396) Thomas Parnell 2024-03-14 18:55:54 +01:00
8fe8386591 [Kernel] change benchmark script so that result can be directly used; tune moe kernel in A100/H100 with tp=2,4,8 (#3389) youkaichao 2024-03-14 01:11:48 -07:00
a37415c31b allow user to chose which vllm's merics to display in grafana (#3393) Allen.Dou 2024-03-14 14:35:13 +08:00
81653d9688 [Hotfix] [Debug] test_openai_server.py::test_guided_regex_completion (#3383) Simon Mo 2024-03-13 17:02:21 -07:00
eeab52a4ff [FIX] Simpler fix for async engine running on ray (#3371) Zhuohan Li 2024-03-13 14:18:40 -07:00
c33afd89f5 Fix lint (#3388) Antoni Baum 2024-03-13 13:56:49 -07:00
7e9bd08f60 Add batched RoPE kernel (#3095) Terry 2024-03-13 13:45:26 -07:00
ae0ccb4017 Add missing kernel for CodeLlama-34B on A/H100 (no tensor parallelism) when using Multi-LoRA. (#3350) Or Sharir 2024-03-13 21:18:25 +02:00
739c350c19 [Minor Fix] Use cupy-cuda11x in CUDA 11.8 build (#3256) 陈序 2024-03-14 00:43:24 +08:00
ba8dc958a3 [Minor] Fix bias in if to remove ambiguity (#3259) Hui Liu 2024-03-13 09:16:55 -07:00
e221910e77 add hf_transfer to requirements.txt (#3031) Ronan McGovern 2024-03-13 06:33:43 +00:00
b167109ba1 [Fix] Fix quantization="gptq" when using Marlin (#3319) Bo-Wen WangandWoosuk Kwon 2024-03-13 13:51:42 +08:00
602358f8a8 Add kernel for GeGLU with approximate GELU (#3337) Woosuk Kwon 2024-03-12 22:06:17 -07:00
49a3c8662b Fixes #1556 double free (#3347) Breno Faria 2024-03-13 01:30:08 +01:00

... 223 224 225 226 227 ...

Commit Graph Select branches Hide Pull Requests main patches sm86 Mono Color

Commit Graph

Select branches

Hide Pull Requests

main

patches

sm86