Commit Graph - vLLM - iHore Software Repository

iHore/vLLM

Fork 0

ceaf4ed003 [Doc] Update the SkyPilot doc with serving and Llama-3 (#4276) Zhanghao Wu 2024-04-22 15:34:31 -07:00
ad8d696a99 [Core] Scheduler perf fix (#4270) SangBin Cho 2024-04-23 06:11:06 +09:00
3d925165f2 Add example scripts to documentation (#4225) Harry MellorandHarry Mellor 2024-04-22 17:36:54 +01:00
1543680691 [Bugfix] Ensure download_weights_from_hf(..) inside loader is using the revision parameter (#4217) alexm-nm 2024-04-22 12:10:48 -04:00
077f0a2e8a [Frontend] Enable support for CPU backend in AsyncLLMEngine. (#3993) Tao He 2024-04-22 17:19:51 +08:00
e73ed0f1c6 [Bugfix] Fix type annotations in CPU model runner (#4256) Woosuk Kwon 2024-04-22 00:54:16 -07:00
296cdf8ac7 [Misc] Add vision language model support to CPU backend (#3968) Isotr0py 2024-04-22 15:44:16 +08:00
747b1a7147 [Core][Distributed] fix _is_full_nvlink detection (#4233) youkaichao 2024-04-21 23:04:16 -07:00
95e5b087cf [AMD][Hardware][Misc][Bugfix] xformer cleanup and light navi logic and CI fixes and refactoring (#4129) Hongxia Yang 2024-04-22 00:57:24 -04:00
a37d815b83 Make initialization of tokenizer and detokenizer optional (#3748)
GeauxEric Yun Ding Roger Wang
2024-04-21 15:06:46 -07:00
7f2593b164 [Doc]: Update the doc of adding new models (#4236) xiaoji 2024-04-22 00:57:08 +08:00
fe7d648fe5 Don't show default value for flags in EngineArgs (#4223) Harry MellorandHarry Mellor 2024-04-21 17:15:28 +01:00
cc74b2b232 Updating lm-format-enforcer version and adding links to decoding libraries in docs (#4222) Noam Gat 2024-04-20 11:33:16 +03:00
91528575ec [Frontend] multiple sampling params support (#3570) nunjunj 2024-04-20 00:11:57 -07:00
a22cdea371 [Kernel][FP8] Initial support with dynamic per-tensor scaling (#4118) Cody Yu 2024-04-19 21:28:57 -07:00
682789d402 Fix missing docs and out of sync EngineArgs (#4219) Harry MellorandHarry Mellor 2024-04-20 04:51:33 +01:00
138485a82d [Bugfix] Add fix for JSON whitespace (#4189) Ayush RautwarandUbuntu 2024-04-19 23:49:22 -04:00
bc9df1571b Pass tokenizer_revision when getting tokenizer in openai serving (#4214) Chirag Jain 2024-04-20 05:43:56 +05:30
15b86408a8 [Misc] add nccl in collect env (#4211) youkaichao 2024-04-19 12:44:51 -07:00
7be4f5628f [Bugfix][Core] Restore logging of stats in the async engine (#4150) Ronen Schaffer 2024-04-19 18:08:26 +03:00
8f20fc04bf [Misc] fix docstrings (#4191) UranusandZhong Wang 2024-04-19 16:18:33 +08:00
221d93ecbf Bump version of 0.4.1 (#4177) Simon Mo 2024-04-19 01:00:22 -07:00
d17c8477f1 [Bugfix] Fix LoRA loading check (#4138) Jee Liandsimon-mo 2024-04-19 15:59:54 +08:00
a134ef6f5e Support eos_token_id from generation_config.json (#4182) Simon Mo 2024-04-18 21:13:36 -07:00
8a7a3e4436 [Core] add an option to log every function call to for debugging hang/crash in distributed inference (#4079) youkaichaoandSimon Mo 2024-04-18 16:15:12 -07:00
8f9c28fd40 [Bugfix] Fix CustomAllreduce nvlink topology detection (#3974) Adam Tilghman 2024-04-18 15:32:47 -07:00
cd2f63fb36 [CI/CD] add neuron docker and ci test scripts (#3571) Liangfu Chen 2024-04-18 15:26:01 -07:00
87fa80c91f [Misc] Bump transformers to latest version (#4176) Nick Hill 2024-04-18 14:36:39 -07:00
e1bb2fd52d [Bugfix] Support logprobs when using guided_json and other constrained decoding fields (#4149) James Whedbee 2024-04-18 16:12:55 -05:00
705578ae14 [Docs] document that Meta Llama 3 is supported (#4175) Simon Mo 2024-04-18 10:55:48 -07:00
e8cc7967ff [Bugfix][Kernel] allow non-power-of-two head sizes in prefix prefill (#4128) Michał Moskal 2024-04-18 00:51:28 -07:00
53b018edcb [Bugfix] Get available quantization methods from quantization registry (#4098) Michael Goin 2024-04-18 03:21:55 -04:00
66ded03067 Allow model to be served under multiple names (#2894) Harry MellorandAlexandre Payot 2024-04-18 08:16:26 +01:00
6dc1fc9cfe [Core] nccl integrity check and test (#4155) youkaichao 2024-04-17 22:28:52 -07:00
533d2a1f39 [Typing] Mypy typing part 2 (#4043) SangBin ChoandSangBin Cho 2024-04-18 09:28:43 +09:00
a53222544c [Kernel] Add punica dimension for Swallow-MS-7B LoRA (#4134) Shoichi Uchinami 2024-04-18 02:02:45 +09:00
fe3b5bbc23 [Bugfix] fix output parsing error for trtllm backend (#4137) ElinxandRoger Wang 2024-04-17 19:07:23 +08:00
8438e0569e [Core] RayWorkerVllm --> WorkerWrapper to reduce duplication (#4024) youkaichao 2024-04-17 01:34:33 -07:00
11d652bd4f [CI] Move CPU/AMD tests to after wait (#4123) Cade Daniel 2024-04-16 22:53:26 -07:00
d150e4f89f [Misc] [CI] Fix CI failure caught after merge (#4126) Cade Daniel 2024-04-16 17:56:01 -07:00
e95cd87959 [Speculative decoding 6/9] Integrate speculative decoding with LLMEngine (#3894) Cade Daniel 2024-04-16 13:09:21 -07:00
69e1d2fb69 [Core] Refactor model loading code (#4097) Antoni Baum 2024-04-16 11:34:39 -07:00
05434764cd LM Format Enforcer Guided Decoding Support (#3868) Noam GatandSimon Mo 2024-04-16 08:54:57 +03:00
4e7ee664e2 [Core] Fix engine-use-ray broken (#4105) SangBin Cho 2024-04-16 14:24:53 +09:00
37e84a403d [Typing] Fix Sequence type GenericAlias only available after Python 3.9. (#4092) SangBin Cho 2024-04-16 06:47:31 +09:00
4695397dcf [Bugfix] Fix ray workers profiling with nsight (#4095) Ricky Xu 2024-04-15 14:24:45 -07:00
d619ae2d19 [Doc] Add better clarity for tensorizer usage (#4090) Sanger SteelandRoger Wang 2024-04-15 16:28:25 -04:00
eb46fbfda2 [Core] Simplifications to executor classes (#4071) Nick Hill 2024-04-15 13:05:09 -07:00
0003e9154b [Misc][Minor] Fix CPU block num log in CPUExecutor. (#4088) Li, Jiang 2024-04-15 23:35:55 +08:00
e11e200736 [Bugfix] Fix filelock version requirement (#4075) Zhuohan Li 2024-04-14 21:50:08 -07:00
8db1bf32f8 [Misc] Upgrade triton to 2.2.0 (#4061) Roy 2024-04-15 08:43:54 +08:00
aceb17cf2d [Docs] document that mixtral 8x22b is supported (#4073) Simon Mo 2024-04-14 14:35:55 -07:00
563c54f760 [BugFix] Fix tensorizer extra in setup.py (#4072) Nick Hill 2024-04-14 22:12:42 +01:00
2cd6b4f362 [Core] avoid too many cuda context by caching p2p test (#4021) youkaichao 2024-04-13 23:40:21 -07:00
711a000255 [Frontend] [Core] feat: Add model loading using tensorizer (#3476) Sanger Steel 2024-04-13 20:13:01 -04:00
989ae2538d [Kernel] Add punica dimension for Baichuan-13B (#4053) Jee Li 2024-04-13 22:55:05 +08:00
0a430b4ae2 [Bugfix] fix_small_bug_in_neuron_executor (#4051) zspo 2024-04-13 22:54:03 +08:00
ec8e3c695f [Bugfix] fix_log_time_in_metrics (#4050) zspo 2024-04-13 22:52:36 +08:00
98afde19fc [Core][Distributed] improve logging for init dist (#4042) youkaichao 2024-04-13 07:12:53 -07:00
5c2e66e487 [Bugfix] More type hint fixes for py 3.8 (#4039) Dylan Hawk 2024-04-12 21:07:04 -07:00
546e721168 [CI/Test] expand ruff and yapf for all supported python version (#4037) youkaichao 2024-04-12 18:43:37 -07:00
b8aacac31a [Bugfix] Fix LoRA bug (#4032) Jee Li 2024-04-13 07:56:37 +08:00
d04973ad54 Fix triton compilation issue (#3984) Bellk17andWoosuk Kwon 2024-04-12 16:41:26 -07:00
fbb9d9eef4 [Core] fix custom allreduce default value (#4040) youkaichao 2024-04-12 16:40:39 -07:00
09473ee41c [mypy] Add mypy type annotation part 1 (#4006) SangBin Cho 2024-04-13 06:35:50 +09:00
d4ec9ffb95 [Misc] Fix typo in scheduler.py (#4022) Zhuohan Li 2024-04-12 13:56:04 -07:00
96b6a6d790 [Bugfix] fix type hint for py 3.8 (#4036) youkaichao 2024-04-12 12:35:44 -07:00
36729bac13 [Test] Test multiple attn backend for chunked prefill. (#4023) SangBin Cho 2024-04-13 01:56:57 +09:00
7fd3949a0b [Frontend][Core] Move merge_async_iterators to utils (#4026) Cyrus Leung 2024-04-12 13:30:54 +08:00
1096717ae9 [Core] Support LoRA on quantized models (#4012) Jee Li 2024-04-12 12:02:44 +08:00
c2b4a1bce9 [Doc] Add typing hints / mypy types cleanup (#3816) Michael FeilandRoger Wang 2024-04-11 17:17:21 -07:00
e46a60aa4c [BugFix] Fix handling of stop strings and stop token ids (#3672) Nick Hill 2024-04-11 23:34:12 +01:00
1e96c3341a Add extra punica sizes to support bigger vocabs (#4015) Antoni Baum 2024-04-11 15:18:57 -07:00
95e7d4a97c Fix echo/logprob OpenAI completion bug (#3441) Dylan HawkandDylan Hawk 2024-04-11 15:15:50 -07:00
559eb852f8 [Core] init_distributed_environment align with init_process_group(#4014) youkaichao 2024-04-11 14:00:48 -07:00
a10d3056da [Core] Set linear_weights directly on the layer (#3977) Antoni Baum 2024-04-11 13:35:51 -07:00
8afca50889 [Hardware][Intel] Isolate CPUModelRunner and ModelRunner for better maintenance (#3824) bigPYJ1151 2024-04-12 02:56:49 +08:00
08ccee1e83 punica fix-bgmv-kernel-640 (#4007) fuchen.ljl 2024-04-11 23:59:26 +08:00
c1dc547129 [Kernel] Fused MoE Config for Mixtral 8x22 (#4002) Roger Wang 2024-04-11 07:50:00 -07:00
f3d0bf7589 [Doc][Installation] delete python setup.py develop (#3989) youkaichao 2024-04-10 20:33:02 -07:00
e9da5a40c6 [Misc] Add indirection layer for custom ops (#3913) Kunshang Ji 2024-04-11 03:26:07 +00:00
e42df7227d [Test] Add xformer and flash attn tests (#3961) SangBin ChoandSimon Mo 2024-04-11 12:09:50 +09:00
caada5e50a [Core][Model] torch.compile for layernorm in commandr (#3985) youkaichao 2024-04-10 18:48:26 -07:00
67b4221a61 [Core][5/N] Fully working chunked prefill e2e (#3884) SangBin Cho 2024-04-11 09:56:48 +09:00
63e7176f26 [Core][Refactor] move parallel_utils into vllm/distributed (#3950) youkaichao 2024-04-10 15:33:30 -07:00
934d3662f7 [Bugfix] handle hf_config with architectures == None (#3982) Travis JohnsonandSimon Mo 2024-04-10 16:28:25 -06:00
92cd2e2f21 [Doc] Fix getting stared to use publicly available model (#3963) Frαnçois 2024-04-10 20:05:52 +02:00
e4c4072c94 [Bugfix] Remove key sorting for guided_json parameter in OpenAi compatible Server (#3945) Daniel E Marasco 2024-04-10 13:15:51 -04:00
e35397468f [Doc] Add doc to state our model support policy (#3948) youkaichaoandRoger Wang 2024-04-10 10:03:02 -07:00
8b317c6dd0 [Model][AMD] ROCm support for 256 head dims for Gemma (#3972) James Whedbee 2024-04-10 10:12:00 -05:00
bd3c144e0b [Bugfix][ROCm] Add numba to Dockerfile.rocm (#3962) Woosuk Kwon 2024-04-10 07:37:17 -07:00
0258b7a94b [Bugfix] handle prompt_logprobs in _apply_min_tokens_penalty (#3876) Travis Johnson 2024-04-10 02:39:56 -06:00
b3104b2a10 [Bugfix] Fix logits processor when prompt_logprobs is not None (#3899) 胡译文 2024-04-10 15:09:36 +08:00
c2e00af523 [Bugfix] fix utils.py/merge_dict func TypeError: 'type' object is not subscriptable (#3955) zhaotyerandtianyi_zhao 2024-04-10 12:49:11 +08:00
c013d32c75 [Benchmark] Add cpu options to bench scripts (#3915) Zedong Peng 2024-04-10 12:30:03 +08:00
11dd6ebb89 [Misc] Avoid loading incorrect LoRA config (#3777) Jee Li 2024-04-10 10:47:15 +08:00
6c0b04515f [ROCm][Hardware][AMD] Use Triton Kernel for default FA on ROCm (#3643)
Juan Villamizar jpvillam Gregory Shtrasberg Woosuk Kwon
2024-04-09 17:10:47 -05:00
e23a43aef8 [Bugfix] Fix KeyError on loading GPT-NeoX (#3925) Junichi Sato 2024-04-10 04:11:31 +09:00
e7c7067b45 [Misc] [Core] Implement RFC "Augment BaseExecutor interfaces to enable hardware-agnostic speculative decoding" (#3837) Cade Daniel 2024-04-09 11:44:15 -07:00
6d592eb430 [Core] separate distributed_init from worker (#3904) youkaichao 2024-04-09 01:49:02 -07:00

... 221 222 223 224 225 ...

Commit Graph Select branches Hide Pull Requests main patches sm86 Mono Color

Commit Graph

Select branches

Hide Pull Requests

main

patches

sm86