iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
14,386 Commits 3 Branches 0 Tags
fa6a6be51978bd4b49ba0da17039e60f96dc5b13
Commit Graph
47 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
SangBin Choandyoukaichao c01a6cb231 [Ray backend] Better error when pg topology is bad. (#7584) 
Co-authored-by: youkaichao <youkaichao@126.com>
 2024-08-22 17:44:25 -07:00
SangBin Cho ff7ec82c4d [Core] Optimize SPMD architecture with delta + serialization optimization (#7109) 2024-08-18 17:57:20 -07:00
SangBin Cho 4706eb628e [aDAG] Unflake aDAG + PP tests (#7600) 2024-08-16 20:49:30 -07:00
SangBin Cho 1adddb14bf [Core] Fix ray forward_dag error mssg (#6792) 2024-07-25 16:53:25 -07:00
SangBin Cho d18bab3587 [CI] Fix base url doesn't strip "/" (#6087) 2024-07-02 21:31:25 -07:00
SangBin Choandsang f5e73c9f1b [Lora] Use safetensor keys instead of adapter_config.json to find unexpected modules. (#5909) 
Co-authored-by: sang <sangcho@anyscale.com>
 2024-06-30 17:11:15 +00:00
SangBin ChoandCyrus Leung e691918e3b [misc] Do not allow to use lora with chunked prefill. (#5538) 
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2024-06-15 14:59:36 +00:00
SangBin Cho 847cdcca1c [CI] Upgrade codespell version. (#5381) 2024-06-12 10:06:14 -07:00
SangBin ChoDarkLight1337ywang96
 		246598a6b1 [CI] docfix (#5410) 
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: ywang96 <ywang@roblox.com>
 2024-06-11 01:28:50 -07:00
SangBin Cho c74c913bfb [misc] remove comments that were supposed to be removed (#4977) 2024-05-22 09:02:58 -04:00
SangBin Cho 2e9a2227ec [Lora] Support long context lora (#4787) 
Currently we need to call rotary embedding kernel for each LoRA, which makes it hard to serve multiple long context length LoRA. Add batched rotary embedding kernel and pipe it through.

It replaces the rotary embedding layer to the one that is aware of multiple cos-sin-cache per scaling factors.

Follow up of https://github.com/vllm-project/vllm/pull/3095/files
 2024-05-18 16:05:23 +09:00
SangBin Cho 65bf2ac165 [Core][2/N] Model runner refactoring part 2. Combine prepare prefill / decode to a single API (#4681) 
This PR combines prepare_prompt and prepare_decode into a single API. This PR also coelsce the attn metadata for prefill/decode to a single class and allow to slice them when running attn backend.

It also refactors subquery_start_loc which was not refactored in the previous PR
 2024-05-15 14:00:10 +09:00
SangBin Cho 8a7cc254a0 Revert "[Kernel] Use flash-attn for decoding (#3648)" (#4820) 
Lora 3 & 4 test seems to have illegal memory access failure after this commit;

[2024-05-14 23:51:18,182 E 22 22] logging.cc:101: Unhandled exception: N3c105ErrorE. what(): CUDA error: an illegal memory access was encountered
<br class="Apple-interchange-newline">
Exmaple: https://buildkite.com/vllm/ci/builds/7382#018f793d-1527-4e1c-ab59-c3a34ec55241

This reverts commit 1356df5.

FILL IN THE PR DESCRIPTION HERE

FIX #xxxx (link existing issues this PR will resolve)
 2024-05-15 11:52:45 +09:00
SangBin ChoandRobert Shaw e7c46b9527 [Scheduler] Warning upon preemption and Swapping (#4647) 
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
 2024-05-13 23:50:44 +09:00
SangBin Cho 6a0f617210 [Core] Fix circular reference which leaked llm instance in local dev env (#4737) 
Storing exception frame is extremely prone to circular refernece because it contains the reference to objects.

When tensorizer is not installed, it leaks llm instance because error frame has references to various modules which cause circular reference problem.

I also found spec decoding has a circular reference issue, and I solved it using weakref.proxy.
 2024-05-10 23:54:32 +09:00
SangBin Cho f6a593093a [CI] Make mistral tests pass (#4596) 2024-05-08 08:44:35 -07:00
SangBin Cho d7740ea4dc [Core] Optimize sampler get_logprobs (#4594) 2024-05-08 08:42:28 -07:00
SangBin Cho 36fb68f947 [Doc] Chunked Prefill Documentation (#4580) 2024-05-04 00:18:00 -07:00
SangBin Cho 3521ba4f25 [Core][Model runner refactoring 1/N] Refactor attn metadata term (#4518) 2024-05-03 10:20:12 -07:00
SangBin Cho 0f8a91401c [Core] Ignore infeasible swap requests. (#4557) 2024-05-02 14:31:20 -07:00
SangBin Cho fb087af52e [mypy][7/N] Cover all directories (#4555) 2024-05-02 10:47:41 -07:00
SangBin ChoandCade Daniel cf8cac8c70 [mypy][6/N] Fix all the core subdirectory typing (#4450) 
Co-authored-by: Cade Daniel <edacih@gmail.com>
 2024-05-02 03:01:00 +00:00
SangBin Cho 0d62fe58db [Bug fix][Core] assert num_new_tokens == 1 fails when SamplingParams.n is not 1 and max_tokens is large & Add tests for preemption (#4451) 2024-05-01 19:24:13 -07:00
SangBin Cho 6f1df80436 [Test] Add ignore_eos test (#4519) 2024-05-01 08:45:42 -04:00
SangBin Cho df29793dc7 [mypy][5/N] Support all typing on model executor (#4427) 2024-04-28 19:01:26 -07:00
SangBin Cho 603ad84815 [Core] Refactoring sampler and support prompt logprob for chunked prefill (#4309) 2024-04-26 13:02:02 +00:00
SangBin ChoandDanny Guinther a88081bf76 [CI] Disable non-lazy string operation on logging (#4326) 
Co-authored-by: Danny Guinther <dguinther@neuralmagic.com>
 2024-04-26 00:16:58 -07:00
SangBin Cho b5b4a398a7 [Mypy] Typing lora folder (#4337) 2024-04-25 19:13:50 +00:00
SangBin Cho 050f285ff6 [Core] Scheduling optimization 2 (#4280) 2024-04-23 08:02:11 +00:00
SangBin Cho 0ae11f78ab [Mypy] Part 3 fix typing for nested directories for most of directory (#4161) 2024-04-22 21:32:44 -07:00
SangBin Cho ad8d696a99 [Core] Scheduler perf fix (#4270) 2024-04-22 21:11:06 +00:00
SangBin ChoandSangBin Cho 533d2a1f39 [Typing] Mypy typing part 2 (#4043) 
Co-authored-by: SangBin Cho <sangcho@sangcho-LT93GQWG9C.local>
 2024-04-17 17:28:43 -07:00
SangBin Cho 4e7ee664e2 [Core] Fix engine-use-ray broken (#4105) 2024-04-16 05:24:53 +00:00
SangBin Cho 37e84a403d [Typing] Fix Sequence type GenericAlias only available after Python 3.9. (#4092) 2024-04-15 14:47:31 -07:00
SangBin Cho 09473ee41c [mypy] Add mypy type annotation part 1 (#4006) 2024-04-12 14:35:50 -07:00
SangBin Cho 36729bac13 [Test] Test multiple attn backend for chunked prefill. (#4023) 2024-04-12 09:56:57 -07:00
SangBin ChoandSimon Mo e42df7227d [Test] Add xformer and flash attn tests (#3961) 
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-04-11 03:09:50 +00:00
SangBin Cho 67b4221a61 [Core][5/N] Fully working chunked prefill e2e (#3884) 2024-04-10 17:56:48 -07:00
SangBin Cho 18de883489 [Chunked Prefill][4/n] Chunked prefill scheduler. (#3853) 2024-04-05 10:17:58 -07:00
SangBin Cho 3dcb3e8b98 [3/N] Refactor scheduler for chunked prefill scheduling (#3550) 2024-04-03 14:13:49 -07:00
SangBin ChoandSimon Mo 26422e477b [Test] Make model tests run again and remove --forked from pytest (#3631) 
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-03-28 21:06:40 -07:00
SangBin Cho b51c1cc9d2 [2/N] Chunked prefill data update (#3538) 2024-03-28 10:06:01 -07:00
SangBin Cho 01bfb22b41 [CI] Try introducing isort. (#3495) 2024-03-25 07:59:47 -07:00
SangBin Cho 3bbff9e5ab Fix 1D query issue from _prune_hidden_states (#3539) 2024-03-21 08:49:06 +00:00
SangBin Cho 6e435de766 [1/n][Chunked Prefill] Refactor input query shapes (#3236) 2024-03-20 14:46:05 -07:00
SangBin Cho 24aecf421a [Tests] Add block manager and scheduler tests (#3108) 2024-03-05 18:23:34 -08:00
SangBin Cho 65b89d16ee [Ray] Integration compiled DAG off by default (#2471) 2024-02-08 09:57:25 -08:00