2023-02-24 12:04:49 +00:00
2023-02-24 11:56:06 +00:00
2023-02-18 19:22:57 +00:00
2023-02-18 19:23:07 +00:00
2023-02-16 07:47:21 +00:00
2023-02-24 12:04:49 +00:00
2023-02-24 11:56:21 +00:00
2023-02-16 20:05:45 +00:00

CacheFlow

Installation

pip install cmake torch transformers
pip install -e .

Run

python server.py
S
Description
A high-throughput and memory-efficient inference and serving engine for LLMs
Readme Apache-2.0
324 MiB
Languages
Python 83%
Rust 6.1%
Cuda 4.1%
C++ 2.9%
JavaScript 1.9%
Other 1.9%