3e9f991d6acd7efd90f04f1f530b837a40c93442
CacheFlow
Installation
pip install cmake torch transformers
pip install flash-attn # This may take up to 10 mins.
pip install -e .
Run
python server.py
Description
A high-throughput and memory-efficient inference and serving engine for LLMs
324 MiB
Languages
Python
83%
Rust
6.1%
Cuda
4.1%
C++
2.9%
JavaScript
1.9%
Other
1.9%