2f49f155858faaf82bfd076a821497e41e961658
CacheFlow
Installation
pip install psutil numpy torch transformers
pip install flash-attn # This may take up to 10 mins.
pip install -e .
Run
ray start --head
python server.py [--tensor-parallel-size <N>]
Description
A high-throughput and memory-efficient inference and serving engine for LLMs
324 MiB
Languages
Python
83%
Rust
6.1%
Cuda
4.1%
C++
2.9%
JavaScript
1.9%
Other
1.9%