iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
1,627 Commits 3 Branches 0 Tags
a3e8a05d4c1b79dd44eb92bb6f57eb40c3fbdb21
Commit Graph
5 Commits
Author SHA1 Message Date
Alexander MatveevandRobert Shaw 6979ade384 Add GPTQ Marlin 2:4 sparse structured support (#4790) 
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>
 2024-05-16 12:56:15 -04:00
SangBin Cho df29793dc7 [mypy][5/N] Support all typing on model executor (#4427) 2024-04-28 19:01:26 -07:00
Cody Yu a62aaf1df5 [Misc][Refactor] Generalize linear_method to be quant_method (#4373) 2024-04-26 16:41:14 -04:00
Woosuk Kwon 8d17774f92 Add AWQ support for all models (#1714) 2023-11-18 17:56:47 -08:00
Zhuohan Li 7076fa1c9f TP/quantization/weight loading refactor part 2 - Refactor quantized linear logic and extend quantization support to all models (#1622) 
Refactor the tensor parallelism, quantization, and weight-loading codes.

Summary of the new features enabled by this PR:
- **All models** are able to be quantized with AWQ and SqueezeLLM, and [soon GPTQ](https://github.com/vllm-project/vllm/pull/1580).
- Model loading code became much simpler.
- Support model parallelism for all MQA/GQA models when the number of key/value heads is smaller than the tensor parallel size.
 2023-11-15 22:50:41 -08:00