iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
6,768 Commits 3 Branches 0 Tags
6881107948c00a8564bc2fa85308f6fc2f065d64
Commit Graph
768 Commits
Author SHA1 Message Date
Roger Wang bd620b01fb [Kernel][CPU] Add Quick gelu to CPU (#5717) 2024-06-21 06:39:40 +00:00
Tyler Michael Smith 3f3b6b2150 [Bugfix] Fix the CUDA version check for FP8 support in the CUTLASS kernels (#5715) 2024-06-20 18:36:10 +00:00
Roger Wang ad137cd111 [Model] Port over CLIPVisionModel for VLMs (#5591) 2024-06-20 11:52:09 +00:00
Dipika Sikka 4a30d7e3cc [Misc] Add per channel support for static activation quantization; update w8a8 schemes to share base classes (#5650) 2024-06-19 18:06:44 -04:00
Shukant Pal 59a1eb59c9 [Bugfix] Fix Phi-3 Long RoPE scaling implementation (#5628) 2024-06-19 01:46:38 +00:00
Dipika Sikka 95db455e7f [Misc] Add channel-wise quantization support for w8a8 dynamic per token activation quantization (#5542) 2024-06-18 12:45:05 -04:00
sroy745 fa9e385229 [Speculative Decoding 1/2 ] Add typical acceptance sampling as one of the sampling techniques in the verifier (#5131) 2024-06-17 21:29:09 -05:00
Kunshang JiJiang LiAbhilash MajumderAbhilash Majumder
 		728c4c8a06 [Hardware][Intel GPU] Add Intel GPU(XPU) inference backend (#3814) 
Co-authored-by: Jiang Li <jiang1.li@intel.com>
Co-authored-by: Abhilash Majumder <abhilash.majumder@intel.com>
Co-authored-by: Abhilash Majumder <30946547+abhilash1910@users.noreply.github.com>
 2024-06-17 11:01:25 -07:00
Dipika Sikka 890d8d960b [Kernel] compressed-tensors marlin 24 support (#5435) 2024-06-17 12:32:48 -04:00
Amit Garg 9333fb8eb9 [Model] Rename Phi3 rope scaling type (#5595) 2024-06-17 12:04:14 -04:00
Cyrus Leung 0e9164b40a [mypy] Enable type checking for test directory (#5017) 2024-06-15 04:45:31 +00:00
Robert ShawmgoinDipika Sikka
 		15985680e2 [ Misc ] Rs/compressed tensors cleanup (#5432) 
Co-authored-by: mgoin <michael@neuralmagic.com>
Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com>
 2024-06-14 10:01:46 -07:00
Tyler Michael Smith 703475f6c2 [Kernel] Fix CUTLASS 3.x custom broadcast load epilogue (#5516) 2024-06-14 09:30:15 -07:00
Tyler Michael Smith e38042d4af [Kernel] Disable CUTLASS kernels for fp8 (#5505) 2024-06-13 13:38:05 -07:00
Tyler Michael SmithMichael GoinyoukaichaozifeitongRobert Shaw
 		85657b5607 [Kernel] Factor out epilogues from cutlass kernels (#5391) 
Co-authored-by: Michael Goin <michael@neuralmagic.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: zifeitong <zifei.tong@parasail.io>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
 2024-06-13 11:22:19 -07:00
wenyujin333 bd43973522 [Kernel] Tune Qwen2MoE kernel configurations with tp2,4 (#5497) 
Tune Qwen2-57B-A14B configs based on #4921

Throughput Performance
command: python benchmarks/benchmark_throughput.py --model=Qwen/Qwen2-57B-A14B-Instruct --input-len 1000 --output-len 50 -tp 2

A100 GPU

benchmark	no config	w/ PR
tp=2	10.53 requests/s, 11058.17 tokens/s	12.47 requests/s, 13088.57 tokens/s
tp=4	17.77 requests/s, 18662.95 tokens/s	20.20 requests/s, 21212.32 tokens/s
 2024-06-13 09:01:10 -07:00
Dipika SikkaandRobert Shaw c2637a613b [Kernel] w4a16 support for compressed-tensors (#5385) 
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
 2024-06-13 10:19:56 -04:00
Woosuk Kwon 1a8bfd92d5 [Hardware] Initial TPU integration (#5292) 2024-06-12 11:53:03 -07:00
Nick Hill a008629807 [Misc] Various simplifications and typing fixes (#5368) 2024-06-11 10:29:02 +08:00
Dipika SikkaandMichael Goin 5884c2b454 [Misc] Update to comply with the new compressed-tensors config (#5350) 
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2024-06-10 03:49:46 +00:00
bnellnm 5467ac3196 [Kernel][Misc] Use TORCH_LIBRARY instead of PYBIND11_MODULE for custom ops (#5047) 2024-06-09 16:23:30 -04:00
Michael Goin c09dade2a2 [Misc][Breaking] Change FP8 checkpoint format from act_scale -> input_scale (#5353) 2024-06-08 13:54:05 -04:00
Cheng Li e69ded7d1c [Bug Fix] Fix the support check for FP8 CUTLASS (#5352) 
Bug description:
With torch 2.4.0.dev20240603+cu121,
cutlass_fp8_supported outputs False, and the (capability, version) before the comparison is (90, 11111111112)

This PR fixes the support check for FP8 CUTLASS ( cutlass_fp8_supported) which was introduced in https://github.com/vllm-project/vllm/pull/5183.
 2024-06-08 00:42:05 +00:00
Dipika SikkaVarun Sundar RabindranathVarun Sundar Rabindranath
 		ca3ea51bde [Kernel] Dynamic Per-Token Activation Quantization (#5037) 
Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
 2024-06-07 09:36:26 -07:00
Tyler Michael Smith 8d75fe48ca [Kernel] Switch fp8 layers to use the CUTLASS kernels (#5183) 
Switching from torch._scaled_mm to vLLM's cutlass fp8 kernels when supported as we are seeing 5-15% improvement in e2e performance on neuralmagic/Meta-Llama-3-8B-Instruct-FP8

see https://docs.google.com/spreadsheets/d/1GiAnmzyGHgZ6zL_LDSTm35Bdrt4A8AaFEurDlISYYA4/ for some quick e2e benchmarks and #5144 for comparisons across different GEMM sizes.
 2024-06-07 08:42:35 +00:00
Antoni Baum ccdc490dda [Core] Change LoRA embedding sharding to support loading methods (#5038) 2024-06-06 19:07:57 -07:00
Philipp Moritz abe855d637 [Kernel] Retune Mixtral 8x22b configs for FP8 on H100 (#5294) 2024-06-06 09:29:29 -07:00
Woosuk Kwon 6a7c7711a2 [Misc] Skip for logits_scale == 1.0 (#5291) 2024-06-05 15:19:02 -07:00
Philipp Moritz 51a08e7d8f [Kernel] Re-tune Mixtral MoE configurations for FP8 on H100 (#5238) 2024-06-05 10:59:14 -07:00
Cody Yu 5563a4dea8 [Model] Correct Mixtral FP8 checkpoint loading (#5231) 2024-06-05 10:58:50 -07:00
Woosuk Kwon 41ca62cf03 [Misc] Add CustomOp interface for device portability (#5255) 2024-06-05 09:18:19 -07:00
Woosuk Kwon 3a434b07ed [Kernel] Enhance MoE benchmarking & tuning script (#4921) 2024-06-03 20:06:59 -07:00
Tyler Michael Smith cbb2f59cc8 [Kernel] Pass a device pointer into the quantize kernel for the scales (#5159) 2024-06-03 09:52:30 -07:00
Divakar Verma a66cf40b20 [Kernel][ROCm][AMD] enable fused topk_softmax kernel for moe layer (#4927) 
This PR enables the fused topk_softmax kernel used in moe layer for HIP
 2024-06-02 14:13:26 -07:00
chenqianfzh b9c0605a8e [Feature][Kernel] Support bitsandbytes quantization and QLoRA (#4776) 2024-06-01 14:51:10 -06:00
Tyler Michael Smith 260d119e86 [Kernel] Refactor CUTLASS kernels to always take scales that reside on the GPU (#5137) 2024-06-01 06:45:32 +00:00
Cody Yu e9899fb7a4 [Model] Enable FP8 QKV in MoE and refine kernel tuning script (#5039) 2024-05-31 14:29:19 -07:00
Robert Shaw b35be5403f [Bugfix] Avoid Warnings in SparseML Activation Quantization (#5120) 2024-05-30 17:04:37 -07:00
Alexander Matveev 5bf185a1c4 [Bugfix] gptq_marlin: Ensure g_idx_sort_indices is not a Parameter (#5108) 2024-05-30 00:30:18 +00:00
Divakar Verma dd8de11f0a [Kernel][ROCm][AMD] Add fused_moe Triton configs for MI300X (#4951) 
This PR adds Triton kernel configs for the MoE kernel for MI300X
 2024-05-28 16:03:23 +00:00
sasha0552 fbdb7b3ee2 [Core] Allow AQLM on Pascal (#5058) 2024-05-27 15:26:14 -07:00
Elisei SmirnovandElisei Smirnov e3470f8753 [Core]: Option To Use Prompt Token Ids Inside Logits Processor (#4985) 
Co-authored-by: Elisei Smirnov <el.smirnov@innopolis.university>
 2024-05-23 22:04:24 +00:00
Dipika SikkaVarun Sundar RabindranathVarun Sundar Rabindranath
 		a1242324c9 [Kernel] Initial Activation Quantization Support (#4525) 
Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
 2024-05-23 21:29:18 +00:00
Alexander Matveev 6066253296 Marlin 24 prefill performance improvement (about 25% better on average) (#4983) 2024-05-23 02:39:27 -04:00
Cody Yu a3a73ab069 [Misc] Load FP8 kv-cache scaling factors from checkpoints (#4893) 
The 2nd PR for #4532.

This PR supports loading FP8 kv-cache scaling factors from a FP8 checkpoint (with .kv_scale parameter).
 2024-05-22 13:28:20 -07:00
Alexander Matveev 27ce85476e [Kernel] Add marlin_24 unit tests (#4901) 2024-05-19 11:37:34 -04:00
SangBin Cho 2e9a2227ec [Lora] Support long context lora (#4787) 
Currently we need to call rotary embedding kernel for each LoRA, which makes it hard to serve multiple long context length LoRA. Add batched rotary embedding kernel and pipe it through.

It replaces the rotary embedding layer to the one that is aware of multiple cos-sin-cache per scaling factors.

Follow up of https://github.com/vllm-project/vllm/pull/3095/files
 2024-05-18 16:05:23 +09:00
Jinzhen Lin 33e0823de5 [Bugfix] fix rope error when load models with different dtypes (#4835) 2024-05-17 18:43:34 +09:00
Alexander MatveevandRobert Shaw 6979ade384 Add GPTQ Marlin 2:4 sparse structured support (#4790) 
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>
 2024-05-16 12:56:15 -04:00
Jinzhen Lin 99caa49106 [Kernel] add bfloat16 support for gptq marlin kernel (#4788) 2024-05-16 09:55:29 -04:00