wenyujin333 bd43973522 [Kernel] Tune Qwen2MoE kernel configurations with tp2,4 (#5497) ...

Tune Qwen2-57B-A14B configs based on #4921

Throughput Performance
command: python benchmarks/benchmark_throughput.py --model=Qwen/Qwen2-57B-A14B-Instruct --input-len 1000 --output-len 50 -tp 2

A100 GPU

benchmark	no config	w/ PR
tp=2	10.53 requests/s, 11058.17 tokens/s	12.47 requests/s, 13088.57 tokens/s
tp=4	17.77 requests/s, 18662.95 tokens/s	20.20 requests/s, 21212.32 tokens/s

2024-06-13 09:01:10 -07:00

..

fused_moe

[Kernel] Tune Qwen2MoE kernel configurations with tp2,4 (#5497)

2024-06-13 09:01:10 -07:00

ops

[Mypy] Part 3 fix typing for nested directories for most of directory (#4161)

2024-04-22 21:32:44 -07:00

quantization

[Kernel] w4a16 support for compressed-tensors (#5385)

2024-06-13 10:19:56 -04:00

__init__.py

Change the name to vLLM (#150)

2023-06-17 03:07:40 -07:00

activation.py

[Misc] Add CustomOp interface for device portability (#5255)

2024-06-05 09:18:19 -07:00

layernorm.py

[Misc] Add CustomOp interface for device portability (#5255)

2024-06-05 09:18:19 -07:00

linear.py

[Feature][Kernel] Support bitsandbytes quantization and QLoRA (#4776)

2024-06-01 14:51:10 -06:00

logits_processor.py

[Misc] Skip for logits_scale == 1.0 (#5291)

2024-06-05 15:19:02 -07:00

pooler.py

[Model][Misc] Add e5-mistral-7b-instruct and Embedding API (#3734)

2024-05-11 11:30:37 -07:00

rejection_sampler.py

[Misc] Various simplifications and typing fixes (#5368)

2024-06-11 10:29:02 +08:00

rotary_embedding.py

[Hardware] Initial TPU integration (#5292)

2024-06-12 11:53:03 -07:00

sampler.py

[CORE] Improvement in ranks code (#4718)

2024-05-12 17:47:47 -07:00

vocab_parallel_embedding.py

[Core] Change LoRA embedding sharding to support loading methods (#5038)

2024-06-06 19:07:57 -07:00