Commit Graph - vLLM - iHore Software Repository

iHore/vLLM

Fork 0

3dd6853bc8 [CI/Build] Add is_quant_method_supported to control quantization test configurations (#5253) Michael Goin 2024-06-12 12:58:02 -04:00
8f89d72090 [Doc] add common case for long waiting time (#5430) youkaichao 2024-06-11 11:12:13 -07:00
99dac099ab [Core][Doc] Default to multiprocessing for single-node distributed case (#5230) Nick HillandAntoni Baum 2024-06-11 11:10:41 -07:00
c4bd03c7c5 [Core][Distributed] add same-node detection (#5369) youkaichao 2024-06-11 10:53:59 -07:00
dcbf4286af [Frontend] Customizable RoPE theta (#5197) sasha0552 2024-06-11 17:42:26 +00:00
00e6a2dc53 [Bugfix] fix lora_dtype value type in arg_utils.py (#5398) Ali Panahi 2024-06-11 10:40:23 -07:00
2e02311a1b [Bugfix] Fix MultiprocessingGPUExecutor.check_health when world_size == 1 (#5254) Junichi Sato 2024-06-12 02:38:07 +09:00
89ec06c33b [Docs] [Spec decode] Fix docs error in code example (#5427) Cade Daniel 2024-06-11 10:31:56 -07:00
9fde251bf0 [Doc] Add an automatic prefix caching section in vllm documentation (#5324) Kuntai Duandsimon-mo 2024-06-11 10:24:59 -07:00
4c2ffb28ff [Speculative decoding] Initial spec decode docs (#5400) Cade Daniel 2024-06-11 10:15:40 -07:00
246598a6b1 [CI] docfix (#5410)
SangBin Cho DarkLight1337 ywang96
2024-06-11 17:28:50 +09:00
8bab4959be [Misc] Remove VLLM_BUILD_WITH_NEURON env variable (#5389) Woosuk Kwon 2024-06-11 00:37:56 -07:00
3c4cebf751 [Doc][Typo] Fixing Missing Comma (#5403) Roger Wang 2024-06-11 00:20:28 -07:00
d8f31f2f8b [Doc] add debugging tips (#5409) youkaichao 2024-06-10 23:21:43 -07:00
640052b069 [Bugfix][Frontend] Cleanup "fix chat logprobs" (#5026) Cyrus Leung 2024-06-11 13:36:46 +08:00
351d5e7b82 [Bugfix] OpenAI entrypoint limits logprobs while ignoring server defined --max-logprobs (#5312) maor-psandDarkLight1337 2024-06-11 05:30:31 +03:00
a008629807 [Misc] Various simplifications and typing fixes (#5368) Nick Hill 2024-06-10 19:29:02 -07:00
76477a93b7 [ci] Fix Buildkite agent path (#5392) Kevin H. Luu 2024-06-10 18:58:07 -07:00
77c87beb06 [Doc] Add documentation for FP8 W8A8 (#5388) Michael Goin 2024-06-10 20:55:12 -04:00
114332b88e Bump version to v0.5.0 (#5384) Simon Mo 2024-06-10 17:56:06 -05:00
cb77ad836f [Docs] Alphabetically sort sponsors (#5386) Woosuk Kwon 2024-06-10 13:17:19 -07:00
856c990041 [Docs] Add Docs on Limitations of VLM Support (#5383) Roger Wang 2024-06-10 09:53:50 -07:00
c5602f0baa [ci] Mount buildkite agent on Docker container to upload benchmark results (#5330) Kevin H. Luu 2024-06-10 09:22:34 -07:00
f7f9c5f97b [ci] Use small_cpu_queue for doc build (#5331) Kevin H. Luu 2024-06-10 09:21:11 -07:00
2c0d933594 [Bugfix] Fix LLaVA-NeXT (#5380) Cyrus Leung 2024-06-10 23:38:47 +08:00
774d1035e4 [Feature][Frontend]: Continued stream_options implementation also in CompletionRequest (#5319) Itay Etelis 2024-06-10 17:22:09 +03:00
6b29d6fe70 [Model] Initial support for LLaVA-NeXT (#4199) Cyrus LeungandRoger Wang 2024-06-10 20:47:15 +08:00
0bfa1c4f13 [Misc] Improve error message when LoRA parsing fails (#5194) Cyrus Leung 2024-06-10 19:38:49 +08:00
c81da5f56d [misc][typo] fix typo (#5372) youkaichao 2024-06-10 02:51:02 -07:00
68bc81703e [Frontend][Misc] Enforce Pixel Values as Input Type for VLMs in API Server (#5374) Roger Wang 2024-06-10 02:13:39 -07:00
5884c2b454 [Misc] Update to comply with the new compressed-tensors config (#5350) Dipika SikkaandMichael Goin 2024-06-09 23:49:46 -04:00
45f92c00cf [Bugfix] Fix KeyError: 1 When Using LoRA adapters (#5164) Bla_ckB 2024-06-10 06:23:14 +07:00
5467ac3196 [Kernel][Misc] Use TORCH_LIBRARY instead of PYBIND11_MODULE for custom ops (#5047) bnellnm 2024-06-09 16:23:30 -04:00
5d7e3d0176 [mis][ci/test] fix flaky test in test_sharded_state_loader.py (#5361) youkaichao 2024-06-08 20:50:14 -07:00
0373e1837e [Core][CUDA Graph] add output buffer for cudagraph (#5074) youkaichao 2024-06-08 19:14:43 -07:00
c09dade2a2 [Misc][Breaking] Change FP8 checkpoint format from act_scale -> input_scale (#5353) Michael Goin 2024-06-08 13:54:05 -04:00
8ea5e44a43 [CI/Test] improve robustness of test (vllm_runner) (#5357) youkaichao 2024-06-08 01:59:20 -07:00
9fb900f90c [CI/Test] improve robustness of test (hf_runner) (#5347) youkaichao 2024-06-07 22:31:32 -07:00
c96fc06747 [ROCm][AMD] Use pytorch sdpa math backend to do naive attention (#4965) Hongxia Yang 2024-06-07 22:13:12 -04:00
b3376e5c76 [Misc] Add args for selecting distributed executor to benchmarks (#5335) Benjamin Kitor 2024-06-07 18:20:16 -07:00
e69ded7d1c [Bug Fix] Fix the support check for FP8 CUTLASS (#5352) Cheng Li 2024-06-07 17:42:05 -07:00
767c727a81 fix DbrxFusedNormAttention missing cache_config (#5340) Calvinn Ngandteam 2024-06-08 05:10:21 +08:00
6840a71610 [Misc] Remove unused cuda_utils.h in CPU backend (#5345) Jie Fu (傅杰) 2024-06-08 05:09:13 +08:00
7a9cb294ae [Frontend] Add OpenAI Vision API Support (#5237) Roger WangandDarkLight1337 2024-06-07 11:23:32 -07:00
ca3ea51bde [Kernel] Dynamic Per-Token Activation Quantization (#5037)
Dipika Sikka Varun Sundar Rabindranath Varun Sundar Rabindranath
2024-06-07 12:36:26 -04:00
dc49fb892c Addition of lacked ignored_seq_groups in _schedule_chunked_prefill (#5296) limingshu 2024-06-07 21:35:42 +08:00
18a277b52d Remove Ray health check (#4693) Antoni Baum 2024-06-07 03:01:56 -07:00
8d75fe48ca [Kernel] Switch fp8 layers to use the CUTLASS kernels (#5183) Tyler Michael Smith 2024-06-07 04:42:35 -04:00
388596c914 [Misc][Utils] allow get_open_port to be called for multiple times (#5333) youkaichao 2024-06-06 22:15:11 -07:00
baa15a9ec3 [Feature][Frontend]: Add support for stream_options in ChatCompletionRequest (#5135) Itay Etelis 2024-06-07 06:29:24 +03:00
15063741e3 [Misc] Missing error message for custom ops import (#5282) Jie Fu (傅杰) 2024-06-07 11:17:21 +08:00
ccdc490dda [Core] Change LoRA embedding sharding to support loading methods (#5038) Antoni Baum 2024-06-06 19:07:57 -07:00
a31cab7556 [Core] Avoid copying prompt/output tokens if no penalties are used (#5289) Antoni Baum 2024-06-06 18:12:00 -07:00
828da0d44e [Frontend] enable passing multiple LoRA adapters at once to generate() (#5300) Matthew Goldey 2024-06-06 16:48:13 -04:00
abe855d637 [Kernel] Retune Mixtral 8x22b configs for FP8 on H100 (#5294) Philipp Moritz 2024-06-06 09:29:29 -07:00
4efff036f0 Bugfix: fix broken of download models from modelscope (#5233) liuyhwangyhandmulin.lyh 2024-06-07 00:28:10 +08:00
89c920785f [CI/Build] Update vision tests (#5307) Cyrus Leung 2024-06-06 18:17:18 +08:00
7b0a0dfb22 [Frontend][Core] Update Outlines Integration from FSM to Guide (#4109)
Breno Faria Simon Mo Breno Faria
2024-06-06 01:49:12 +02:00
3a6ae1d33c [CI] Disable flash_attn backend for spec decode (#5286) Simon Mo 2024-06-05 17:49:27 -05:00
8f1729b829 [Docs] Add Ray Summit CFP (#5295) Simon Mo 2024-06-05 17:25:18 -05:00
6a7c7711a2 [Misc] Skip for logits_scale == 1.0 (#5291) Woosuk Kwon 2024-06-05 15:19:02 -07:00
0f83ddd4d7 [Bugfix][Frontend/Core] Don't log exception when AsyncLLMEngine gracefully shuts down. (#5290) Alex Wu 2024-06-05 15:18:12 -07:00
065aff6c16 [Bugfix] Make EngineArgs use named arguments for config construction (#5285) Michael Goin 2024-06-05 18:16:56 -04:00
3d33e372a1 [BugFix] Fix log message about default max model length (#5284) Nick Hill 2024-06-05 14:53:16 -07:00
faf71bcd4b [Speculative Decoding] Add ProposerWorkerBase abstract class (#5252) Nick Hill 2024-06-05 14:53:05 -07:00
f270a39537 [Docs] Add Sequoia as sponsors (#5287) Simon Mo 2024-06-05 13:02:56 -05:00
51a08e7d8f [Kernel] Re-tune Mixtral MoE configurations for FP8 on H100 (#5238) Philipp Moritz 2024-06-05 10:59:14 -07:00
eb8fcd2666 [BugFix] Apply get_cached_tokenizer to the tokenizer setter of LLM (#5207) DriverSongandqiujiawei9 2024-06-06 01:59:02 +08:00
5563a4dea8 [Model] Correct Mixtral FP8 checkpoint loading (#5231) Cody Yu 2024-06-05 10:58:50 -07:00
ccd4f129e8 [Kernel] Add GPU architecture guards to the CUTLASS w8a8 kernels to reduce binary size (#5157) Tyler Michael SmithandCody Yu 2024-06-05 13:44:15 -04:00
02cc3b51a7 [misc] benchmark_serving.py -- add ITL results and tweak TPOT results (#5263) Tyler Michael Smith 2024-06-05 13:17:51 -04:00
d5b1eb081e [CI] Add nightly benchmarks (#5260) Simon Mo 2024-06-05 11:42:08 -05:00
f0a500545f [Frontend] OpenAI API server: Add add_special_tokens to ChatCompletionRequest (default False) (#5278) tomeras91 2024-06-05 19:32:58 +03:00
c65146e75e [Misc] Fix docstring of get_attn_backend (#5271) Woosuk Kwon 2024-06-05 09:18:59 -07:00
41ca62cf03 [Misc] Add CustomOp interface for device portability (#5255) Woosuk Kwon 2024-06-05 09:18:19 -07:00
974fc9b845 [Bugfix] Fix prompt_logprobs when SamplingParams.detokenize is set to True (#5226) zifeitong 2024-06-04 19:37:28 -07:00
fee4dcc33a [Misc] update collect env (#5261) youkaichao 2024-06-04 15:29:09 -07:00
650a4cc55e [Misc] Add transformers version to collect_env.py (#5259) Michael Goin 2024-06-04 15:52:28 -04:00
9ca62d8668 [CI] mark AMD test as softfail to prevent blockage (#5256) Simon Mo 2024-06-04 13:34:53 -05:00
45c35f0d58 [CI/Build] Reducing CPU CI execution time (#5241) Li, Jiang 2024-06-05 01:26:40 +08:00
9ba093b4f4 [CI/Build] Simplify model loading for HfRunner (#5251) Cyrus Leung 2024-06-05 01:09:19 +08:00
27208be66e [Kernel] Add back batch size 1536 and 3072 to MoE tuning (#5242) Woosuk Kwon 2024-06-04 09:58:47 -07:00
87d5abef75 [Bugfix] Fix a bug caused by pip install setuptools>=49.4.0 for CPU backend (#5249) Jie Fu (傅杰) 2024-06-05 00:57:51 +08:00
ec784b2526 [CI/Build] Add inputs tests (#5215) Cyrus Leung 2024-06-04 12:01:46 +08:00
a58f24e590 [Bugfix] Fix torch.compile() error when using MultiprocessingGPUExecutor (#5229) zifeitong 2024-06-03 20:55:50 -07:00
f42a006b15 [Bugfix]: During testing, use pytest monkeypatch for safely overriding the env var that indicates the vLLM backend (#5210) afeldman-nm 2024-06-03 23:32:57 -04:00
3a434b07ed [Kernel] Enhance MoE benchmarking & tuning script (#4921) Woosuk Kwon 2024-06-03 20:06:59 -07:00
bd0e7802e0 [Bugfix] Add warmup for prefix caching example (#5235) Zhuohan Li 2024-06-03 19:36:41 -07:00
06b2550cbb [Bugfix] Support prompt_logprobs==0 (#5217) Toshiki Kataoka 2024-06-04 09:59:30 +09:00
f775a07e30 [FRONTEND] OpenAI tools support named functions (#5032) Breno Faria 2024-06-04 01:25:29 +02:00
4f0d17c05c New CI template on AWS stack (#5110) Kevin H. Luu 2024-06-03 16:16:43 -07:00
10c38e3e46 [Misc]: Implement CPU/GPU swapping in BlockManagerV2 (#3834) Kaiyang Chen 2024-06-04 04:37:11 +08:00
cafb8e06c5 [CI/BUILD] enable intel queue for longer CPU tests (#4113) Yuan 2024-06-04 01:39:50 +08:00
cbb2f59cc8 [Kernel] Pass a device pointer into the quantize kernel for the scales (#5159) Tyler Michael Smith 2024-06-03 12:52:30 -04:00
0ab278ca31 [Core] Remove unnecessary copies in flash attn backend (#5138) Antoni Baum 2024-06-03 09:39:31 -07:00
7a64d24aad [Core] Support image processor (#4197) Cyrus Leung 2024-06-03 13:56:41 +08:00
dfbe60dc62 [Misc] Simplify code and fix type annotations in conftest.py (#5118) Cyrus Leung 2024-06-03 07:05:50 +08:00
a66cf40b20 [Kernel][ROCm][AMD] enable fused topk_softmax kernel for moe layer (#4927) Divakar Verma 2024-06-02 16:13:26 -05:00
f790ad3c50 [Frontend][OpenAI] Support for returning max_model_len on /v1/models response (#4643) Avinash Raj 2024-06-02 13:36:13 +05:30
ed59a7ed23 Update test_ignore_eos (#4898) Simon Mo 2024-06-01 21:21:53 -05:00

... 217 218 219 220 221 ...

Commit Graph Select branches Hide Pull Requests main patches sm86 Mono Color

Commit Graph

Select branches

Hide Pull Requests

main

patches

sm86