Commit Graph - vLLM - iHore Software Repository

iHore/vLLM

Fork 0

e02ac55617 [Performance] Optimize e2e overheads: Reduce python allocations (#7162) Alexander Matveev 2024-08-09 00:34:28 -04:00
73388c07a4 [TPU] Fix dockerfile.tpu (#7331) Woosuk Kwon 2024-08-08 20:24:58 -07:00
7eb4a51c5f [Core] Support serving encoder/decoder models (#7258) Cyrus Leung 2024-08-09 10:39:41 +08:00
0fa14907da [TPU] Add Load-time W8A16 quantization for TPU Backend (#7005) Siyuan Liu 2024-08-08 18:35:49 -07:00
5923532e15 Add Skywork AI as Sponsor (#7314) Simon Mo 2024-08-08 13:59:57 -07:00
a049b107e2 [Misc] Temporarily resolve the error of BitAndBytes (#7308) Jee Jee Li 2024-08-09 04:42:58 +08:00
8334c39f37 [Bugfix] Fix new Llama3.1 GGUF model loading (#7269) Isotr0py 2024-08-09 04:42:44 +08:00
e904576743 [CI/Build] Dockerfile.cpu improvements (#7298) Daniele 2024-08-08 21:24:52 +02:00
e14fb22e59 [Doc] Put collect_env issue output in a <detail> block (#7310) Michael Goin 2024-08-08 14:22:49 -04:00
782e53ab59 [Bugfix][fast] Fix the get_num_blocks_touched logic (#6849) Zach Zheng 2024-08-08 10:43:30 -07:00
21b9c49aa3 [Frontend] Kill the server on engine death (#6594) Joe Runde 2024-08-08 10:47:48 -06:00
5fb4a3f678 [Bugfix][Kernel] Increased atol to fix failing tests (#7305) Luka Govedič 2024-08-08 12:16:13 -04:00
757ac70a64 [Model] Rename MiniCPMVQwen2 to MiniCPMV2.6 (#7273) Jee Jee Li 2024-08-08 22:02:41 +08:00
6dffa4b0a6 [Bugfix] Fix LoRA with PP (#7292) Murali Andoorveedu 2024-08-08 00:02:27 -07:00
48abee9e54 [Frontend] remove max_num_batched_tokens limit for lora (#7288) Cherilyn Buren 2024-08-08 14:17:29 +08:00
746709642c [Misc] Fix typos in scheduler.py (#7285) Rui Qiao 2024-08-07 17:06:01 -07:00
e53dfd3eaf [Kernel] Fix Flashinfer Correctness (#7284) Lily Liu 2024-08-07 16:26:52 -07:00
6d94420246 [Doc] Update supported_hardware.rst (#7276) Michael Goin 2024-08-07 17:21:50 -04:00
fc1493a01e [FrontEnd] Make merge_async_iterators is_cancelled arg optional (#7282) Nick Hill 2024-08-07 13:35:14 -07:00
311f743831 [Bugfix] Fix gptq failure on T4s (#7264) Lucas Wilkinson 2024-08-07 16:05:37 -04:00
469b3bc538 [ci] Make building wheels per commit optional (#7278) Kevin H. Luu 2024-08-07 11:34:25 -07:00
5223199e03 [Bugfix][FP8] Fix dynamic FP8 Marlin quantization (#7219) Michael Goin 2024-08-07 14:23:12 -04:00
fde47d3bc2 [BugFix] Fix frontend multiprocessing hang (#7217) Maximilien de BayserandRobert Shaw 2024-08-07 15:09:36 -03:00
0e12cd67a8 [Doc] add online speculative decoding example (#7243) Stas Bekman 2024-08-07 09:58:02 -07:00
80cbe10c59 [OpenVINO] migrate to latest dependencies versions (#7251) Ilya Lavrenov 2024-08-07 20:49:10 +04:00
b764547616 [Bugfix] Fix input processor for InternVL2 model (#7164) Isotr0pyandCyrus Leung 2024-08-08 00:32:07 +08:00
ab0f5e2823 Fixes typo in function name (#7275) Rafael Vasquez 2024-08-07 12:29:27 -04:00
564985729a [ BugFix ] Move zmq frontend to IPC instead of TCP (#7222) Robert Shaw 2024-08-07 12:24:56 -04:00
0f7052bc7e [Misc] Refactor linear layer weight loading; introduce BasevLLMParameter and weight_loader_v2 (#5874) Dipika Sikka 2024-08-07 12:17:58 -04:00
639159b2a6 [distributed][misc] add specialized method for cuda platform (#7249) youkaichao 2024-08-07 08:54:52 -07:00
66d617e343 [Frontend] Gracefully handle missing chat template and fix CI failure (#7238) Cyrus LeungandRoger Wang 2024-08-07 17:12:05 +08:00
7b261092de [BUGFIX]: top_k is expected to be an integer. (#7227) Atilla Akkuş 2024-08-07 10:32:16 +03:00
2385c8f374 [Doc] Mock new dependencies for documentation (#7245) Roger Wang 2024-08-06 23:43:03 -07:00
9a3f49ae07 [BugFix] Overhaul async request cancellation (#7111) Nick Hill 2024-08-06 22:21:41 -07:00
f9a5600649 [Bugfix] Fix GPTQ and GPTQ Marlin CPU Offloading (#7225) Michael Goin 2024-08-06 21:34:26 -04:00
fd95e026e0 [Core] Subclass ModelRunner to support cross-attention & encoder sequences (towards eventual encoder/decoder model support) (#4942)
afeldman-nm Andrew Feldman Nick Hill
2024-08-06 16:51:47 -04:00
660470e5a3 [Core] Optimize evictor-v2 performance (#7193) xiaobochen123 2024-08-07 03:34:25 +08:00
8d59dbb000 [Kernel] Add per-tensor and per-token AZP epilogues (#5941) Luka GovedičandTyler Michael Smith 2024-08-06 14:17:08 -04:00
5c60c8c423 [SpecDecode] [Minor] Fix spec decode sampler tests (#7183) Lily Liu 2024-08-06 10:40:32 -07:00
00afc78590 [Bugfix] add gguf dependency (#7198) Katarzyna Papisandkatarzyna.papis 2024-08-06 19:08:35 +02:00
541c1852d3 [ BugFix ] Fix ZMQ when VLLM_PORT is set (#7205) Robert Shaw 2024-08-06 12:26:26 -04:00
a3bbbfa1d8 [BugFix] Fix DeepSeek remote code (#7178) Dipika Sikka 2024-08-06 11:16:53 -04:00
1f26efbb3a [Model] Support SigLIP encoder and alternative decoders for LLaVA models (#7153) Cyrus LeungandRoger Wang 2024-08-06 16:55:31 +08:00
9118217f58 [LoRA] Relax LoRA condition (#7146) Jee Jee Li 2024-08-06 09:57:25 +08:00
e3c664bfcb [Build] Add initial conditional testing spec (#6841) Simon Mo 2024-08-05 17:39:22 -07:00
360bd67cf0 [Core] Support loading GGUF model (#5191) Isotr0pyandMichael Goin 2024-08-06 07:54:23 +08:00
ef527be06c [MISC] Use non-blocking transfer in prepare_input (#7172) Cody Yu 2024-08-05 16:41:27 -07:00
89b8db6bb2 [Bugfix] Specify device when loading LoRA and embedding tensors (#7129) Jacob ScheinandJacob Schein 2024-08-05 16:35:47 -07:00
789937af2e [Doc] [SpecDecode] Update MLPSpeculator documentation (#7100) Thomas Parnell 2024-08-06 01:29:43 +02:00
dfb1a15dcb [ci][frontend] deduplicate tests (#7101) youkaichao 2024-08-05 15:59:22 -07:00
4db5176d97 bump version to v0.5.4 (#7139) Simon Mo 2024-08-05 14:39:48 -07:00
4cf1dc39be [Bugfix][CI/Build] Fix CUTLASS FetchContent (#7171) Tyler Michael Smith 2024-08-05 17:22:57 -04:00
6e4852ce28 [CI/Build] Suppress divide-by-zero and missing return statement warnings (#7001) Tyler Michael Smith 2024-08-05 16:00:01 -04:00
8571ac4672 [Kernel] Update CUTLASS to 3.5.1 (#7085) Tyler Michael Smith 2024-08-05 15:13:43 -04:00
997cf78308 [Misc] Fix typo in GroupCoordinator.recv() (#7167) Rui Qiao 2024-08-05 11:10:16 -07:00
57f560aa23 [BugFix] Use args.trust_remote_code (#7121) Aditya Paliwal 2024-08-05 09:26:14 -07:00
003f8ee128 [BugFix] Use IP4 localhost form for zmq bind (#7163) Nick Hill 2024-08-05 08:41:03 -07:00
e9630458c7 [SpecDecode] Support FlashInfer in DraftModelRunner (#6926) Bongwon Jang 2024-08-06 00:05:05 +09:00
82a1b1a82b [Speculative decoding] Add periodic log with time spent in proposal/scoring/verification (#6963) Cade Daniel 2024-08-05 01:46:44 -07:00
c0d8f1636c [Model] SiglipVisionModel ported from transformers (#6942) Jungho Christopher ChoandRoger Wang 2024-08-05 15:22:12 +09:00
cc08fc7225 [Frontend] Reapply "Factor out code for running uvicorn" (#7095) Cyrus Leung 2024-08-05 11:40:51 +08:00
7b86e7c9cd [Model] Add multi-image support for minicpmv (#7122)
Alphi hezhihui Cyrus Leung
2024-08-05 09:23:17 +08:00
f80ab3521c Clean up remaining Punica C information (#7027) Jee Jee Li 2024-08-05 06:37:08 +08:00
16a1cc9bb2 [misc][distributed] improve libcudart.so finding (#7127) youkaichao 2024-08-04 11:31:51 -07:00
b1c9aa3daa [Bugfix] [SpecDecode] Default speculative_draft_tensor_parallel_size to 1 when using MLPSpeculator (#7105) Thomas Parnell 2024-08-04 16:13:18 +02:00
179a6a36f2 [Model]Refactor MiniCPMV (#7020) Jee Jee LiandCyrus Leung 2024-08-04 16:12:41 +08:00
83c644fe7e [core][misc] simply output processing with shortcut code path (#7117) youkaichao 2024-08-04 00:22:19 -07:00
9fadc7b7a0 [misc] add zmq in collect env (#7119) youkaichao 2024-08-03 22:03:46 -07:00
654bc5ca49 Support for guided decoding for offline LLM (#6878) Yihuan BuandCyrus Leung 2024-08-03 23:12:09 -04:00
825b044863 [Frontend] Warn if user max_model_len is greater than derived max_model_len (#7080) Jeff FialhoandNick Hill 2024-08-03 20:01:38 -03:00
44dcb52e39 [ci][test] finalize fork_new_process_for_each_test (#7114) youkaichao 2024-08-03 10:44:53 -07:00
67d745cc68 [CI] Temporarily turn off H100 performance benchmark (#7104) Kuntai Du 2024-08-02 23:52:44 -07:00
99d7cabd7b [LoRA] ReplicatedLinear support LoRA (#7081) Jee Jee Li 2024-08-03 13:40:19 +08:00
fb2c1c86c1 [Bugfix] Fix block table for seqs that have prefix cache hits (#7018) Zach Zheng 2024-08-02 22:38:15 -07:00
0c25435daa [Model] Refactor and decouple weight loading logic for InternVL2 model (#7067) Isotr0py 2024-08-03 13:36:14 +08:00
a0d164567c [ci][distributed] disable ray dag tests (#7099) youkaichao 2024-08-02 22:32:04 -07:00
04e5583425 [ci][distributed] merge distributed test commands (#7097) youkaichaoandCyrus Leung 2024-08-02 21:33:53 -07:00
8c025fa703 [Frontend] Factor out chat message parsing (#7055) Cyrus Leung 2024-08-03 12:31:27 +08:00
69ea15e5cc [ci][distributed] shorten wait time if server hangs (#7098) youkaichao 2024-08-02 21:05:16 -07:00
ed812a73fa [ Frontend ] Multiprocessing for OpenAI Server with zeromq (#6883)
Robert Shaw Joe Runde Joe Runde Nick Hill Simon Mo
2024-08-02 21:27:28 -04:00
708989341e [misc] add a flag to enable compile (#7092) youkaichao 2024-08-02 16:18:45 -07:00
22e718ff1a [Misc] Revive to use loopback address for driver IP (#7091) Rui Qiao 2024-08-02 15:50:00 -07:00
05308891e2 [Core] Pipeline parallel with Ray ADAG (#6837) Rui Qiao 2024-08-02 13:55:40 -07:00
a8d604ca2a [Misc] Disambiguate quantized types via a new ScalarType (#6396) Lucas Wilkinson 2024-08-02 16:51:58 -04:00
b482b9a5b1 [CI/Build] Add support for Python 3.12 (#7035) Michael Goin 2024-08-02 16:51:22 -04:00
806949514a [ci] set timeout for test_oot_registration.py (#7082) youkaichao 2024-08-02 10:03:24 -07:00
c16eaac500 [Hardware][Intel CPU] Update torch 2.4.0 for CPU backend (#6931) Jie Fu (傅杰) 2024-08-02 23:55:58 +08:00
db35186391 [Core] Comment out unused code in sampler (#7023) Peng Guanwen 2024-08-02 15:58:26 +08:00
660dea1235 [cuda][misc] remove error_on_invalid_device_count_status (#7069) youkaichao 2024-08-02 00:14:21 -07:00
cf2a1a4d9d Fix tracing.py (#7065) Bongwon Jang 2024-08-02 15:28:00 +09:00
252357793d [ci][distributed] try to fix pp test (#7054) youkaichao 2024-08-01 22:03:12 -07:00
3bb4b1e4cd [mypy] Speed up mypy checking (#7056) Cyrus Leung 2024-08-02 10:49:43 +08:00
954f7305a1 [Kernel] Fix input for flashinfer prefill wrapper. (#7008) Lily Liu 2024-08-01 18:44:16 -07:00
6ce01f3066 [Performance] Optimize get_seqs (#7051) Woosuk Kwon 2024-08-01 18:29:52 -07:00
6a11fdfbb8 [CI/Build][Bugfix] Fix CUTLASS header-only line (#7034) Tyler Michael Smith 2024-08-01 16:51:15 -04:00
805a8a75f2 [Misc] Support attention logits soft-capping with flash-attn (#7022) Woosuk Kwon 2024-08-01 13:14:37 -07:00
562e580abc Update run-amd-test.sh (#7044) omkar kakarparthi 2024-08-01 15:12:37 -05:00
fc912e0886 [Models] Support Qwen model with PP (#6974) Murali Andoorveedu 2024-08-01 12:40:43 -07:00
f4fd390f5d [Bugfix] Lower gemma's unloaded_params exception to warning (#7002) Michael Goin 2024-08-01 15:01:07 -04:00
fb3db61688 [CI/Build] Remove sparseml requirement from testing (#7037) Michael Goin 2024-08-01 15:00:51 -04:00

... 210 211 212 213 214 ...

Commit Graph Select branches Hide Pull Requests main patches sm86 Mono Color

Commit Graph

Select branches

Hide Pull Requests

main

patches

sm86