iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
6,768 Commits 3 Branches 0 Tags
6881107948c00a8564bc2fa85308f6fc2f065d64
Commit Graph
69 Commits
Author SHA1 Message Date
Isotr0pyandCyrus Leung 75f81750f3 [VLM] Initialize video input support for InternVL models (#18499) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2025-05-25 04:51:25 +00:00
Chenheli Hua 04eb88dc80 Re-submit: Fix: Proper RGBA -> RGB conversion for PIL images. (#18569) 
Signed-off-by: Chenheli Hua <huachenheli@outlook.com>
 2025-05-23 01:59:18 +00:00
Harry Mellor 26d0419309 Update deprecated type hinting in models (#18132) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-14 22:06:50 -07:00
Woosuk Kwon b411418ff0 [Chore] Remove Sampler from Model Code (#17084) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-04-24 02:49:33 -07:00
Cyrus Leung 56d4aefa33 [VLM] Avoid unnecessary dummy multimodal data during processing (#16416) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-04-10 19:32:14 +00:00
Cyrus Leung 83b824c8b4 [VLM] Remove BaseProcessingInfo.get_mm_max_tokens_per_item (#16408) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-04-10 09:06:58 -07:00
Nicolò Lucchesi d55244df31 [Model] Add SupportsMultiModal.get_language_model interface (#16007) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-04-09 04:12:54 -07:00
Roger WangDarkLight1337mgoinJennifer Zhao
 		f2ebb6f541 [V1] Scatter and gather placeholders in the model runner (#16076) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>
 2025-04-08 10:43:41 +08:00
Roger Wang af51d80fa1 Revert "[V1] Scatter and gather placeholders in the model runner" (#16075) 2025-04-04 14:50:57 -07:00
Cyrus LeungmgoinRoger Wang
 		f5722a5052 [V1] Scatter and gather placeholders in the model runner (#15712) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
 2025-04-04 21:26:44 +00:00
Cyrus Leung e6c9053f9e [Misc] Clean up scatter_patch_features (#15559) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-03-27 07:45:00 +00:00
Cyrus Leung 5994430b84 [Misc] Remove redundant num_embeds (#15443) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-03-25 18:27:57 +08:00
Cyrus Leung ffa443afed [Bugfix] Fix embedding assignment for InternVL-based models (#15086) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-03-20 03:40:13 +00:00
Cyrus Leung ab656f2c2f [Bugfix] Loosen type check to avoid errors in V1 (#15021) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-03-18 12:54:40 +00:00
Cyrus Leung 601bd3268e [Misc] Clean up type annotation for SupportsMultiModal (#14794) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-03-14 00:59:56 -07:00
Isotr0py b1cc4dfef5 [VLM] Support loading InternVideo2.5 models as original InternVLChatModel (#14738) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-03-13 03:10:02 -07:00
lkchen b3cf368d79 [V1][Molmo] Fix get_multimodal_embeddings() in molmo.py (#14161) 2025-03-04 15:43:59 +00:00
Cyrus Leung f1579b229d [VLM] Generalized prompt updates for multi-modal processor (#13964) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-02-27 17:44:25 +00:00
Harry Mellor cdc1fa12eb Remove unused kwargs from model definitions (#13555) 2025-02-24 17:13:52 -08:00
Cyrus Leung 377d10bd14 [VLM][Bugfix] Pass processor kwargs properly on init (#13516) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-02-19 13:13:50 +00:00
Cyrus Leung a4ce74c14a [VLM] Use shared field to pass token ids to model 2025-02-05 13:30:46 -08:00
Cyrus LeungandIsotr0py d1ca7df84d [VLM] Merged multi-modal processor for InternVL-based models (#12553) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
 2025-02-04 16:44:52 +08:00
Russell Bryant e489ad7a21 [Misc] Add SPDX-License-Identifier headers to python source files (#12628) 
- **Add SPDX license headers to python source files**
- **Check for SPDX headers using pre-commit**

commit 9d7ef44c3cfb72ca4c32e1c677d99259d10d4745
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:18:24 2025 -0500

    Add SPDX license headers to python source files
    
This commit adds SPDX license headers to python source files as
recommended to
the project by the Linux Foundation. These headers provide a concise way
that is
both human and machine readable for communicating license information
for each
source file. It helps avoid any ambiguity about the license of the code
and can
    also be easily used by tools to help manage license compliance.
    
The Linux Foundation runs license scans against the codebase to help
ensure
    we are in compliance with the licenses of the code we use, including
dependencies. Having these headers in place helps that tool do its job.
    
    More information can be found on the SPDX site:
    
    - https://spdx.dev/learn/handling-license-info/
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

commit 5a1cf1cb3b80759131c73f6a9dddebccac039dea
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:36:32 2025 -0500

    Check for SPDX headers using pre-commit
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

---------

Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-02-02 11:58:18 -08:00
Roger Wang 969da7d70b [V1][VLM] Fix edge case bug for InternVL2 (#11165) 
Signed-off-by: Roger Wang <ywang@roblox.com>
 2024-12-13 11:09:30 +00:00
Roger Wang a11f326528 [V1] Initial support of multimodal models for V1 re-arch (#10699) 
Signed-off-by: Roger Wang <ywang@roblox.com>
 2024-12-08 12:50:51 +00:00
Isotr0py 10398b4706 [Model] Consolidate ViTs attention implementation without mask (#10893) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2024-12-04 18:11:08 +00:00
Cyrus Leung 133707123e [Model] Replace embedding models with pooling adapter (#10769) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-01 08:02:54 +08:00
Roger Wang 2f0a0a17a4 [V1] Refactor model executable interface for multimodal models (#10570) 
Signed-off-by: Roger Wang <ywang@roblox.com>
 2024-11-26 20:46:11 +00:00
Isotr0py d5ec121f95 [Model] Expose dynamic_image_size as mm_processor_kwargs for InternVL2 models (#10518) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2024-11-21 14:20:08 +00:00
youkaichao 388ee3de66 [torch.compile] limit inductor threads and lazy import quant (#10482) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-20 18:36:33 -08:00
Isotr0py c4e464333e [Misc] Add uninitialized params tracking for AutoWeightsLoader (#10327) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2024-11-18 09:07:46 +08:00
youkaichao 504ac53d18 [misc] error early for old-style class (#10304) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-13 18:55:39 -08:00
Cyrus Leung 0b8bb86bf1 [1/N] Initial prototype for multi-modal processor (#10044) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-11-13 12:39:03 +00:00
youkaichao f89d18ff74 [6/N] pass whole config to inner model (#10205) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-11 06:41:46 +00:00
youkaichao 1a95f10ee7 [5/N] pass the whole config to model (#9983) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-09 14:17:28 +08:00
Cyrus Leung e0191a95d8 [0/N] Rename MultiModalInputs to MultiModalKwargs (#10040) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-11-09 11:31:02 +08:00
Joe Runde d58268c56a [V1] Make v1 more testable (#9888) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2024-11-06 11:57:35 -08:00
Peter Salas 6c0b7f548d [Core][VLM] Add precise multi-modal placeholder tracking (#8346) 
Signed-off-by: Peter Salas <peter@fixie.ai>
 2024-11-01 16:21:10 -07:00
Michael Goin bc73e9821c [Bugfix] Fix prefix strings for quantized VLMs (#9772) 2024-10-29 16:02:59 -07:00
Cyrus LeungandIsotr0py c18e1a3418 [VLM] Enable overriding whether post layernorm is used in vision encoder + fix quant args (#9217) 
Co-authored-by: Isotr0py <2037008807@qq.com>
 2024-10-23 11:27:37 +00:00
Isotr0py bb392ea2d2 [Model][VLM] Initialize support for Mono-InternVL model (#9528) 2024-10-22 16:01:46 +00:00
Cyrus Leung cee711fdbb [Core] Rename input data types (#8688) 2024-10-16 10:49:37 +00:00
hhzhang16 55e081fbad [Bugfix] Update InternVL input mapper to support image embeds (#9351) 2024-10-14 21:29:19 -07:00
Cyrus Leung 8bfaa4e31e [Bugfix] fix composite weight loading and EAGLE weight loading (#9160) 2024-10-09 00:36:55 -07:00
Cyrus LeungRoger WangIsotr0py
 		151ef4efd2 [Model] Support NVLM-D and fix QK Norm in InternViT (#9045) 
Co-authored-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2024-10-07 11:55:12 +00:00
Murali AndoorveeduandDarkLight1337 0f6d7a9a34 [Models] Add remaining model PP support (#7168) 
Signed-off-by: Muralidhar Andoorveedu <muralidhar.andoorveedu@centml.ai>
Signed-off-by: Murali Andoorveedu <muralidhar.andoorveedu@centml.ai>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-10-04 10:56:58 +08:00
Isotr0py 2ae25f79cf [Model] Expose InternVL2 max_dynamic_patch as a mm_processor_kwarg (#8946) 2024-09-30 13:01:20 +08:00
DefTruth 0c4d2ad5e6 [VLM][Bugfix] internvl with num_scheduler_steps > 1 (#8614) 2024-09-25 09:35:53 -07:00
zifeitong e3dd0692fa [BugFix] Propagate 'trust_remote_code' setting in internvl and minicpmv (#8250) 2024-09-25 05:53:43 +00:00
Isotr0py 13d88d4137 [Bugfix] Refactor composite weight loading logic (#8656) 2024-09-22 04:33:27 +00:00