iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
4,383 Commits 3 Branches 0 Tags
cb3e73e4c8142b5ce8ac34efc2fa04d90f142dc5
Commit Graph
703 Commits
Author SHA1 Message Date
Lucas WilkinsonWoosuk Kwonsimon-moMichael GoinZhuohan LiTyler Michael SmithAlexander Matveevsimon-mo
 		cabaf4eff3 [Attention] MLA decode optimizations (#12528) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
 2025-01-30 23:49:37 -08:00
Aleksandr MalyshevandAleksandr Malyshev a1fc18c030 [ROCm][AMD][Model] llama 3.2 support upstreaming (#12421) 
Signed-off-by: Aleksandr Malyshev <maleksan@amd.com>
Co-authored-by: Aleksandr Malyshev <maleksan@amd.com>
 2025-01-31 12:24:28 +08:00
Pavani Majetyandmgoin b02fd288b2 [Hardware][NV] Fix Modelopt model loading for k-v-scales for Llama models. (#11787) 
Signed-off-by: Pavani Majety <pmajety@nvidia.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
 2025-01-29 01:46:12 -08:00
+29
AlphiSungjae Leeshaochangxushaochangxu.scxCyrus LeungNicolò LucchesisixgodIsotr0pyRoger WangRafael VasquezIsotr0pyCyrus LeungAkshat TripathiOleg MosalovJee Jee LiAvshalom ManevichRobert ShawYangcheng LiSiyuan LiConcurrenseeChenguang LiyoukaichaoAlex BrooksChen ZhangHarry MellorShanshan ShenelijahYikun JiangSteve LuomgoinWoosuk KwonKonrad ZaworaTJiantjtanaawangxiyuanmaang-hElfie GuoRui QiaoRoger Wang
 		d93bf4da85 [Model] Refactoring of MiniCPM-V and add MiniCPM-o-2.6 support for vLLM (#12069) 
Signed-off-by: hzh <hezhihui_thu@163.com>
Signed-off-by: Sungjae Lee <33976427+llsj14@users.noreply.github.com>
Signed-off-by: shaochangxu.scx <shaochangxu.scx@antgroup.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
Signed-off-by: Akshat Tripathi <akshat@krai.ai>
Signed-off-by: Oleg Mosalov <oleg@krai.ai>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Signed-off-by: Yida Wu <yidawu@alumni.cmu.edu>
Signed-off-by: Chenguang Li <757486878@qq.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Alex-Brooks <Alex.brooks@ibm.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Shanshan Shen <467638484@qq.com>
Signed-off-by: elijah <f1renze.142857@gmail.com>
Signed-off-by: Yikun <yikunkero@gmail.com>
Signed-off-by: mgoin <michael@neuralmagic.com>
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Signed-off-by: Konrad Zawora <kzawora@habana.ai>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Co-authored-by: Sungjae Lee <33976427+llsj14@users.noreply.github.com>
Co-authored-by: shaochangxu <85155497+shaochangxu@users.noreply.github.com>
Co-authored-by: shaochangxu.scx <shaochangxu.scx@antgroup.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: sixgod <evethwillbeok@outlook.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Co-authored-by: Rafael Vasquez <rafvasq21@gmail.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Akshat Tripathi <Akshat.tripathi6568@gmail.com>
Co-authored-by: Oleg Mosalov <oleg@krai.ai>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Avshalom Manevich <12231371+avshalomman@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
Co-authored-by: Yangcheng Li <liyangcheng.lyc@alibaba-inc.com>
Co-authored-by: Siyuan Li <94890248+liaoyanqing666@users.noreply.github.com>
Co-authored-by: Concurrensee <yida.wu@amd.com>
Co-authored-by: Chenguang Li <757486878@qq.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Alex Brooks <alex.brooks@ibm.com>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>
Co-authored-by: elijah <30852919+e1ijah1@users.noreply.github.com>
Co-authored-by: Yikun Jiang <yikunkero@gmail.com>
Co-authored-by: Steve Luo <36296769+SunflowerAries@users.noreply.github.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Konrad Zawora <kzawora@habana.ai>
Co-authored-by: TJian <tunjian1996@gmail.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: wangxiyuan <wangxiyuan1007@gmail.com>
Co-authored-by: maang-h <55082429+maang-h@users.noreply.github.com>
Co-authored-by: Elfie Guo <164945471+elfiegg@users.noreply.github.com>
Co-authored-by: Rui Qiao <161574667+ruisearch42@users.noreply.github.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
 2025-01-29 09:24:59 +00:00
Travis JohnsonandWallas Santos 036ca94c25 [Bugfix] handle alignment of arguments in convert_sparse_cross_attention_mask_to_dense (#12347) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
Signed-off-by: Wallas Santos <wallashss@ibm.com>
Co-authored-by: Wallas Santos <wallashss@ibm.com>
 2025-01-29 08:54:35 +00:00
Michael Goin bd02164cf9 Bugfix for whisper quantization due to fake k_proj bias (#12524) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2025-01-29 04:49:03 +00:00
Cyrus Leung 8f58a51358 [VLM] Merged multi-modal processor and V1 support for Qwen-VL (#12504) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-28 16:25:05 +00:00
Harry Mellor 823ab79633 Update pre-commit hooks (#12475) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-01-27 17:23:08 -07:00
Isotr0py ce69f7f754 [Bugfix] Fix gpt2 GGUF inference (#12467) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-01-27 18:31:49 +08:00
Cyrus Leung 5204ff5c3f [Bugfix] Fix Granite 3.0 MoE model loading (#12446) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-26 21:26:44 -08:00
Cyrus Leung fb30ee92ee [Bugfix] Fix BLIP-2 processing (#12412) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-25 11:42:42 +08:00
Gregory ShtrasbergandMicah Williamson e97f802b2d [FP8][Kernel] Dynamic kv cache scaling factors computation (#11906) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Co-authored-by: Micah Williamson <micah.williamson@amd.com>
 2025-01-23 18:04:03 +00:00
Isotr0py c5b4b11d7f [Bugfix] Fix k_proj's bias for whisper self attention (#12342) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-01-23 10:15:33 +00:00
Cyrus Leung 6609cdf019 [Doc] Add docs for prompt replacement (#12318) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-22 14:56:29 +00:00
Roger Wang 16366ee8bb [Bugfix][VLM] Fix mixed-modality inference backward compatibility for V0 (#12313) 
Signed-off-by: Roger Wang <ywang@roblox.com>
 2025-01-22 21:06:36 +08:00
zhou fan 528dbcac7d [Model][Bugfix]: correct Aria model output (#12309) 
Signed-off-by: xffxff <1247714429@qq.com>
 2025-01-22 11:39:19 +00:00
Cyrus Leung cd7b6f0857 [VLM] Avoid unnecessary tokenization (#12310) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-22 11:08:31 +00:00
Kevin H. Luu 64ea24d0b3 [ci/lint] Add back default arg for pre-commit (#12279) 
Signed-off-by: kevin <kevin@anyscale.com>
 2025-01-22 01:15:27 +00:00
Cyrus Leung df76e5af26 [VLM] Simplify post-processing of replacement info (#12269) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-21 16:48:13 -08:00
wangxiyuan fa9ee08121 [Misc] Set default backend to SDPA for get_vit_attn_backend (#12235) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2025-01-21 11:52:11 -08:00
Cyrus Leung f2e9f2a3be [Misc] Remove redundant TypeVar from base model (#12248) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-21 08:40:39 +00:00
Jee Jee Li 1f1542afa9 [Misc]Add BNB quantization for PaliGemmaForConditionalGeneration (#12237) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2025-01-21 07:49:08 +00:00
Cyrus Leung 96912550c8 [Misc] Rename MultiModalInputsV2 -> MultiModalInputs (#12244) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-21 07:31:19 +00:00
Cyrus Leung b37d82791e [Model] Upgrade Aria to transformers 4.48 (#12203) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-20 17:58:48 +08:00
Isotr0py 83609791d2 [Model] Add Qwen2 PRM model support (#12202) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-01-20 14:59:46 +08:00
Martin Gleizeandmgleize user bbe5f9de7d [Model] Support for fairseq2 Llama (#11442) 
Signed-off-by: Martin Gleize <mgleize@meta.com>
Co-authored-by: mgleize user <mgleize@a100-st-p4de24xlarge-4.fair-a100.hpcaas>
 2025-01-19 10:40:40 -08:00
Roger WangimkeroDarkLight1337
 		81763c58a0 [V1] Add V1 support of Qwen2-VL (#12128) 
Signed-off-by: Roger Wang <ywang@roblox.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: imkero <kerorek@outlook.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-19 19:52:13 +08:00
Isotr0py edaae198e7 [Misc] Add BNB support to GLM4-V model (#12184) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-01-19 19:49:22 +08:00
Cyrus Leung 630eb5b5ce [Bugfix] Fix multi-modal processors for transformers 4.48 (#12187) 2025-01-18 19:16:34 -08:00
Isotr0py 02798ecabe [Model] Port deepseek-vl2 processor, remove dependency (#12169) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-01-18 13:59:39 +08:00
Isotr0py 62b06ba23d [Model] Add support for deepseek-vl2-tiny model (#12068) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-01-16 17:14:48 +00:00
Roger Wang 874f7c292a [Bugfix] Fix max image feature size for Llava-one-vision (#12104) 
Signed-off-by: Roger Wang <ywang@roblox.com>
 2025-01-16 14:54:06 +00:00
Isotr0py dd7c9ad870 [Bugfix] Remove hardcoded head_size=256 for Deepseek v2 and v3 (#12067) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-01-16 10:11:54 +00:00
Michael Goin 9aa1519f08 Various cosmetic/comment fixes (#12089) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2025-01-16 09:59:06 +00:00
kewang-xlnxkewang2Michael Goin
 		de0526f668 [Misc][Quark] Upstream Quark format to VLLM (#10765) 
Signed-off-by: kewang-xlnx <kewang@xilinx.com>
Signed-off-by: kewang2 <kewang2@amd.com>
Co-authored-by: kewang2 <kewang2@amd.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2025-01-15 11:05:15 -05:00
RunningLeon 97eb97b5a4 [Model]: Support internlm3 (#12037) 2025-01-15 11:35:17 +00:00
Jee Jee Li 42f5e7c52a [Kernel] Support MulAndSilu (#11624) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2025-01-15 02:29:53 +00:00
Jee Jee Li a3a3ee4e6f [Misc] Merge bitsandbytes_stacked_params_mapping and packed_modules_mapping (#11924) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2025-01-15 07:49:49 +08:00
Concurrensee cf6bbcb493 [Misc] Fix Deepseek V2 fp8 kv-scale remapping (#11947) 
Signed-off-by: Yida Wu <yidawu@alumni.cmu.edu>
 2025-01-12 23:05:06 -08:00
Sungjae Lee 80ea3af1a0 [CI][Spec Decode] fix: broken test for EAGLE model (#11972) 
Signed-off-by: Sungjae Lee <33976427+llsj14@users.noreply.github.com>
 2025-01-13 06:50:35 +00:00
Siyuan Li 9dd02d85ca [Bug] Fix usage of .transpose() and .view() consecutively. (#11979) 2025-01-13 06:24:10 +00:00
Isotr0pyandCyrus Leung d14e98d924 [Model] Support GGUF models newly added in transformers 4.46.0 (#9685) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2025-01-13 00:13:44 +00:00
Isotr0pyandCyrus Leung f967e51f38 [Model] Initialize support for Deepseek-VL2 models (#11578) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2025-01-12 00:17:24 -08:00
sixgodandIsotr0py 4b657d3292 [Model] Add cogagent model support vLLM (#11742) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
 2025-01-11 19:05:56 +00:00
Nicolò Lucchesi d697dc01b4 [Bugfix] Fix RobertaModel loading (#11940) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-01-11 14:05:09 +00:00
Sungjae Lee 2118d0565c [Bugfix][SpecDecode] Adjust Eagle model architecture to align with intended design (#11672) 
Signed-off-by: Sungjae Lee <33976427+llsj14@users.noreply.github.com>
 2025-01-10 20:49:38 -08:00
Travis Johnson d45cbe70f5 [Bugfix] Check that number of images matches number of <|image|> tokens with mllama (#11939) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
 2025-01-10 23:26:00 +00:00
Isotr0py 46fa98ccad [Misc] Clean up debug code in Deepseek-V3 (#11930) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-01-10 19:19:15 +00:00
wangxiyuanandyoukaichao 20410b2fda [platform] support custom torch.compile backend key (#11318) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
 2025-01-10 23:46:51 +08:00
Harry Mellor d85c47d6ad Replace "online inference" with "online serving" (#11923) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-01-10 12:05:56 +00:00