iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
14,386 Commits 3 Branches 0 Tags
fa6a6be51978bd4b49ba0da17039e60f96dc5b13
Commit Graph
10211 Commits
Author SHA1 Message Date
Siyuan Li 9dd02d85ca [Bug] Fix usage of .transpose() and .view() consecutively. (#11979) 2025-01-13 06:24:10 +00:00
Robert Shaw 619ae268c3 [V1] [2/n] Logging and Metrics - OutputProcessor Abstraction (#11973) 
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
 2025-01-13 04:54:10 +00:00
Isotr0pyandCyrus Leung d14e98d924 [Model] Support GGUF models newly added in transformers 4.46.0 (#9685) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2025-01-13 00:13:44 +00:00
Robert Shaw 9597a095f2 [V1][Core][1/n] Logging and Metrics (#11962) 
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
 2025-01-12 21:02:02 +00:00
Avshalom Manevich 263a870ee1 [Hardware][TPU] workaround fix for MoE on TPU (#11764) 2025-01-12 10:53:51 -05:00
Akshat TripathiOleg MosalovJee Jee LiIsotr0py
 		8bddb73512 [Hardware][CPU] Multi-LoRA implementation for the CPU backend (#11100) 
Signed-off-by: Akshat Tripathi <akshat@krai.ai>
Signed-off-by: Oleg Mosalov <oleg@krai.ai>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Oleg Mosalov <oleg@krai.ai>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
 2025-01-12 13:01:52 +00:00
Isotr0pyandCyrus Leung f967e51f38 [Model] Initialize support for Deepseek-VL2 models (#11578) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2025-01-12 00:17:24 -08:00
Roger Wang b25cfab9a0 [V1] Avoid sending text prompt to core engine (#11963) 
Signed-off-by: Roger Wang <ywang@roblox.com>
 2025-01-12 06:36:38 +00:00
sixgodandIsotr0py 4b657d3292 [Model] Add cogagent model support vLLM (#11742) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
 2025-01-11 19:05:56 +00:00
Nicolò Lucchesi d697dc01b4 [Bugfix] Fix RobertaModel loading (#11940) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-01-11 14:05:09 +00:00
shaochangxuandshaochangxu.scx c32a7c7c0c [Bugfix] fused_experts_impl wrong compute type for float32 (#11921) 
Signed-off-by: shaochangxu.scx <shaochangxu.scx@antgroup.com>
Co-authored-by: shaochangxu.scx <shaochangxu.scx@antgroup.com>
 2025-01-11 13:49:39 +08:00
Sungjae Lee 2118d0565c [Bugfix][SpecDecode] Adjust Eagle model architecture to align with intended design (#11672) 
Signed-off-by: Sungjae Lee <33976427+llsj14@users.noreply.github.com>
 2025-01-10 20:49:38 -08:00
Fred Reiss c9f09a4fe8 [mypy] Fix mypy warnings in api_server.py (#11941) 
Signed-off-by: Fred Reiss <frreiss@us.ibm.com>
 2025-01-11 01:04:58 +00:00
Travis Johnson d45cbe70f5 [Bugfix] Check that number of images matches number of <|image|> tokens with mllama (#11939) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
 2025-01-10 23:26:00 +00:00
Isotr0py 46fa98ccad [Misc] Clean up debug code in Deepseek-V3 (#11930) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-01-10 19:19:15 +00:00
Li, Jiang aa1e77a19c [Hardware][CPU] Support MOE models on x86 CPU (#11831) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2025-01-10 11:07:58 -05:00
Kuntai Du f33e033e27 [Docs] Fix docstring in get_ip function (#11932) 
Signed-off-by: Kuntai Du <kuntai@uchicago.edu>
 2025-01-10 23:51:02 +08:00
wangxiyuanandyoukaichao 20410b2fda [platform] support custom torch.compile backend key (#11318) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
 2025-01-10 23:46:51 +08:00
Cyrus Leung 12664ddda5 [Doc] [1/N] Initial guide for merged multi-modal processor (#11925) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-10 14:30:25 +00:00
Harry Mellor d85c47d6ad Replace "online inference" with "online serving" (#11923) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-01-10 12:05:56 +00:00
wangxiyuan ef725feafc [platform] support pytorch custom op pluggable (#11328) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2025-01-10 10:02:38 +00:00
cennnandyoukaichao d907be7dc7 [misc] remove python function call for custom activation op (#11885) 
Co-authored-by: youkaichao <youkaichao@gmail.com>
 2025-01-10 17:18:25 +08:00
youkaichao d53575a5f0 [ci] fix gh200 tests (#11919) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-01-10 16:25:17 +08:00
Kunshang Ji 61af633256 [BUGFIX] Fix UnspecifiedPlatform package name (#11916) 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
 2025-01-10 16:20:46 +08:00
Joe RundeandJee Jee Li ac2f3f7fee [Bugfix] Validate lora adapters to avoid crashing server (#11727) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
 2025-01-10 15:56:36 +08:00
Chen Zhang cf5f000d21 [torch.compile] Hide KV cache behind torch.compile boundary (#11677) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-01-10 13:14:42 +08:00
Cyrus Leung b844b99ad3 [VLM] Enable tokenized inputs for merged multi-modal processor (#11900) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-10 03:24:00 +00:00
wangxiyuanandMengqing Cao 405eb8e396 [platform] Allow platform specify attention backend (#11609) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Mengqing Cao <cmq0113@163.com>
Co-authored-by: Mengqing Cao <cmq0113@163.com>
 2025-01-09 21:46:50 +08:00
Cyrus Leung 65097ca0af [Doc] Add model development API Reference (#11884) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-09 09:43:40 +00:00
Ye (Charlotte) Qiandyeq 1d967acb45 [Bugfix] fix beam search input errors and latency benchmark script (#11875) 
Signed-off-by: Ye Qi <yeq@meta.com>
Co-authored-by: yeq <yeq@devgpu004.lla3.facebook.com>
 2025-01-09 17:36:39 +08:00
Cyrus Leung 0bd1ff4346 [Bugfix] Override dunder methods of placeholder modules (#11882) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-09 09:02:53 +00:00
youkaichao 310aca88c9 [perf]fix current stream (#11870) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-01-09 07:18:21 +00:00
Cyrus LeungandMaxime Fournioux d848800e88 [Misc] Move print_*_once from utils to logger (#11298) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>
Co-authored-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>
 2025-01-09 12:48:12 +08:00
Maximilien de Bayser 1fe554bac3 treat do_lower_case in the same way as the sentence-transformers library (#11815) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
 2025-01-09 11:05:43 +08:00
rasmith 526de822d5 [Kernel][Triton][AMD] Use block size heuristic for avg 2.8x speedup for int8 models (#11698) 
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
 2025-01-08 20:23:15 +00:00
Robert ShawandWoosuk Kwon 56fe4c297c [TPU][Quantization] TPU W8A8 (#11785) 
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-01-08 19:33:29 +00:00
WangErXiao 47de8821d3 [Misc]add some explanations for BlockHashType (#11847) 2025-01-08 18:21:30 +00:00
Cyrus Leung 5984499e47 [Doc] Expand Multimodal API Reference (#11852) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-08 17:14:14 +00:00
Cyrus Leung ca47e176af [Misc] Move some model utils into vision file (#11848) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-08 17:04:46 +00:00
Yan Ma 78f4590b60 [Bugfix][XPU] fix silu_and_mul (#11823) 
Signed-off-by: yan ma <yan.ma@intel.com>
 2025-01-09 00:11:50 +08:00
Li, Jiang 2f7024987e [CI/Build][Bugfix] Fix CPU CI image clean up (#11836) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2025-01-08 15:18:28 +00:00
Cyrus Leung 6cd40a5bfe [Doc][4/N] Reorganize API Reference (#11843) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-08 21:34:44 +08:00
Harry Mellor aba8d6ee00 [Doc] Move examples into categories (#11840) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-01-08 13:09:53 +00:00
Cyrus Leung 2a0596bc48 [VLM] Reorganize profiling/processing-related code (#11812) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-08 18:59:58 +08:00
youkaichao f12141170a [torch.compile] consider relevant code in compilation cache (#11614) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-01-08 10:46:43 +00:00
Wallas HenriqueandMichael Goin cfd3219f58 [Hardware][Apple] Native support for macOS Apple Silicon (#11696) 
Signed-off-by: Wallas Santos <wallashss@ibm.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2025-01-08 16:35:49 +08:00
youkaichaoandCyrus Leung 889e662eae [misc] improve memory profiling (#11809) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2025-01-08 06:36:03 +00:00
Cyrus Leung ef68eb28d8 [Bug] Fix pickling of ModelConfig when RunAI Model Streamer is used (#11825) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-08 13:40:09 +08:00
Jee Jee Li f645eb6954 [Bugfix] Add checks for LoRA and CPU offload (#11810) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2025-01-08 13:08:48 +08:00
WangErXiao dc71af0a71 Remove the duplicate imports of MultiModalKwargs and PlaceholderRange… (#11824) 2025-01-08 04:09:25 +00:00