iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
14,386 Commits 3 Branches 0 Tags
fa6a6be51978bd4b49ba0da17039e60f96dc5b13
Commit Graph
1000 Commits
Author SHA1 Message Date
Vlad Tiberiu Mihailescu 2e3e3c86dc Export NaNs in logits to scheduler_stats if output is corrupted (#18777) 
Signed-off-by: Vlad Mihailescu <vtmihailescu@gmail.com>
 2025-06-20 22:47:16 +08:00
Maximilien de Bayserand22quinn 799397ee4f Support embedding models in V1 (#16188) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Signed-off-by: Max de Bayser <maxdebayser@gmail.com>
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-06-18 21:36:33 -07:00
Richard Zou ed33349738 [BugFix] Fix use_cudagraph=False (#19612) 
Signed-off-by: Richard Zou <zou3519@gmail.com>
 2025-06-19 08:23:12 +08:00
Chen Zhang a89209b78d [v1] Support mamba2 (#19327) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-06-18 20:34:15 +00:00
afeldman-nm 19a53b2783 [V1] Decouple GPU and TPU InputBatch (#19778) 
Signed-off-by: Andrew Feldman <afeldman@redhat.com>
 2025-06-18 06:38:13 +00:00
Russell Bryant 90f9c2eb5c [V1] Change return type on get_multimodal_embeddings() (#19446) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-06-16 13:32:15 -04:00
Isotr0py 1173804dca [Bugfix] Fix TP inference for Flex attention backend (#19657) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-06-16 11:21:37 +00:00
Luka Govedič 3597b06a4f [CUDA] Enable full cudagraph for FlashMLA (#18581) 
Signed-off-by: luka <luka@neuralmagic.com>
 2025-06-13 18:12:26 +00:00
汪志鹏 cefdb9962d [Fix] The zip function in Python 3.9 does not have the strict argument (#19549) 
Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>
 2025-06-13 14:57:48 +08:00
Li, Jiang 6458721108 [CPU] Refine default config for the CPU backend (#19539) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2025-06-13 13:27:39 +08:00
Russell Bryant c57bb199b3 [V1] Resolve failed concurrent structured output requests (#19565) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-06-12 23:30:09 +00:00
Robert Shaw 97a9465bbc [UX] Add Feedback During CUDAGraph Capture (#19501) 
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
 2025-06-11 21:09:05 +00:00
Louie TsaiandLi, Jiang <bigpyj64@gmail.com> 9368cc90b2 Automatically bind CPU OMP Threads of a rank to CPU ids of a NUMA node. (#17930) 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Co-authored-by: Li, Jiang <bigpyj64@gmail.com>
 2025-06-10 06:22:05 +00:00
Lukas Geiger 319cb1e351 [Core] Batch multi modal input using pinned memory (#19169) 
Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>
 2025-06-10 13:44:59 +08:00
Nick Hill 646d62f636 [Core] Use tuple for kv cache group block ids (#19175) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-06-10 07:01:17 +02:00
XiongfeiWei 9af6d22e4c Use xla flag to improve the quantized model performance (#19303) 
Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>
 2025-06-10 01:28:45 +00:00
Ye (Charlotte) Qi cc867be19c [V1] Reuse V0's memory_profiling util for gpu worker memory profiling (#19312) 
Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>
 2025-06-10 08:40:01 +08:00
Varun Sundar RabindranathandVarun 5cf2daea9a [Misc] Fixes and Optimizations for DeepEP + DeepGEMM combination. (#19298) 
Signed-off-by: Varun <vsundarr@redhat.com>
Co-authored-by: Varun <vsundarr@redhat.com>
 2025-06-09 10:50:39 -04:00
Yinghai Lu 770e5dcdb8 [full_graph] Fix query_start_loc padding (#19321) 
Signed-off-by: Yinghai Lu <yinghai@thinkingmachines.ai>
 2025-06-09 21:32:56 +08:00
Luka Govedič 2d8476e465 [BugFix][V1] Fix memory profiling bug (#18974) 
Signed-off-by: luka <luka@neuralmagic.com>
 2025-06-07 10:34:51 -07:00
Nick Hill 46ecc57973 [BugFix] Fix tpu_model_runner block_id concatenation (#19228) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-06-06 16:28:17 -07:00
Chengji Yao 0d49483ea9 [TPU] fix kv cache dtype in model runner (#19244) 
Signed-off-by: Chengji Yao <chengjiyao@google.com>
 2025-06-06 16:20:16 +08:00
Chengji Yao b61dc5f972 [TPU] update torch_xla pin (#19231) 
Signed-off-by: Chengji Yao <chengjiyao@google.com>
 2025-06-06 04:27:38 +00:00
Chen Zhang f8a1a2d108 [v1] Hybrid Memory Allocator (#17996) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-06-05 20:47:09 -07:00
Woosuk Kwon af7fc84fd2 [BugFix][Minor] Fix full cuda graph bug when max_num_seqs < 512 (#19171) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-06-05 13:41:25 +08:00
Woosuk Kwon b124e1085b [Bugfix] Fix FA3 full cuda graph correctness (#19106) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-06-03 23:10:15 -07:00
Lukas Geiger 1409ef9134 [Core] Cast multimodal input in hf processor (#18862) 
Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>
 2025-06-03 20:24:56 -07:00
Li, Jiang 4555143ea7 [CPU] V1 support for the CPU backend (#16441) 2025-06-03 18:43:01 -07:00
Chen Zhang 6cac54f4d1 [v1] Re-init input batch for multiple kv cache groups (#18654) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-06-03 21:41:36 +00:00
Yong Hoon Shin bdf13965ab [V1] Support cross-layer KV sharing (#18212) 
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
 2025-06-03 20:33:07 +00:00
Simon Mo 02f0c7b220 [Misc] Add SPDX-FileCopyrightText (#19100) 
Signed-off-by: simon-mo <simon.mo@hey.com>
 2025-06-03 11:20:17 -07:00
Siyuan LiuHossein SarsharChengji Yao
 		9112b443a0 [Hardware][TPU] Initial support of model parallelism with single worker using SPMD (#18011) 
Signed-off-by: Siyuan Liu <lsiyuan@google.com>
Co-authored-by: Hossein Sarshar <hossein.sarshar@gmail.com>
Co-authored-by: Chengji Yao <chengjiyao@google.com>
 2025-06-03 00:06:20 +00:00
22quinn 9760fd8f6a [Core] Support inplace model weights loading (#18745) 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-06-02 17:38:50 +08:00
zhrrrzhuhaoranTyler Michael Smith
 		d6fd3a33b8 [Misc] reuse num_tokens_across_dp of get_dp_padding to avoid unnecessary dp all reduce in set_forward_context (#18935) 
Signed-off-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
 2025-06-01 19:41:18 +00:00
Yu Guo 7782464a17 create util function for batched arange (#18937) 2025-05-31 13:50:38 +08:00
Carol Zheng fba02e3bd1 [Bugfix][TPU] Fix tpu model runner testcase failure (#18810) 
Signed-off-by: Carol Zheng <cazheng@google.com>
 2025-05-30 18:04:03 +08:00
Nicolò Lucchesi 32ce3cf7c9 [V1] Allocate kv_cache with stride order for V1 (#18775) 
Signed-off-by: nicklucche <nlucches@redhat.com>
 2025-05-29 17:54:16 +00:00
Varun Sundar RabindranathandVarun Sundar Rabindranath 7951d78738 [Core] Enable CUDA graphs for DP + All2All kernels (#18724) 
Signed-off-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
 2025-05-28 22:55:30 +00:00
Akshat TripathiChengji YaoxihajunJorge de FreitasJorge de Freitas
 		643622ba46 [Hardware][TPU][V1] Multi-LoRA Optimisations for the V1 TPU backend (#15655) 
Signed-off-by: Akshat Tripathi <akshat@krai.ai>
Signed-off-by: Chengji Yao <chengjiyao@google.com>
Signed-off-by: xihajun <junfan@krai.ai>
Signed-off-by: Jorge de Freitas <jorge.de-freitas22@imperial.ac.uk>
Signed-off-by: Jorge de Freitas <jorge@krai.ai>
Co-authored-by: Chengji Yao <chengjiyao@google.com>
Co-authored-by: xihajun <junfan@krai.ai>
Co-authored-by: Jorge de Freitas <jorge.de-freitas22@imperial.ac.uk>
Co-authored-by: Jorge de Freitas <jorge@krai.ai>
 2025-05-28 19:59:09 +00:00
Aaron Pham a09c7ca9f2 [Chore][Spec Decode] Update check NoneType instead of assigning variables (#18836) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
 2025-05-28 18:57:19 +00:00
Divakar Verma 774c5fde30 [V1] fix torch profiling for V1 offline scenarios (#18445) 
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
 2025-05-28 04:16:30 +00:00
Cyrus Leung 696259ca01 [Core] Automatically cast multi-modal input dtype (#18756) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-05-27 23:45:48 +08:00
Hyogeun Oh (오효근) a68e293cb9 [Doc] Convert Sphinx directives ( {class}, {meth}, {attr}, ...) to MkDocs format for better documentation linking (#18663) 
Signed-off-by: Zerohertz <ohg3417@gmail.com>
 2025-05-27 01:44:20 -07:00
qizixi c1e4a4052d [V1][Spec Decode] Support multi-layer eagle draft model (#18030) 
Signed-off-by: qizixi <qizixi@meta.com>
 2025-05-24 09:45:34 +00:00
qizixi d55e446d13 [V1][Spec Decode] Small refactors to improve eagle bookkeeping performance (#18424) 
Signed-off-by: qizixi <qizixi@meta.com>
 2025-05-24 06:51:22 +00:00
Jiayi YaoandRui Qiao 2628a69e35 [V1] Support Deepseek MTP (#18435) 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Signed-off-by: YaoJiayi <120040070@link.cuhk.edu.cn>
Co-authored-by: Rui Qiao <ruisearch42@gmail.com>
 2025-05-23 10:26:28 -07:00
Chen Zhang 6550114c9c [v1] Redo "Support multiple KV cache groups in GPU model runner (#17945)" (#18593) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-05-23 09:39:47 -07:00
youkaichao 6a7988c55b Refactor pplx init logic to make it modular (prepare for deepep) (#18200) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-05-23 23:43:43 +08:00
Harry Mellor 2edb533af2 Replace {func} with mkdocs style links (#18610) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-23 05:51:38 -07:00
Harry Mellor a1fe24d961 Migrate docs from Sphinx to MkDocs (#18145) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-23 02:09:53 -07:00