iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
14,386 Commits 3 Branches 0 Tags
fa6a6be51978bd4b49ba0da17039e60f96dc5b13
Commit Graph
2339 Commits
Author SHA1 Message Date
lkchen d4629dc43f [Misc] Add __str__ for RequestStatus (#19780) 
Signed-off-by: Linkun Chen <github@lkchen.net>
 2025-06-18 03:03:01 +00:00
Charlie Fu a44b1c951d [Feature][ROCm] Add full graph capture support for TritonAttentionBackend (#19158) 
Signed-off-by: charlifu <charlifu@amd.com>
 2025-06-17 17:03:06 -04:00
CYJiang 5a1c2e15d8 [Mis] remove duplicate engine status checks (#19647) 
Signed-off-by: googs1025 <googs1025@gmail.com>
 2025-06-17 08:17:38 -07:00
Nicolò Lucchesi 4c8f64faa7 [V1][Kernel] Flashinfer HND KV cache layout (#19280) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-06-17 09:09:22 -04:00
Driss Guessous ddfed314f9 Fixes IMA for TP w/ flex-attention (#19712) 
Signed-off-by: drisspg <drisspguessous@gmail.com>
 2025-06-17 04:01:50 +00:00
Russell Bryant 90f9c2eb5c [V1] Change return type on get_multimodal_embeddings() (#19446) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-06-16 13:32:15 -04:00
Isotr0py 1173804dca [Bugfix] Fix TP inference for Flex attention backend (#19657) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-06-16 11:21:37 +00:00
Nick Hill ee35e96ac3 [BugFix] Don't catch BaseException when dumping execute_model errors (#19626) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-06-16 11:01:08 +00:00
Chengji Yaoandmgoin a77aea59fd [TPU] support attention head dim smaller than 128 (#19620) 
Signed-off-by: Chengji Yao <chengjiyao@google.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2025-06-16 06:40:53 +00:00
22quinn 0b73736a0d [Kernel] Raise verbose error and consolidate num_heads/num_kv_heads divisibility check (#19339) 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-06-15 13:43:48 +08:00
Saheli Bhattacharjee d1e34cc9ac [V1][Metrics] Deprecate metrics with gpu_ prefix for non GPU specific metrics. (#18354) 
Signed-off-by: Saheli Bhattacharjee <saheli@krai.ai>
 2025-06-14 11:07:36 +08:00
Nick Hill bd517eb9fe [BugFix] Fix DP Coordinator incorrect debug log message (#19624) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-06-14 00:18:03 +00:00
Luka Govedič 3597b06a4f [CUDA] Enable full cudagraph for FlashMLA (#18581) 
Signed-off-by: luka <luka@neuralmagic.com>
 2025-06-13 18:12:26 +00:00
Nick Hill 7e8d97dd3f [BugFix] Honor enable_caching in connector-delayed kvcache load case (#19435) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-06-13 09:46:32 +00:00
youkaichao d70bc7c029 [torch.compile] reorganize the cache directory to support compiling multiple models (#19064) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-06-13 15:23:25 +08:00
汪志鹏 cefdb9962d [Fix] The zip function in Python 3.9 does not have the strict argument (#19549) 
Signed-off-by: 汪志鹏 <wangzhipeng628@gmail.com>
 2025-06-13 14:57:48 +08:00
Li, Jiang 6458721108 [CPU] Refine default config for the CPU backend (#19539) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2025-06-13 13:27:39 +08:00
qizixi c68698b326 [Bugfix] Fix EAGLE vocab embedding for multimodal target model (#19570) 
Signed-off-by: qizixi <qizixi@meta.com>
 2025-06-12 23:09:19 -04:00
Russell Bryant c57bb199b3 [V1] Resolve failed concurrent structured output requests (#19565) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-06-12 23:30:09 +00:00
Luka GovedičandSage Moore f98548b9da [torch.compile][ROCm] Fuse quantization onto attention using a torch.compile pass (#16756) 
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Co-authored-by: Sage Moore <sage@neuralmagic.com>
 2025-06-12 08:31:04 -07:00
jmswen c9280e6346 [Bugfix] Respect num-gpu-blocks-override in v1 (#19503) 
Signed-off-by: Jon Swenson <jmswen@gmail.com>
 2025-06-12 11:00:23 +00:00
Nick Hill d5bdf899e4 [BugFix] Work-around incremental detokenization edge case error (#19449) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-06-12 06:43:20 +00:00
Robert Shaw 97a9465bbc [UX] Add Feedback During CUDAGraph Capture (#19501) 
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
 2025-06-11 21:09:05 +00:00
Rachel Guo 467bef18a3 [BugFix][FlashInfer] Fix attention backend interface mismatch with unexpected keyword use_irope (#19134) 
Signed-off-by: Yunqiu Guo <guorachel@meta.com>
 2025-06-10 16:48:51 +08:00
Louie TsaiandLi, Jiang <bigpyj64@gmail.com> 9368cc90b2 Automatically bind CPU OMP Threads of a rank to CPU ids of a NUMA node. (#17930) 
Signed-off-by: Tsai, Louie <louie.tsai@intel.com>
Co-authored-by: Li, Jiang <bigpyj64@gmail.com>
 2025-06-10 06:22:05 +00:00
Lukas Geiger 319cb1e351 [Core] Batch multi modal input using pinned memory (#19169) 
Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>
 2025-06-10 13:44:59 +08:00
Nick Hill 646d62f636 [Core] Use tuple for kv cache group block ids (#19175) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-06-10 07:01:17 +02:00
XiongfeiWei 9af6d22e4c Use xla flag to improve the quantized model performance (#19303) 
Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>
 2025-06-10 01:28:45 +00:00
Ye (Charlotte) Qi cc867be19c [V1] Reuse V0's memory_profiling util for gpu worker memory profiling (#19312) 
Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>
 2025-06-10 08:40:01 +08:00
Pavani Majety 8058c91108 [HOT-FIX] Add kv_sharing_target_layer_name argument to cutlass_mla backend (#19374) 
Signed-off-by: Pavani Majety <pmajety@nvidia.com>
 2025-06-09 19:00:07 -04:00
Varun Sundar RabindranathandVarun 5cf2daea9a [Misc] Fixes and Optimizations for DeepEP + DeepGEMM combination. (#19298) 
Signed-off-by: Varun <vsundarr@redhat.com>
Co-authored-by: Varun <vsundarr@redhat.com>
 2025-06-09 10:50:39 -04:00
Yinghai Lu 770e5dcdb8 [full_graph] Fix query_start_loc padding (#19321) 
Signed-off-by: Yinghai Lu <yinghai@thinkingmachines.ai>
 2025-06-09 21:32:56 +08:00
jennyyyyzhenandyZhen cda10fa3e2 [Multi Modal] Add an env var for message queue max chunk bytes (#19242) 
Signed-off-by: yZhen <yZhen@fb.com>
Co-authored-by: yZhen <yZhen@fb.com>
 2025-06-08 21:39:12 +08:00
Chauncey d77f7fb871 [Bugfix]: Fix TypeError: 'float' object cannot be interpreted as an integer (#19283) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2025-06-08 08:16:31 +08:00
Luka Govedič 2d8476e465 [BugFix][V1] Fix memory profiling bug (#18974) 
Signed-off-by: luka <luka@neuralmagic.com>
 2025-06-07 10:34:51 -07:00
Driss Guessous cf02f9b283 Add FlexAttention to V1 (#16078) 
Signed-off-by: drisspg <drisspguessous@gmail.com>
 2025-06-06 21:58:55 -07:00
Nick Hill 46ecc57973 [BugFix] Fix tpu_model_runner block_id concatenation (#19228) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-06-06 16:28:17 -07:00
Nicolò LucchesiandNick Hill b6a3a9f76d [Core] Fix abrupt request abort (#18485) 
Signed-off-by: nicklucche <nlucches@redhat.com>
Signed-off-by: Nick Hill <nhill@redhat.com>

Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-06-06 16:27:59 -07:00
Adolfo VictoriaandAdolfo Victoria ca27f0f9c1 [Bugfix][Core] Update cancellation logic in generate() to handle Generator exits (#19225) 
Co-authored-by: Adolfo Victoria <adovi@meta.com>
 2025-06-06 20:17:54 +00:00
Nick Hill aad30bd306 [BugFix] Fix MultiConnector test after HMA changes (#19291) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-06-06 20:16:24 +00:00
Yu Guo 8267f9916f improve logits bias (#19041) 2025-06-06 19:59:25 +08:00
jmswen 7353492a47 [Core] Raise when non-multi-instance DP clients target a DP rank (#19227) 
Signed-off-by: Jon Swenson <jmswen@gmail.com>
 2025-06-06 19:03:01 +08:00
Nick Hill 65c69444b1 [Docs] Improve V1 KVConnector interface documentation (#19172) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-06-06 16:22:45 +08:00
Chengji Yao 0d49483ea9 [TPU] fix kv cache dtype in model runner (#19244) 
Signed-off-by: Chengji Yao <chengjiyao@google.com>
 2025-06-06 16:20:16 +08:00
Jinghui Zhangandjinghui 90b78ec5f9 [v1][P/D] Fix a edge case in kv cache schedule (#19182) 
Co-authored-by: jinghui <jinghui@fb.com>
 2025-06-05 23:32:55 -07:00
Chengji Yao b61dc5f972 [TPU] update torch_xla pin (#19231) 
Signed-off-by: Chengji Yao <chengjiyao@google.com>
 2025-06-06 04:27:38 +00:00
Chen Zhang f8a1a2d108 [v1] Hybrid Memory Allocator (#17996) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-06-05 20:47:09 -07:00
Benjamin Chislett 3465b87ef8 [Bugfix] Fix EAGLE vocab embedding construction for Llama 70B (#19033) 
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
 2025-06-05 19:10:08 -07:00
vllmellm 18093084be [Misc] Remove unnecessary fallback to prefill-decode attention (#19138) 
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
 2025-06-05 16:08:26 +08:00
Woosuk Kwon af7fc84fd2 [BugFix][Minor] Fix full cuda graph bug when max_num_seqs < 512 (#19171) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-06-05 13:41:25 +08:00