iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
14,386 Commits 3 Branches 0 Tags
fa6a6be51978bd4b49ba0da17039e60f96dc5b13
Commit Graph
162 Commits
Author SHA1 Message Date
JartXandtjtanaa 3b11b26b50 [FIXBUG ] Allow disabling rocm_aiter_fa backend for ROCm GPUs not compatible with AITER (#22795) 
Signed-off-by: JartX <sagformas@epdcenter.es>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
 2025-08-20 09:08:29 -07:00
Woosuk Kwon d6d13bd49e [Misc] Add max_seq_len to CommonAttentionMetadata (#23216) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-08-20 09:05:29 -07:00
xyxinyangandzhouchong 7cd17e22d7 [Model][V1] Support Ernie MTP (#22169) 
Signed-off-by: zhouchong <zhouchong03@baidu.com>
Co-authored-by: zhouchong <zhouchong03@baidu.com>
 2025-08-20 20:41:55 +08:00
Woosuk Kwon c9b38be8aa [Spec Decode] Make propose_draft_token_ids non-blocking for lower TTFT (#23041) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-08-18 17:20:38 -07:00
Jialin Ouyang 31a500c86f [Core] [N-gram SD Optimization][1/n] Propose tokens with a single KMP (#22437) 
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2025-08-13 14:44:06 -07:00
Giancarlo Delfin d94e3026de [V1] Add tree drafting tests for eagle spec decoding (#22705) 
Signed-off-by: Giancarlo Delfin <gdelfin@meta.com>
 2025-08-13 04:11:28 -07:00
TJian 1ee5ead5f8 [ROCm] [V1] [SpecDec] Enable Speculative Decoding on ROCm V1 Engine (#21496) 
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
 2025-08-07 19:13:17 -07:00
Lucas Wilkinson 1dc8a70b6d [Attention] Support multiple attention metadata builders per kv_cache_spec + proper local attention no hybrid kv cache fix (#21588) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2025-08-06 18:40:52 -07:00
Giancarlo Delfin aa7012eb6d Add tree attention backend for v1 (part 1) (#20401) 
Signed-off-by: Giancarlo Delfin <gdelfin@meta.com>
 2025-08-03 22:13:26 -07:00
zhiweizandRoger Wang 9e0726e5bf [Meta] Official Eagle mm support, first enablement on llama4 (#20788) 
Signed-off-by: morgendave <morgendave@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.me>
 2025-07-31 10:35:07 -07:00
Maximilien de BayserandRussell Bryant 1cd6eaba54 Support encoder-only models without KV-Cache (#21270) 
Signed-off-by: Max de Bayser <maxdebayser@gmail.com>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
 2025-07-26 21:09:52 +08:00
Lucas Wilkinson 76b494444f [Attention] Refactor attention metadata builder interface (#20466) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2025-07-17 04:44:25 +00:00
Cyrus Leung 9fb52e523a [V1] Support any head size for FlexAttention backend (#20467) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-07-06 09:54:36 -07:00
afeldman-nmandNick Hill 48fb076cbc [V1] LogitsProcessor programming model (#16728) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Andrew Feldman <afeldman@neuralmagic.com>
Signed-off-by: Andrew Feldman <afeldman@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-07-02 09:10:42 -07:00
cjackal 8359f4c8d8 [V1][Speculative Decoding] Fix DeepSeek MTP (#20022) 
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com>
 2025-06-25 08:41:02 -07:00
Luka Govedič 3597b06a4f [CUDA] Enable full cudagraph for FlashMLA (#18581) 
Signed-off-by: luka <luka@neuralmagic.com>
 2025-06-13 18:12:26 +00:00
youkaichao d70bc7c029 [torch.compile] reorganize the cache directory to support compiling multiple models (#19064) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-06-13 15:23:25 +08:00
qizixi c68698b326 [Bugfix] Fix EAGLE vocab embedding for multimodal target model (#19570) 
Signed-off-by: qizixi <qizixi@meta.com>
 2025-06-12 23:09:19 -04:00
Benjamin Chislett 3465b87ef8 [Bugfix] Fix EAGLE vocab embedding construction for Llama 70B (#19033) 
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
 2025-06-05 19:10:08 -07:00
Simon Mo 02f0c7b220 [Misc] Add SPDX-FileCopyrightText (#19100) 
Signed-off-by: simon-mo <simon.mo@hey.com>
 2025-06-03 11:20:17 -07:00
RonaldBXu 5e13c07d00 [V1] [Bugfix] eagle bugfix and enable correct lm_head for multimodal (2) (#18781) 
Signed-off-by: Ronald Xu <ronaldxu@amazon.com>
 2025-05-28 05:09:14 +00:00
Mark McLoughlin 06a0338015 [V1][Metrics] Add API for accessing in-memory Prometheus metrics (#17010) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-05-27 09:37:06 +00:00
qizixi c1e4a4052d [V1][Spec Decode] Support multi-layer eagle draft model (#18030) 
Signed-off-by: qizixi <qizixi@meta.com>
 2025-05-24 09:45:34 +00:00
qizixi d55e446d13 [V1][Spec Decode] Small refactors to improve eagle bookkeeping performance (#18424) 
Signed-off-by: qizixi <qizixi@meta.com>
 2025-05-24 06:51:22 +00:00
Jiayi YaoandRui Qiao 2628a69e35 [V1] Support Deepseek MTP (#18435) 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Signed-off-by: YaoJiayi <120040070@link.cuhk.edu.cn>
Co-authored-by: Rui Qiao <ruisearch42@gmail.com>
 2025-05-23 10:26:28 -07:00
Cyrus Leung fbb13a2c15 Revert "[V1] [Bugfix] eagle bugfix and enable correct lm_head for multimodal (#18034)" (#18600) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-05-23 02:18:22 -07:00
RonaldBXu 4c611348a7 [V1] [Bugfix] eagle bugfix and enable correct lm_head for multimodal (#18034) 
Signed-off-by: Ronald Xu <ronaldxu@amazon.com>
 2025-05-23 00:37:18 -07:00
Mark McLoughlin c6b636f9fb [V1][Spec Decoding] Use model_loader.get_model() to load models (#18273) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-05-23 02:05:44 +00:00
Ekagra Ranjan a04720bc36 [V1][Spec Decode][Bugfix] Load quantize weights for EAGLE (#18290) 2025-05-22 15:17:33 -07:00
Seiji Eicher 541817670c [Misc] Add Ray Prometheus logger to V1 (#17925) 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
 2025-05-16 01:02:42 -07:00
Sky Leeandlisiqi23 f4937a51c1 [Model] vLLM v1 supports Medusa (#17956) 
Signed-off-by: lisiqi23 <lisiqi23@xiaomi.com>
Signed-off-by: skylee-01 <497627264@qq.com>
Co-authored-by: lisiqi23 <lisiqi23@xiaomi.com>
 2025-05-15 21:05:31 -07:00
Ekagra RanjanrootWoosuk Kwon
 		418d2f8bfb [V1][Spec Decode] Share input embedding of target model with EAGLE draft model to free ~1GB for llama 3 model (#17326) 
Co-authored-by: root <root@ekagra-8xh100.us-east5-a.c.serving-efficiency-poc.internal>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-05-14 12:31:46 -07:00
wwl2755 dc9905368d [V1][Spec Decode] Eagle unit tests (#17350) 
Signed-off-by: wwl2755 <wangwenlong2755@gmail.com>
 2025-05-12 23:01:17 +00:00
Mark McLoughlin 7042cc96b0 [V1][Spec Decoding] Log accumulated metrics after system goes idle (#17913) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-05-09 18:23:07 -07:00
Mark McLoughlin 7e3571134f [V1][Spec Decoding] Include bonus tokens in mean acceptance length (#17908) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-05-09 13:32:36 -07:00
Jee Jee Li 822de7fb94 [Misc] Split model loader (#17712) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2025-05-07 12:42:26 +08:00
Woosuk Kwon 8d84d836d1 [BugFix][Spec Decode] Fix hidden size mismatch between target and eagle head (#17740) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-05-06 19:51:26 -07:00
Chen Zhang cba31c47c4 [v1] AttentionMetadata for each layer (#17394) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-05-06 07:58:37 -07:00
Mengqing Cao f9bc5a0693 [Bugfix] Fix triton import with local TritonPlaceholder (#17446) 
Signed-off-by: Mengqing Cao <cmq0113@163.com>
 2025-05-06 17:53:09 +08:00
qizixi 39c0813a7f [V1][Spec Decode] Apply torch.compile & cudagraph to EAGLE3 (#17504) 
Signed-off-by: qizixi <qizixi@meta.com>
 2025-05-01 16:19:30 -07:00
Bryan Lu 70788bdbdc [V1][Spec Decode] Apply torch.compile & cudagraph to EAGLE (#17211) 
Signed-off-by: Bryan Lu <yuzhelu@amazon.com>
 2025-04-29 21:10:00 +00:00
Ekagra Ranjan e136000595 [V1][Spec Decode] Make Eagle model arch config driven (#17323) 2025-04-29 10:22:02 +08:00
Benjamin ChislettandBryan Lu a0e619e62a [V1][Spec Decode] EAGLE-3 Support (#16937) 
Signed-off-by: Bryan Lu <yuzhelu@amazon.com>
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Co-authored-by: Bryan Lu <yuzhelu@amazon.com>
 2025-04-25 15:43:07 -07:00
Mark McLoughlin 340d7b1b21 [V1][Spec Decoding] Add num_drafts and num_accepted_tokens_per_position metrics (#16665) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-04-24 08:57:40 -07:00
Woosuk Kwon 41fb013d29 [V1][Spec Decode] Always use argmax for sampling draft tokens (#16899) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-04-23 14:57:43 -07:00
Woosuk Kwon 1311913f55 [BugFix][Spec Decode] No in-place update to draft probs (#16952) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-04-21 19:54:19 -07:00
Woosuk Kwon 3a0fba5cf4 [V1][Spec Decode] Handle draft tokens beyond max_model_len (#16087) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-04-21 12:38:50 -07:00
Russell Bryant 9dbf7a2dc1 [V1] Remove log noise when idle (#16735) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-04-16 21:34:08 -07:00
Lily Liu e8224f3dca [V1][Spec Decode] Eagle Model loading (#16035) 
Signed-off-by: LiuXiaoxuanPKU <lilyliupku@gmail.com>
 2025-04-10 11:21:48 -07:00
Woosuk Kwon 63375f0cdb [V1][Spec Decode] Update N-gram Proposer Interface (#15750) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-04-04 16:32:54 -07:00