iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
14,683 Commits 3 Branches 0 Tags
42fadebecb79290ad722f33f3094de23b121f33d
Commit Graph
402 Commits
Author SHA1 Message Date
Vadim Gimpelson 4ff8c3c8f9 [BUGFIX][Mamba][Qwen3.5] Zero freed SSM cache blocks on GPU (#35219) 
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
 2026-03-10 03:32:20 -07:00
Nicolò Lucchesi 5b3ba94ab4 [Core][KVConnector] Support HMA+NixlConnector (#35758) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-03-06 08:51:21 +01:00
cong-or 57c84ff129 perf: add __slots__ to KVCacheBlock (#36164) 
Signed-off-by: cong-or <conchubhar.gannon@gmail.com>
 2026-03-05 22:04:09 -08:00
Jiayi YanandHarry Mellor 6a895197fa [Bugfix][CI] fix typos (#34934) 
Signed-off-by: 1195343015 <1195343015@qq.com>
Signed-off-by: Jiayi Yan <66017932+1195343015@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-05 17:05:46 +00:00
Or Ozeri 612e7729c2 [KVConnector] Scheduler: Fix num_computed_tokens after async KV load (#34616) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-03-05 14:25:15 +00:00
Qi Wang 6aa6ad8992 [BugFix] Fix implicit and incorrect assumption on ECConnector is_producer (#34783) 
Signed-off-by: Qi Wang <qiwa@nvidia.com>
 2026-03-04 15:01:30 +01:00
aykoppol 25e02647c2 [Core] Add optional flags to check for repetitive token patterns in engine output (#35451) 
Signed-off-by: aykoppol <aykoppol+git@gmail.com>
 2026-03-03 12:23:25 +08:00
Harry Mellor 01914445b0 Remove bc-lint (#35274) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-02-26 03:01:01 -08:00
Benjamin Chislett 682566b18e [Bug] Refactor max_num_batched_tokens to account for drafting (#34898) 
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
 2026-02-22 11:18:46 -05:00
Nick HillandCyrus Leung 820d7815eb [Core] Minor structured-output related scheduler optimization (#34765) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-02-21 01:38:28 -08:00
zhongdaor-nvNicolò LucchesiCyrus Leung
 		a0fe7ea2f0 [feat] Add per-block extra_keys to KV events (#33304) 
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-02-20 20:11:40 -08:00
Ekagra RanjanandNicolò Lucchesi cd81cdb399 [Scheduler][ASR] Fix CrossAttn blocks per-request for Variable length encoder inputs (#31058) 
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-02-16 11:08:44 +00:00
Thomas Parnell d5fe3f702c [Hybrid] Enable mamba prefix cache "align" mode with async scheduling (#33997) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
 2026-02-14 13:15:56 -08:00
Aaron Haogemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Nick Hill
 		dddbff4624 [Core] Move pause and resume functions into engine (#34125) 
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
Signed-off-by: Aaron Hao <ahao@anyscale.com>
Signed-off-by: hao-aaron <ahao@anyscale.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-02-13 00:15:10 -08:00
haosdent 4137c5dfa7 [Bug Fix] Fix MambaManager.cache_blocks() crash on null blocks in align mode (#34418) 
Signed-off-by: haosdent <haosdent@gmail.com>
 2026-02-13 00:13:22 -08:00
Harry Huang bf37812ca7 [Hybrid] Fix and optimize block-aligned splitting in mamba cache align mode (#33706) 
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
 2026-02-12 18:21:52 -08:00
Cyrus Leung ea5ff3a1f6 [Refactor] Simplify BOS/EOS token handling (#34435) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-02-12 18:18:24 -08:00
junuxyz c5a66d1697 [Core][BugFix] Fix PP KV cache sharding memory validation (#33698) 
Signed-off-by: junuxyz <216036880+junuxyz@users.noreply.github.com>
 2026-02-10 10:46:24 -05:00
Krish Gupta 748625cdaf [V1][BugFix] Fix EAGLE3 encoder cache miss with disable_chunked_mm_input (#34220) 
Signed-off-by: KrxGu <krishom70@gmail.com>
 2026-02-10 13:05:32 +00:00
Chen Zhang 97fa8f6590 [BugFix] Avoid prefix cache hit in the same schedule step for mamba layers (#29387) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2026-02-10 07:41:16 +00:00
Roger Wang 8a5e0e2b2b [Bugfix][Core] Fix CPU memory leak from Request reference cycle in prefix caching (#34183) 
Signed-off-by: Roger Wang <hey@rogerw.io>
 2026-02-10 13:03:32 +08:00
Reagan LeeandReagan Lee 7c233dbb36 [Tiny] Rename encoder budget file to more specific name (#34103) 
Signed-off-by: Reagan Lee <“reaganjlee@gmail.com”>
Co-authored-by: Reagan Lee <“reaganjlee@gmail.com”>
 2026-02-09 03:48:19 +00:00
Cyrus Leung 48312e579a [Misc] Make PlaceholderRange.get_num_embeds a method (#34035) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-02-07 05:30:17 +00:00
Cyrus Leung 035a6cb09a [Misc] Update code for encoder-decoder models (#33900) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-02-06 11:38:39 +08:00
Mark McLoughlin 2abd97592f [KV Connector][Metrics] Do not count local prefix cache hits in connector queries (#30522) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2026-02-05 09:57:27 +02:00
Nick Hill fa4e0fb028 [Core] Don't schedule spec tokens with prefill chunks (#33652) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-04 23:40:22 +00:00
Wentao YeandNick Hill 711edaf0d0 [Perf] Optimize spec decoding + async scheduling, 1.5% Throughput improvement (#33612) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2026-02-04 09:34:32 -05:00
Or Ozeri 8e32690869 [KV Connector][BugFix] scheduler: Delay freeing blocks of aborted async loads (#32255) 
Fixes a not-yet-reported case where it was possible for blocks to be
freed by an abort before an async transfer completed, resulting
in corrupted KV data.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-02-04 11:16:34 +00:00
zhanqiuhu 4403e3ed4c [Metrics] Add labeled prompt token metrics for P/D disaggregation (#33290) 
Add labeled Prometheus metrics to distinguish where prompt tokens come
from in P/D disaggregated deployments.

In P/D disaggregation, decode instances receive KV cache from prefill instances.
Currently, decode reports inflated prompt throughput because it counts all
prompt tokens as "computed", even though most were transferred.

This PR adds labeled metrics so users can understand actual compute work vs
transferred work:

vllm:prompt_tokens_by_source_total{source="local_compute"}        # Tokens prefilled locally
vllm:prompt_tokens_by_source_total{source="external_kv_transfer"} # Tokens received via KV transfer  
vllm:prompt_tokens_by_source_total{source="local_cache_hit"}      # Tokens from local prefix cache
vllm:prompt_tokens_cached_total                                    # Total cached (local + external, -1 when all 

Signed-off-by: Zhanqiu Hu <zh338@cornell.edu>
 2026-02-04 07:46:48 +00:00
Nick Hill 52ee21021a [BugFix][Spec Decoding] Fix negative accepted tokens metric crash (#33729) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-03 23:34:41 +00:00
Kuntai Du fbb3cf6981 [Bugfix][Async][Connector] avoid vllm-side double free during async scheduling + request abort + async KV cache transfer (#33377) 
Signed-off-by: KuntaiDu <kuntai@uchicago.edu>
 2026-02-03 21:50:15 +08:00
Nick Hill 61397891ce [Minor] Some code simplification in scheduler.py (#33597) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-03 15:00:00 +08:00
Cyrus Leung d7e17aaacd [Refactor] Move profiling methods to MM budget (#33559) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-02-02 23:27:00 +08:00
Yifan Qiao a01ef3fa51 [Fix] prefix cache hit rate == 0 bug with gpt-oss style models (#33524) 
Signed-off-by: Yifan Qiao <yifanqiao@berkeley.edu>
 2026-02-02 01:59:58 +00:00
jma99_2333andRoger Wang 22d9a056d5 Support clear mm and encoder cache (#33452) 
Signed-off-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-01-31 15:22:25 +00:00
杨朱 · KikiandClaude Opus 4.5 1a7894dbdf [Misc] Replace Optional[X] with X | None syntax (#33332) 
Signed-off-by: carlory <baofa.fan@daocloud.io>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-30 01:56:59 -08:00
Didier Durandandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 31b25f6516 [Doc]: fixing multiple typos in diverse files (#33256) 
Signed-off-by: Didier Durand <durand.didier@gmail.com>
Signed-off-by: Didier Durand <2927957+didier-durand@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-01-29 16:52:03 +08:00
Wentao YeandNick Hill 3e440786af [Feature] Fully support for async scheduling + PP, 30.8% E2E throughput improvement, 31.8% TPOT improvement (#32618) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-01-28 20:30:32 +00:00
Harry Mellor 2eb673a088 Add flake8-implicit-str-concat rules to Ruff (#33191) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-01-28 04:56:10 +00:00
Joshua DengRoger WangPatrick von PlatenNick Hill
 		91601ff478 [Feature] add session based streaming input support to v1 (#28973) 
Signed-off-by: Joshua Deng <joshuakdeng@gmail.com>
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-01-24 12:06:28 -08:00
Lukas Geiger 5fa0f6efa9 [EncoderCacheManager] Remove unnecessary copy (#32800) 
Signed-off-by: Lukas Geiger <lukas.geiger94@gmail.com>
 2026-01-24 14:28:57 +00:00
Harry HuangandChen Zhang 5206e5e28c [V1][Hybrid] Mamba Prefix Caching with align mode (#30877) 
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
 2026-01-23 09:56:48 -08:00
Nick Hill bc14663e6a [Cleanup] Move scheduler get_routed_experts logic to separate method (#32706) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-22 10:46:00 -05:00
Nicolò Lucchesi ea6102b85d [Bugfix] Fix Whisper/encoder-decoder GPU memory leak (#32789) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-01-22 10:50:37 +00:00
knlnguyen1802 378385b90c [EC Connector] Optimize remote cache check in scheduler (#32585) 
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
 2026-01-22 03:30:59 +00:00
Tomas Ruiz 4a5299c93f feat: spec decode with draft models (#24322) 
Signed-off-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
 2026-01-19 16:05:46 -05:00
Hongxin Xuandarlenxu 03da3b52ef [Bugfix] Refactor to support DP parallel in R3 (#32306) 
Signed-off-by: xhx1022 <1737006628@qq.com>
Co-authored-by: arlenxu <arlenxu@tencent.com>
 2026-01-16 15:13:58 +08:00
Wentao Ye b34474bf2c [Feature] Support async scheduling + PP (#32359) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-01-15 12:06:23 -05:00
Lumosis 66652e8082 [BugFix] Assign page_size_padded when unifying kv cache spec. (#32283) 
Signed-off-by: Lihao Ran <imlihao.ran@gmail.com>
 2026-01-14 20:10:01 +00:00
Or Ozeri 2be765b68a [BugFix] scheduler: Fix ordering preserving of skipped requests (#32173) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-01-12 18:39:38 +00:00