iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
12,946 Commits 3 Branches 0 Tags
eb28e8068dd870283f1a4fe8251ed9254d6f477e
Commit Graph
1013 Commits
Author SHA1 Message Date
Nick Hill 9273a427b5 [Misc] Allow enabling NCCL for DP sync when async scheduling (#32197) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-13 02:03:08 +00:00
Hongxin Xu22quinnarlenxu
 		49e6b86c91 [Feature] Support recording expert indices for rollout router replay (#28284) 
Signed-off-by: xhx1022 <1737006628@qq.com>
Signed-off-by: Hongxin Xu <70438206+xhx1022@users.noreply.github.com>
Signed-off-by: arlenxu <arlenxu@tencent.com>
Co-authored-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: arlenxu <arlenxu@tencent.com>
 2026-01-12 06:23:04 -08:00
Matthew Bonanni 2612ba9285 [1/N][Attention] Restructure attention: move files (#31916) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-01-09 13:10:24 -08:00
Max HuandMax Hu 6ebe34d6fa [Feature] Add iteration level logging and enhance nvtx marker (#31193) 
Signed-off-by: Max Hu <maxhu@nvidia.com>
Signed-off-by: Max Hu <hyoung2991@gmail.com>
Co-authored-by: Max Hu <maxhu@nvidia.com>
 2026-01-09 00:13:39 +00:00
Robert ShawRobert ShawWentao Ye
 		5825bbc1f7 [Quantization] Deprecate Long Tail of Schemes (#31688) 
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-01-08 19:07:45 -05:00
omer-dayanIsotr0pyIsotr0py
 		04a49669d1 RayLLM Bugfix - Preserve obj store URL for multi engine_config creation (#30803) 
Signed-off-by: Omer Dayan <omdayan@nvidia.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-01-08 10:00:25 +00:00
prashanth058 d3235cb503 [Fix] Enable mm_processor_cache with vision LoRA (#31927) 
Signed-off-by: prashanth058 <prashanth.dannamaneni@uipath.com>
 2026-01-08 15:31:51 +08:00
Michael Goin a68e703c32 [UX] Add -ep shorthand for --enable-expert-parallel (#30890) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-01-05 19:13:36 -08:00
Nick Hill bd877162eb [BugFix] Support online dense model DP without overhead (#30739) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: njhill <nickhill123@gmail.com>
 2026-01-02 23:36:38 +08:00
Reagan Lee 1f5b7c41c3 Add Multimodal Processor Benchmark (#29105) 
Signed-off-by: Reagan Lee <reaganjlee@gmail.com>
Signed-off-by: Reagan <reaganjlee@gmail.com>
 2026-01-01 19:26:53 -08:00
Jee Jee Libk-201prashanth058Anexdeus
 		ce1eafd1a5 [Core] Initialize LoRA support for tower and connector in multi-modal models (#26674) 
Signed-off-by: bk-201 <joy25810@foxmail.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: prashanth058 <prashanth.dannamaneni@uipath.com>
Co-authored-by: bk-201 <joy25810@foxmail.com>
Co-authored-by: prashanth058 <prashanth.dannamaneni@uipath.com>
Co-authored-by: Anexdeus <5142168@mail.ru>
 2025-12-26 04:48:20 -08:00
Cyrus Leung 7adeb4bfa8 [Bugfix] Fix max_model_len="auto" handling (#31260) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-24 19:15:27 +08:00
Michael Goin 8ee90c83f8 Add --max-model-len auto to auto-fit context to available memory (#29431) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-12-23 21:37:14 -08:00
Jeffrey Wang 1501a4070e [Bugfix] Read truncate_prompt_tokens from pooling_params in AsyncLLM.encode() (#31013) 
Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
 2025-12-20 10:29:31 +00:00
Elizabeth ThomasandHarry Mellor 41b6f9200f Remove all2all backend envvar (#30363) 
Signed-off-by: Elizabeth Thomas <email2eliza@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-12-18 19:46:28 +00:00
SungMinChoandMark McLoughlin a0b782f9cc [Metrics] Model FLOPs Utilization estimation (#30738) 
Signed-off-by: SungMinCho <tjdals4565@gmail.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
 2025-12-18 01:40:51 +00:00
danielafrimiandroot 7b966ae2ba [Fix]Load kv-cache dtype from hf_quant_config.json automatically (fix for reverted PR) (#30785) 
Signed-off-by: <>
Co-authored-by: root <root@gpu-937.slurm-workers-slurm.slurm.svc.cluster.local>
 2025-12-17 01:56:38 -08:00
jiangkuaixue123androot b9ff4f2a8d [feature] extend DBO to XBO (#30120) 
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Co-authored-by: root <root@hk01dgx028.cm.cluster>
 2025-12-16 00:04:01 -05:00
Nicolò Lucchesi 185c22bf2f [Misc][Hybrid allocator + kv connector] Optionally enable hybrid allocator + KV cache connector (#29805) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-12-15 11:17:58 +00:00
Michael Goin 09ad3b76b3 [Bug] Fix attention_backend arg string parsing (#30534) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-12-12 08:40:50 -07:00
QiuandCyrus Leung a11f4a81e0 [Misc][PCP&DCP] relocate PCP feature check (#30050) 
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2025-12-11 03:36:18 -08:00
Ning Xie d02d1043de fix: enhance human_readable_int function (#30337) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-12-10 23:30:33 -08:00
Cyrus Leung 7e24e5d4d6 [Deprecation] Remove deprecated task, seed and MM settings (#30397) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-10 19:59:39 -08:00
Cyrus Leung 5a87d8b9b1 [Deprecation] Remove deprecated plugin and compilation fields for v0.13 release (#30396) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-10 19:59:35 -08:00
Benjamin Chislettgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Harry Mellor
 		e858bfe051 [Cleanup] Refactor profiling env vars into a CLI config (#29912) 
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-12-09 13:29:33 -05:00
Cyrus Leung e83b7e379c Revert "[Renderer] Separate out RendererConfig from ModelConfig (#30145)" (#30199) 2025-12-07 00:00:22 -08:00
Cyrus Leung 27f4c2fd46 [Renderer] Separate out RendererConfig from ModelConfig (#30145) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-06 23:15:42 -08:00
Nick Hill 4026ae31e9 [Misc] Move disable_nccl_for_dp_synchronization init logic into VllmConfig (#30161) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-12-05 20:59:04 -08:00
Tova Movshovitzandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> adb315060c [KVConnector][Feature] Support KV connector cache reset via /reset_prefix_cache (#27170) 
Signed-off-by: tovam <tovam@pliops.com>
Signed-off-by: Tova Movshovitz <tovam@pliops.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-12-05 18:33:26 +00:00
Matthew Bonannigemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Harry MellorLucas Wilkinson
 		66e674cdd5 [Attention][UX][1/N] Add AttentionConfig and change attention env vars to CLI arguments (#26315) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni001@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
 2025-12-05 09:48:43 -08:00
Max HuandMax Hu c2894d3883 [Feature] Add Layer-wise NVTX Support (#29990) 
Signed-off-by: Max Hu <hyoung2991@gmail.com>
Signed-off-by: Max Hu <maxhu@nvidia.com>
Co-authored-by: Max Hu <maxhu@nvidia.com>
 2025-12-05 11:20:07 +00:00
Yong Hoon Shin 69520bc695 Add logging for cudagraph related info (#29825) 
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
 2025-12-03 01:01:48 -08:00
Sage Moore e6f114ac25 [Bugfix][EPLB] Prevent user-provided EPLB config from being overwritten with defaults (#29911) 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
 2025-12-02 13:20:22 -09:00
Isotr0py 63b1da76ba [Chore]: Reorganize gguf utils funtions under transformers_utils (#29891) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2025-12-02 17:33:23 +00:00
Zhuohan LiandNick Hill d0cd728907 [Core] Support reseting all running requests' KV while calling reset_prefix_cache (#28827) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-12-02 02:25:05 +00:00
shivampr cabc77cc86 [Core][Observability] Add KV cache residency metrics (#27793) 
Introduces three new Prometheus histograms for fine-grained observability of KV cache residency behavior:

vllm:kv_block_lifetime_seconds — total lifetime from allocation to free
vllm:kv_block_idle_before_evict_seconds — idle duration before eviction
vllm:kv_block_reuse_gap_seconds — time between consecutive reuses of the same block

These metrics help operators analyze KV cache efficiency, reuse patterns, and eviction timing beyond simple utilization rates.

Implementation uses monotonic timestamps for accuracy, 1% sampling for minimal overhead (~48 bytes/block), and is fully thread-safe with zero runtime cost when disabled.

Two new runtime flags are introduced:

--kv-cache-metrics – enable KV cache residency metrics
--kv-cache-metrics-sample – control sampling ratio (default: 0.01)

Signed-off-by: Shivam <shivamprasad91@gmail.com>
 2025-12-01 18:27:53 +00:00
Cyrus Leung f0a28bf661 [Misc] Unify tokenizer registration (#29767) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-01 11:34:58 +00:00
Cyrus Leung 34a984274e [Misc] Refactor tokenizer interface (#29693) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-29 04:02:21 -08:00
Tsukasa OI 762a4a6ca9 [Frontend] Perform offline path replacement to tokenizer (#29706) 
Signed-off-by: Tsukasa OI <floss_llm@irq.a4lg.com>
 2025-11-28 18:32:08 -08:00
Yanan CaoandClaude 3461e7efd8 [Frontend] Remap -O to -cc commandline flag (#29557) 
Signed-off-by: Yanan Cao <gmagogsfm@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2025-11-28 21:51:12 +00:00
Cyrus Leung 8d9338fae4 [Chore] Rename Processor to InputProcessor (#29682) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-28 09:35:41 -08:00
Isotr0py f946a8d743 [Chore]: Reorganize model repo operating functions in transformers_utils (#29680) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2025-11-28 08:46:51 -08:00
Cyrus Leung 9e6bcda3ac [mypy] Enable type checking for more directories (#29674) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-28 08:39:27 -08:00
wang.yuqiandCyrus Leung f4b76056ee Improve enable chunked_prefill & prefix_caching logic. (#26623) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2025-11-27 22:05:48 -08:00
Cyrus Leung ea228b4491 [Misc] Remove unused code from protocol.py (#29616) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-27 18:39:59 +00:00
Morrison TurnanskyadabeytaLuka GovedičHarry Mellor
 		0838b52e2e [Frontend][torch.compile] CompilationConfig Overhaul (#20283): Set up -O infrastructure (#26847) 
Signed-off-by: morrison-turnansky <mturnans@redhat.com>
Signed-off-by: adabeyta <aabeyta@redhat.com>
Signed-off-by: Morrison Turnansky <mturnans@redhat.com>
Co-authored-by: adabeyta <aabeyta@redhat.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-11-27 01:55:58 -08:00
Harry Mellorandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> a1f2676879 Scheduled removal of override_pooler_config and disable_log_requests (#29402) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-11-25 16:08:57 +00:00
Yifan QiaoandChen Zhang 48ddb02b79 [Hybrid Allocator] Support KV cache groups with different block_size (#29143) 
Signed-off-by: Yifan Qiao <yifanqiao@berkeley.edu>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
 2025-11-25 10:30:57 -05:00
Injae RyouandIsotr0py 794029f012 [Feature]: Improve GGUF loading from HuggingFace user experience like repo_id:quant_type (#29137) 
Signed-off-by: Injae Ryou <injaeryou@gmail.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2025-11-25 14:28:53 +00:00
Thomas Parnell 516c3f7847 [Bugfix] Fix logic for choosing default prefix caching setting (#29393) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
 2025-11-25 14:05:10 +00:00