iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
12,777 Commits 3 Branches 0 Tags
b634e619bbcfed0abe4e01d0e2d97fb1fdfdbbd5
Commit Graph
466 Commits
Author SHA1 Message Date
prashanth058 d3235cb503 [Fix] Enable mm_processor_cache with vision LoRA (#31927) 
Signed-off-by: prashanth058 <prashanth.dannamaneni@uipath.com>
 2026-01-08 15:31:51 +08:00
R3hankhan 1ab055efe6 [OpenAI] Extend VLLMValidationError to additional validation parameters (#31870) 
Signed-off-by: Rehan Khan <Rehan.Khan7@ibm.com>
 2026-01-07 14:45:49 +00:00
Benjamin ChislettandNick Hill f7008ce1c4 [Perf] Async Scheduling + Speculative Decoding + Structured Outputs (#29821) 
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-01-06 18:50:37 +00:00
Masataro Asai 142c4d1738 make 500: InternalServerError more informative (#20610) 
Signed-off-by: Masataro Asai <guicho2.71828@gmail.com>
 2026-01-06 17:36:24 +00:00
John CalderonandBenjamin Chislett 2f4e6548ef [Bugfix] vLLM produces invalid UTF-8 tokens and “�” (#28874) 
Signed-off-by: John Calderon <jcalderon@nvidia.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com>
 2026-01-06 00:23:00 +00:00
Nick Hill b53b89fdb3 [BugFix] Async scheduling: handle model forward errors more cleanly (#31611) 
Signed-off-by: njhill <nickhill123@gmail.com>
 2026-01-04 11:04:37 -08:00
Nick Hill bd877162eb [BugFix] Support online dense model DP without overhead (#30739) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: njhill <nickhill123@gmail.com>
 2026-01-02 23:36:38 +08:00
Reagan Lee 1f5b7c41c3 Add Multimodal Processor Benchmark (#29105) 
Signed-off-by: Reagan Lee <reaganjlee@gmail.com>
Signed-off-by: Reagan <reaganjlee@gmail.com>
 2026-01-01 19:26:53 -08:00
Nick Hill 3b312fb792 [Minor] Various small code cleanups/simplifications (#31508) 
Signed-off-by: njhill <nickhill123@gmail.com>
 2025-12-29 22:42:06 -08:00
Nick Hill e54ee3ea33 [Core] Deduplicate generate/encode logic in AsyncLLM (#31510) 
Signed-off-by: njhill <nickhill123@gmail.com>
 2025-12-30 10:42:45 +08:00
Jee Jee Libk-201prashanth058Anexdeus
 		ce1eafd1a5 [Core] Initialize LoRA support for tower and connector in multi-modal models (#26674) 
Signed-off-by: bk-201 <joy25810@foxmail.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: prashanth058 <prashanth.dannamaneni@uipath.com>
Co-authored-by: bk-201 <joy25810@foxmail.com>
Co-authored-by: prashanth058 <prashanth.dannamaneni@uipath.com>
Co-authored-by: Anexdeus <5142168@mail.ru>
 2025-12-26 04:48:20 -08:00
Cyrus Leung 09dc7c690c [Chore][1/2] Drop v0.14 deprecations (#31285) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-24 09:54:01 -08:00
Michael Goin 8ee90c83f8 Add --max-model-len auto to auto-fit context to available memory (#29431) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-12-23 21:37:14 -08:00
Mark McLoughlinandNick Hill f790068600 [Core] Add a random suffix to frontend-provided request IDs (#27987) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-12-23 13:05:39 -08:00
Jeffrey Wang 1501a4070e [Bugfix] Read truncate_prompt_tokens from pooling_params in AsyncLLM.encode() (#31013) 
Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
 2025-12-20 10:29:31 +00:00
Seiji Eicher 1ab5213531 Make engine core client handshake timeout configurable (#27444) 
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
 2025-12-19 20:38:30 +00:00
Nick Hill 45c0526ac9 [BugFix] Handle errors when preprocessing added requests (#30895) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-12-19 01:29:11 +00:00
Chen Zhang 24b65eff0d [BugFix] Spec decode with VLLM_ENABLE_V1_MULTIPROCESSING=0 (#30319) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-12-18 19:47:56 +00:00
AlecandNick Hill 62be3670cb [BugFix] Add sleep to fix tight loop and release GIL (#29476) 
Signed-off-by: alec-flowers <aflowers@nvidia.com>
Signed-off-by: Alec <35311602+alec-flowers@users.noreply.github.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-12-18 09:52:55 -08:00
Andreas Karatzas be2ad5f920 [ROCm][Bugfix] fix(structured_output): Skip guidance backend for schemas with patternProperties (#30730) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2025-12-18 07:04:57 +00:00
Jee Jee Li 0e391e7570 [Bugfix] Fix RequestOutput miss lora_request (#30636) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2025-12-16 01:36:35 -08:00
Cyrus Leung 39cefbdf17 [Refactor] TokenizerRegistry only uses lazy imports (#30609) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-13 23:16:22 +08:00
Cyrus Leung 64251f48df [Chore] Adjust tokenizer import to avoid circular imports (#30601) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-13 04:42:39 -08:00
Cyrus Leung b09806e28f [Bugfix] Dictionary MM embeddings for online chat (#30507) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-13 15:48:56 +08:00
Jialin Ouyang 9f042ba26b [Perf] Enable environment cache in EngineCore to enable the feature for UniProcExecutor as well (#29289) 
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2025-12-10 14:13:01 -05:00
Cyrus Leung e72d65b959 {Deprecation] Remove tokenizer setter (#30400) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-10 19:10:58 +00:00
Will EatonMark McLoughlinNick Hillchaunceyjiang
 		a9e4106f28 [P/D] KV Load Failure Recovery/Abort Configuration (#26813) 
Signed-off-by: Will Eaton <weaton@redhat.com>
Signed-off-by: Will Eaton <me@wseaton.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2025-12-10 11:00:52 -08:00
Cyrus Leung 253305d5b2 [Chore] Delay recent deprecations (#30398) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-10 17:48:38 +00:00
Benjamin Chislettgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Harry Mellor
 		e858bfe051 [Cleanup] Refactor profiling env vars into a CLI config (#29912) 
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-12-09 13:29:33 -05:00
Victor Ziliang Peng f1599ca55d feat(metrics): Add prefill KV compute metric excluding cached tokens (#30189) 
Signed-off-by: Ziliang Peng <ziliang@character.ai>
 2025-12-09 00:08:48 +00:00
Cyrus Leung e83b7e379c Revert "[Renderer] Separate out RendererConfig from ModelConfig (#30145)" (#30199) 2025-12-07 00:00:22 -08:00
Cyrus Leung 27f4c2fd46 [Renderer] Separate out RendererConfig from ModelConfig (#30145) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-06 23:15:42 -08:00
Tova Movshovitzandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> adb315060c [KVConnector][Feature] Support KV connector cache reset via /reset_prefix_cache (#27170) 
Signed-off-by: tovam <tovam@pliops.com>
Signed-off-by: Tova Movshovitz <tovam@pliops.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-12-05 18:33:26 +00:00
Nick Hill dc264bcea1 [BugFix] Eagerly abort cancelled final-step requests (#29987) 
Currently, when requests are cancelled while executing their final
step, "completion" is handled based on normal stop processing (e.g.
length or stop token), so the abort has no effect. This is typically
not a problem, but when a kv connector is involved it thinks the
request completed successfully rather than being aborted.

This is problematic for disaggregated prefill which will free kv
cache blocks if the request was aborted but not if it completed
successfully—since the cancelled request will never be sent to
the decode side, kv cache blocks remain pinned until the fall-back
timeout expires. The problem is exacerbated when many requests
are cancelled and/or there are large prefills whose forward pass
takes a long time (since the window is bigger).

This PR fixes the problem by processing pending aborts
immediately prior to processing model output each step; we process
only aborts, not new requests, since it's preferable for latency to
process model outputs before new incoming requests.

Fixes #26400.

Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-12-05 17:28:32 +00:00
Yongtao Huang 2fc5d6e0d7 Fix LLMEngine.del dp_group cleanup condition (#29954) 
Signed-off-by: Yongtao Huang <yongtaoh2022@gmail.com>
 2025-12-03 12:14:44 -08:00
Cyrus Leung 653591d5e7 [Chore] Move tokenizer initialization methods (#29793) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-12-02 13:33:37 +08:00
Zhuohan LiandNick Hill d0cd728907 [Core] Support reseting all running requests' KV while calling reset_prefix_cache (#28827) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-12-02 02:25:05 +00:00
Yifei Zhang 1ab8fc8197 Make PyTorch profiler gzip and CUDA time dump configurable (#29568) 
Signed-off-by: Yifei Zhang <yifei.zhang1992@outlook.com>
 2025-12-01 04:30:46 +00:00
Cyrus Leung 2afcec4dec [Misc] Update TokenizerLike interface and move get_cached_tokenizer (#29730) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-30 14:59:47 +08:00
Cyrus Leung fa59fe417f [Chore] Move detokenizer_utils to vllm/tokenizers (#29727) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-29 06:25:17 -08:00
Cyrus Leung 34a984274e [Misc] Refactor tokenizer interface (#29693) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-29 04:02:21 -08:00
Cyrus Leung 8d9338fae4 [Chore] Rename Processor to InputProcessor (#29682) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-28 09:35:41 -08:00
maang-h cc0f2a0e19 [Doc] Improve abnormal information string (#29655) 
Signed-off-by: maang <maang_h@163.com>
 2025-11-28 00:12:20 -08:00
wang.yuqiandCyrus Leung f4b76056ee Improve enable chunked_prefill & prefix_caching logic. (#26623) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2025-11-27 22:05:48 -08:00
Harry Mellorandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> a1f2676879 Scheduled removal of override_pooler_config and disable_log_requests (#29402) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-11-25 16:08:57 +00:00
Avishek Goswami 32c40b95e0 [BugFix] bad_words filtering ineffective when n > 1 (#29313) 
Signed-off-by: GOavi101 <1704178@kiit.ac.in>
 2025-11-25 09:36:34 +00:00
Julien Denizegemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>mgoin
 		57430fc95c Default model load/config/tokenizer to mistral format if relevant files exist (#28659) 
Signed-off-by: Julien Denize <julien.denize@mistral.ai>
Signed-off-by: Julien Denize <40604584+juliendenize@users.noreply.github.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2025-11-21 13:58:59 -08:00
Cyrus Leung 56e96b37e4 [V0 Deprecation] Remove best_of (#29090) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-11-21 11:40:40 +08:00
Samitand22quinn 371b1d4c61 [RL] Add Pause and Resume Generation for Asynchronous RL Training (#28037) 
Signed-off-by: SamitHuang <285365963@qq.com>
Signed-off-by: Samit <285365963@qq.com>
Signed-off-by: samithuang <285365963@qq.com>
Co-authored-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-11-20 03:01:03 -08:00
Benjamin Chislettandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> fcbcba6c70 [Feat] Iteration-level profiling for Torch and CUDA profiler (#28987) 
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-11-19 19:17:48 -08:00