Commit Graph
763 Commits
Author SHA1 Message Date
Yong Hoon Shin 69520bc695 Add logging for cudagraph related info (#29825)
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
2025-12-03 01:01:48 -08:00
Arpit KhandelwalandLuka Govedič d7284a2604 [Core] Rename PassConfig flags as per RFC #27995 (#29646)
Signed-off-by: arpitkh101 <arpit5khandelwal@gmail.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
2025-12-03 03:38:55 +00:00
Lucas Wilkinson 5cdd664509 [BugFix] Fix assert in build_for_cudagraph_capture (#29893)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
2025-12-02 16:56:54 -08:00
Wushi Dong 0037b5746a [Core] Eliminate redundant is_encoder_decoder lookups (20-40us/step) (#29800)
Signed-off-by: Wushi Dong <dongws@meta.com>
2025-12-02 07:08:07 +00:00
Zhuohan LiandNick Hill d0cd728907 [Core] Support reseting all running requests' KV while calling reset_prefix_cache (#28827)
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
2025-12-02 02:25:05 +00:00
Yifei Zhang 1ab8fc8197 Make PyTorch profiler gzip and CUDA time dump configurable (#29568)
Signed-off-by: Yifei Zhang <yifei.zhang1992@outlook.com>
2025-12-01 04:30:46 +00:00
Woosuk Kwon ec38a7368d [Model Runner V2] Use packed mask for prompt bin counts (#29756)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-30 14:15:42 -08:00
Cyrus Leung 64bc09ba27 [Core] Enable inputs_embeds_size separate from hidden_size (#29741)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2025-11-30 17:31:12 +08:00
VensenandTJian 66b5840287 [Bugfix][sleepmode][fp8 kv cache]: Fix FP8 KV cache + sleep(level=2) gibberish output (#28783)
Signed-off-by: vensen <vensenmu@gmail.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
2025-11-30 14:24:25 +08:00
Woosuk Kwon f223ed4181 [Model Runner V2] Fuse penalties and temperature into single kernel (#29720)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-29 02:29:16 -08:00
Woosuk Kwon 6afc0ffaf6 [Model Runner V2] Add sample/ directory and reorganize files (#29719)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-29 00:41:01 -08:00
Jee Jee LiandCyrus Leung 39e63dec7c [LoRA] Cleanup LoRA unused code (#29611)
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2025-11-28 22:52:58 -08:00
Woosuk Kwon 4a80ad0a25 [Model Runner V2] Don't use UVA buffer for prefill_len (#29713)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-28 20:27:16 -08:00
Lucas Wilkinson e23f665d83 [BugFix] Fix DBO failing with TypeError: 'NoneType' object is not iterable (#29698)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
2025-11-28 20:19:01 -08:00
Woosuk Kwon ca1b1e7296 [Model Runner V2] Refactor prefill token preparation (#29712)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-28 19:49:17 -08:00
Woosuk Kwon 1dcafb3dea [Model Runner V2] Support penalties using bin counts (#29703)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-28 17:53:17 -08:00
Benjamin Chislett 1986de1375 [Perf] Optimize EAGLE prepare_inputs_padded with triton kernels (#28597)
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
2025-11-28 22:25:05 +00:00
Didier Durand fae6943068 [Doc]: fixing typos in multiple files. (#29685)
Signed-off-by: Didier Durand <durand.didier@gmail.com>
2025-11-28 08:41:41 -08:00
Lucas Wilkinson be493e0b3c [BugFix] Fix new nightly failures (#29578)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
2025-11-27 13:45:38 -08:00
Woosuk Kwon ae0ce1be27 [Model Runner V2][BugFix] Keep reference to GPU tensors in AsyncOutput (#29623)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-27 12:38:53 -08:00
Matthew Bonanni fc1d8be3dc [Attention] Update attention imports (#29540)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
2025-11-27 11:19:09 -05:00
Didier Durand 66d3d5422c [Doc]: fixing typos in diverse files (#29492)
Signed-off-by: Didier Durand <durand.didier@gmail.com>
2025-11-27 07:15:50 -08:00
Cyrus Leung e6d4f3c254 [Bugfix] Fix pre-commit (#29601)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2025-11-27 02:23:06 -08:00
0838b52e2e [Frontend][torch.compile] CompilationConfig Overhaul (#20283): Set up -O infrastructure (#26847)
Signed-off-by: morrison-turnansky <mturnans@redhat.com>
Signed-off-by: adabeyta <aabeyta@redhat.com>
Signed-off-by: Morrison Turnansky <mturnans@redhat.com>
Co-authored-by: adabeyta <aabeyta@redhat.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2025-11-27 01:55:58 -08:00
Woosuk Kwon da3222f371 [Model Runner V2] Implement multi-step Eagle with CUDA graph (#29559)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-27 00:09:41 -08:00
Woosuk Kwon 11ea5ec1ff [Model Runner V2] Refactor CudaGraphManager (#29583)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-26 21:37:59 -08:00
Woosuk Kwon ee80aee1ca [Model Runner V2] Minor cleanup for build_attn_metadata (#29576)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-26 20:10:12 -08:00
Woosuk Kwon 0aeb698b77 [Model Runner V2] Minor code cleanup (#29570)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-26 19:47:17 -08:00
Johnny Yang ba1fcd84a7 [TPU] add tpu_inference (#27277)
Signed-off-by: Johnny Yang <johnnyyang@google.com>
2025-11-26 14:46:36 -08:00
Lucas Wilkinson 56539cddac [Core] Refactor padding logic and pad for CUDA graphs before attention metadata building (#28579) 2025-11-26 14:07:13 -05:00
Matthew Bonanni 430dd4d9eb [Attention] Remove imports from vllm/attention/__init__.py (#29342)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
2025-11-26 10:53:15 -07:00
Xieyang Xu 12866af748 dummy run corner case (#29433) 2025-11-26 00:20:35 +00:00
Andrey KhalyavinandTyler Michael Smith de75b0bb70 [BugFix] Fix initialization of draft model. (#29319)
Signed-off-by: Andrey Khalyavin <halyavin@yandex-team.ru>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
2025-11-25 18:45:58 -05:00
Nick Hill 4e57c6587f [Core] Support logprobs with spec decode + async scheduling (#29223)
Signed-off-by: Nick Hill <nhill@redhat.com>
2025-11-25 12:55:24 -08:00
Roger Wang c2c661af9b [Bugfix] Fix overallocation in MM profiling (#29386)
Signed-off-by: Roger Wang <hey@rogerw.io>
2025-11-25 12:38:36 +00:00
Nicolò LucchesiandEkagra Ranjan 798e87db5c [Core] Generalize Encoder-Decoder seq_lens computation to avoid Whisper hardcoded logic (#29268)
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
2025-11-25 11:32:11 +00:00
Rémi Delacourt 12c007e288 EAGLE Support DP>1 (#26086)
Signed-off-by: Rémi Delacourt <remi@mistral.ai>
Signed-off-by: Rémi Delacourt <54138269+Flechman@users.noreply.github.com>
Signed-off-by: remi <remi@mistral.ai>
2025-11-25 07:32:21 +00:00
92effb07a4 [Model] Add HunyuanOCR support (#29327)
Signed-off-by: manayang <jackmanayang@gmail.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Roger Wang <hey@rogerw.io>
Co-authored-by: sergeywang <sergeywang@tencent.com>
Co-authored-by: manayang <jackmanayang@gmail.com>
Co-authored-by: manayang <manayang@tencent.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
2025-11-25 03:28:51 +00:00
Hanjie QiuandBenjamin Chislett 5f9679a43b [Spec Decode] Add support for EAGLE3 heads that do not use_aux_hidden_states (#27688)
Signed-off-by: hjjq <hanjieq@nvidia.com>
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com>
2025-11-24 20:13:12 -05:00
Woosuk Kwon f32c7d6f54 [Model Runner V2] Simplify Eagle bookkeeping with num_rejected (#29347)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-24 13:54:59 -08:00
Woosuk Kwon 97588c4d12 [Model Runner V2] Add minor clarification comments for Eagle (#29332)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-24 11:28:56 -08:00
Woosuk Kwon cec418b5df [Model Runner V2] Change Numba AoT to JIT (#29328)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-24 09:34:37 -08:00
Woosuk Kwon cc313cb73d [Model Runner V2] Implement Single-step Eagle 1 (#29300)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-24 09:32:27 -08:00
WeiQing ChenandSunChenxiang123 2601f18a82 [EPLB] Optimize EPLB for Async Rearrange Experts (#22179)
Signed-off-by: David Chen <530634352@qq.com>
Co-authored-by: SunChenxiang123 <1291824390@qq.com>
2025-11-24 09:08:29 -05:00
Didier DurandandHarry Mellor eca7a8fb59 [Doc]: fix typos in various files (#29230)
Signed-off-by: Didier Durand <durand.didier@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2025-11-24 11:10:48 +00:00
Woosuk Kwon 3e1ad40655 [Model Runner V2] Add apply_temperature option to gumbel_sample (#29276)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-23 14:13:00 -08:00
Woosuk Kwon 62d54ba46d [Model Runner V2] Optimize CUDA graph capture time (#29275)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-23 11:15:32 -08:00
Woosuk Kwon b004c00418 [Model Runner V2] Support spec decoding [1/N] (#29274)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-23 10:09:06 -08:00
Woosuk Kwon 7f12c82fa6 [Model Runner V2] Change bookkeeping logic in preparation for spec decoding (#29194)
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2025-11-23 09:42:52 -08:00
Woosuk Kwon 20ee418adc [Model Runner V2] Minor fix for cudagraph_utils (#29256) 2025-11-22 20:12:50 -08:00