99 Commits
Author SHA1 Message Date
b544587408 [Core][Feat] Pluggable KVCacheConfigBuilder for platform/model-specific KV cache planning (#53558)
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
2026-10-16 23:16:47 +00:00
d12ea8e309 [Bugfix][MRV2] Keep input_ids on later PP stages for raw-token models (#61016)
Signed-off-by: booth-algo <kevinlauofficial01@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 12:54:21 -07:00
bfdbfa1632 [Bugfix][PCP] Align MoE routing padding with gathered tokens (#60946)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: wangyicong <wangyicong@bytedance.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-10-14 21:04:00 -04:00
Ayushman Singh 9b5c6707a3 [Bugfix][MRV2] Capture DCP FULL cudagraphs with KV context (#61496)
Signed-off-by: Ayushman Singh <40520701+ayush1399@users.noreply.github.com>
2026-10-14 19:20:48 +00:00
df6c7ad031 [Core][Refactor] Move KV cache planning logic to kv_cache_planning.py (#61183)
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 03:05:39 +00:00
Wentao Ye 74c5cbcd7b [Bugfix][MRV2][Spec Decode] Honor dynamic K in autoregressive speculators, 1.2~1.3x kernel perf improvement (#57053)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-10-05 14:54:11 -04:00
aoshen02andClaude Opus 5.5 5e56e9af2a [Feature] Release the CUDA graph pool on sleep (#59160)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-04 01:13:10 +00:00
Almog Tavor 171d1b9ef6 [Elastic EP] Support Model Runner V2 (#53934)
Signed-off-by: almogtavor <almogtavor@gmail.com>
Signed-off-by: Almog Tavor <70065337+almogtavor@users.noreply.github.com>
2026-09-29 00:20:40 +00:00
a4eb3f25d6 [Spec Decode] Enable Gemma4 DSpark adaptive verification with FlashInfer (#57263)
Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-26 17:01:17 +08:00
Andreas KaratzasandOpenAI Codex 23110a0c9e [Bugfix] Capture prefill kernels for mixed FULL graphs (#58275)
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-24 02:05:07 -07:00
Nick Hill afec265bca [MRV2] Miscellaneous code cleanup (#57980) 2026-09-23 01:51:52 -07:00
Mi Jiazhiandjiazhimi e378275a8f [Model] Pass intermediate_tensors to the model when capturing CUDA gr… (#57745)
Co-authored-by: jiazhimi <jiazhimi@tencent.com>
2026-09-19 23:23:29 -07:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
Qiu ChunshuoCodexLucas Wilkinsonmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
676650397f [Feature][PCP] Support decode-only FULL CUDA graphs (#53867)
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
Signed-off-by: QiuChunshuo <chunshuoq@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-15 16:27:46 +08:00
Anjie HouandCodex 73d2a8cf86 [2/2][Model Runner V2] FULL CUDA graph capture for microbatched steps (DBO) (#51700)
Signed-off-by: specture724 <specture724@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-13 22:42:33 -07:00
d43bb2f37f [3/N] HiSparse: host-resident sparse-MLA decode hot-buffering (#53781)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Matthew Bonanni <matthew@neuralmagic.com>
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
2026-09-12 00:36:47 -04:00
Sy03Raphael RiallandZiyang ZhangCodexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
83990f5bcc [Bugfix][MRV2] Initialize DCP metadata after batch partitioning (#55212)
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Ziyang Zhang <hafuhafu@qq.com>
Signed-off-by: Sy03 <1370724210@qq.com>
Co-authored-by: Raphael Rialland <raphael.rialland@mistral.ai>
Co-authored-by: Ziyang Zhang <hafuhafu@qq.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-10 06:50:09 +00:00
Wentao Ye 07950d4734 [Kimi Bug] Fix kimi k3 startup cuda graph issue with recoverSSM (#55774)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-09-08 10:29:53 -04:00
685074cd61 [Bugfix][V2] Warm up kernels before capturing CUDA graphs (#55341)
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-04 22:44:18 +00:00
3f41d102c5 [1/2][Model Runner V2] DBO support, eager mode (#50945)
Signed-off-by: specture724 <specture724@gmail.com>
Signed-off-by: Anjie Hou <149605198+specture724@users.noreply.github.com>
Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
2026-09-04 14:17:39 -07:00
YizhouandOpenAI Codex 8f816a3f66 [MRV2][Metrics] Support CUDAGraphStat in MRV2 (#52358)
Signed-off-by: Yizhou Liu <liu_yizhou@outlook.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-09-04 07:19:50 +00:00
Isotr0py 1c26e57d3c [Bugfix] Raise for unavailable piecewise CUDA graphs (#54782)
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
2026-09-02 08:52:24 +00:00
Canlin Guoandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> b383e16396 [Bugfix] Reset cached Mamba align metadata on profiling teardown (#54044)
Signed-off-by: Canlin Guo <canlinguosdu@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-30 00:43:20 -07:00
Rahul ChalamalaJanelle CaiDevin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>Devin
d3d79ffc1e [Bugfix][Spec Decode] Capture the widest uniform decode batch by default (#50488)
Signed-off-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com>
Signed-off-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Signed-off-by: Rahul Chalamala <rahul@modal.com>
Co-authored-by: Janelle Cai <janelle.cai@modal.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai>
2026-08-28 22:38:13 -07:00
Nick HillandKimi Code 96242aa50d [Bugfix][MRV2] Release layer-bound KV cache memory in shutdown() (#54246)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Kimi Code <noreply@moonshot.ai>
2026-08-28 21:27:09 +00:00
Qiu Chunshuo b3af042abd Bugfix: use PCP slot mappings for PIECEWISE capture (#53869)
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
2026-08-27 05:39:36 +00:00
Wei Zhao 94d96e2446 [Bugfix] Release CUDA graph profiling memory before KV cache allocation (#53955)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
2026-08-27 01:35:59 +00:00
fanxingran 080a66a69c [Bugfix][ROCm] Capture CUDA graphs on the current stream (#53818)
Signed-off-by: fanxingran <xingran.fan@amd.com>
2026-08-26 23:16:45 +08:00
Wentao Ye a447955aca [Kimi Bug] Fix k3 torch.AcceleratorError: CUDA error: an illegal memory access was encountered (#53773)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-08-25 17:38:48 -07:00
Nick Hill e6cc0899b6 [Bugfix][MRV2] Run cudagraph memory profiling in a throwaway graph pool (#53682)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-08-25 18:52:15 +00:00
c74dd2d208 [Bugfix][MRV2] Dispatch uniform decode to a padded FULL cudagraph (#53407)
Signed-off-by: Xiaohu Guo <Xiaohu.Guo@amd.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: xiaohuguo2023 <149615094+xiaohuguo2023@users.noreply.github.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
2026-08-25 06:59:38 +00:00
a4d70bef37 [Model Runner V2] Reserve CUDA graph memory (#53306)
Signed-off-by: Anh Tran <anh.tran.3889@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Anh Tran <anh.tran.3889@gmail.com>
Co-authored-by: anhtra3889 <anhtra@nvidia.com>
2026-08-23 23:11:19 -07:00
Lucas WilkinsonOpenAI CodexClaude Opus 5Benjamin Chislettmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Nick HillOpenAI Codex
7f7a32cfec [Spec Decode] DSpark confidence-scheduled verification (#47808)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Signed-off-by: Lucas Wilkinson <wilkinson.lucas@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-08-12 12:27:48 -07:00
Lucas WilkinsonandOpenAI Codex 02ac17851d [Refactor][MRV2] Unify uniform decode token count helper (#51917)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-08-12 05:58:43 +00:00
0a94d85a66 [Bugfix][MRV2] Require all requests to be decoding for uniform-decode dispatch (#51865)
Signed-off-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com>
Co-authored-by: Janelle Cai <janelle.cai@modal.com>
2026-08-11 17:15:06 -07:00
Nick HillandWoosuk Kwon 8112b6c997 [MRV2] Always build attn metadata at capture time (#49364) (#49995)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
2026-07-27 17:25:17 +00:00
vllm-agentandvllm-agent CI bot 494845e79f Revert "[MRV2] Always build attn metadata at capture time" (#49364) (#49451)
Co-authored-by: vllm-agent CI bot <ci-bot@vllm-agent.local>
2026-07-23 09:51:33 -07:00
Woosuk Kwon 6049424b7e [MRV2] Always build attn metadata at capture time (#49364)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
2026-07-21 19:27:16 -07:00
Nick HillandClaude Opus 4.8 de6ec294ef [Bugfix] Fix DSA crash under breakable piecewise cudagraphs (#49302)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 18:28:13 +01:00
Ilya MarkovandMarkov Ilya f890e1dbe2 [BugFix] Set graph_pool_id before FULL CUDA graph capture in ModelRunner V2 (#48843)
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Markov Ilya <markovilya19@gmail.com>
2026-07-21 08:59:26 -04:00
251f7e478e [Model] Add PW CUDA graph support for Inkling [2/N] (#48822)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <mozf@inferact.ai>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-07-16 10:28:56 -07:00
Nick Hill 26587f9519 [BugFix][ModelRunner V2] Fix stale attn metadata in speculator prefill cudagraph capture (#48261)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-07-13 09:39:15 -07:00
07516fda67 [MRV2][SD] Make Dynamic SD comatible with Full Cuda Graphs (#45953)
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-07-03 23:58:27 -04:00
Woosuk Kwonandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 0a3e2dbc09 [Optimization] Skip DP padding tokens in MoE (#46428)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-23 14:54:46 -07:00
eacff17c8d [Model Runner V2][Bugfix] Fix MRV2 LoRA warmup (#35536)
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
2026-06-15 13:17:23 -07:00
Giancarlo DelfinandWoosuk Kwon 91945b6e4a [Bug Fix][Model Runner V2][Spec Decode] Warmup & capture with different attention states for speculator prefill (#44253)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
2026-06-03 13:32:40 -07:00
Woosuk Kwon e4a2e584e5 [MRV2] Remove assignment of graph_pool in cudagraph_utils (#44338)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
2026-06-02 11:50:27 -07:00
Woosuk Kwon 27fa5aa3b9 [MRV2] Support breakable CUDA graph (#44050)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
2026-05-30 09:40:52 -07:00
Nick Hill 39bba710be [MRV2][BugFix] Fix default-stream CG capture in P/W LoRA case (#43160)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-05-19 19:19:05 -07:00