MOVZX
be95edb7f6
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-16 17:35:42 +07:00
MOVZX
76df77c04e
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:16:39 +07:00
littlecircle0730 and Shanshan Shen
71536bd5f8
[MM][CG] Support ViT full CUDA graph for DeepSeek-VL2 ( #46005 )
...
Signed-off-by: littlecircle0730 <littlecircle0730@gmail.com >
Signed-off-by: littlecircle0730 <43994952+littlecircle0730@users.noreply.github.com >
Co-authored-by: Shanshan Shen <467638484@qq.com >
2026-10-15 21:38:52 +08:00
bfdbfa1632
[Bugfix][PCP] Align MoE routing padding with gathered tokens ( #60946 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: wangyicong <wangyicong@bytedance.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-10-14 21:04:00 -04:00
Ayushman Singh
9b5c6707a3
[Bugfix][MRV2] Capture DCP FULL cudagraphs with KV context ( #61496 )
...
Signed-off-by: Ayushman Singh <40520701+ayush1399@users.noreply.github.com >
2026-10-14 19:20:48 +00:00
Woosuk Kwon and Claude Opus 5.5
e1a5086256
[Attention] Remove cascade attention ( #61126 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 10:41:07 -07:00
liuzhenwei and Kunshang Ji
bfa7e5b7af
[XPU][CI] Enable XPU graph in graph tests ( #60784 )
...
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-10-12 10:39:40 +08:00
MOVZX
ac305c44cb
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-09 17:41:13 +00:00
Isotr0py and Shanshan Shen
8352b2427f
[Multimodal] Optimize encoder cudagraph padding logic ( #58925 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Shanshan Shen <467638484@qq.com >
2026-10-08 03:38:09 +00:00
Giancarlo Delfin
6ee9be6b3c
[Spec Decode] Rename AR speculators to StandaloneAR / TargetDependentAR ( #60335 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-10-07 14:01:59 -04:00
Wentao Ye
74c5cbcd7b
[Bugfix][MRV2][Spec Decode] Honor dynamic K in autoregressive speculators, 1.2~1.3x kernel perf improvement ( #57053 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-10-05 14:54:11 -04:00
MOVZX
f83ecfac16
Merge remote-tracking branch 'upstream/main' into patches
2026-10-05 08:52:31 +07:00
aoshen02 and Claude Opus 5.5
5e56e9af2a
[Feature] Release the CUDA graph pool on sleep ( #59160 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-04 01:13:10 +00:00
MOVZX
2877121b2d
Merge remote-tracking branch 'upstream/main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-03 21:59:03 +07:00
Qiu Chunshuo and Lucas Wilkinson
8486f1ebfe
[Perf][PCP] Shard decode requests across PCP ranks ( #52162 )
...
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-10-01 23:50:13 -04:00
Harry Mellor and Claude Opus 5.5
fc2c801ad9
Disable mypy arg-type and assignment checks in tests ( #59428 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 15:18:50 +00:00
5dd693122c
[MyPy][2/N] Fix mypy errors in small tests/ dirs ( #51043 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-29 21:27:45 +01:00
MOVZX and Hinata
ae0350c0de
[Spec Decode][MRV2] RecoverSSM for Qwen GDN and PLE short conv (PR #58863 )
...
Intake of open upstream PR vllm-project/vllm#58863 . Active only with
VLLM_USE_V2_MODEL_RUNNER=1; inert otherwise.
Speculative decoding on Gated DeltaNet snapshotted the full recurrent
state after every draft position and reserved one Mamba block per draft
position. On a 24 GB card that is reserved VRAM we never use, plus
per-step copy traffic.
RecoverSSM, the protocol Kimi-K3 KDA already uses, replaces this:
verify runs from a read-only checkpoint and stores a small per-token
replay record [correction (V), normalized key (K), gate (1)] in fp32;
commit replays only the accepted tokens and writes the recurrent state
once, plus the block-boundary state in align mode. The PLE dilated
short conv compacts its accepted window with the same plan.
Conflict in vllm/models/qwen4_exp/nvidia/mtp.py: our get_top_tokens()
work put LocalArgmaxMixin on Qwen4ExpMTP, the PR adds SupportsReplaySSM
to the same base list. Kept both.
Co-Authored-By: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-09-29 13:39:09 +07:00
Almog Tavor
171d1b9ef6
[Elastic EP] Support Model Runner V2 ( #53934 )
...
Signed-off-by: almogtavor <almogtavor@gmail.com >
Signed-off-by: Almog Tavor <70065337+almogtavor@users.noreply.github.com >
2026-09-29 00:20:40 +00:00
Jürgen Schmied and Claude
a80e41d690
[Mamba] Share RecoverSSM commit kernels and helpers across models
...
RecoverSSM's commit-plan and conv-window compaction kernels are model independent.
Move them from models/kimi_k3 into model_executor/layers/mamba/ops/recoverssm.py
(Kimi-K3 keeps its private names as import aliases; kernel source unchanged), add
mamba/recoverssm_utils.py (activation predicate, argument validation) and the
generic spec-row/request mapping helpers to v1/attention/backends/recoverssm_metadata.py.
CacheConfig.use_kda_recoverssm becomes use_recoverssm: the switch now selects RecoverSSM
for any model whose recurrent layers implement it, not only KDA.
No functional change for Kimi-K3.
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 00:07:04 +02:00
Misha Goin
79ac28c5bd
[CI] Reduce CUDA graph mode test overhead ( #58749 )
2026-09-26 11:22:33 -04:00
a4eb3f25d6
[Spec Decode] Enable Gemma4 DSpark adaptive verification with FlashInfer ( #57263 )
...
Signed-off-by: zixi-qi <zixi@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-26 17:01:17 +08:00
Andreas Karatzas and OpenAI Codex
23110a0c9e
[Bugfix] Capture prefill kernels for mixed FULL graphs ( #58275 )
...
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-24 02:05:07 -07:00
jiangkuaixue123 and Codex
de24e51908
[MM][V2] Enable encoder-only ViT CUDA graph capture ( #56922 )
...
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-17 16:12:17 +08:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
Kevin H. Luu and OpenAI Codex
56ca9904ce
[CI] Initialize ubatch runner in cudagraph unit test ( #56808 )
...
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-14 13:37:08 -05:00
Nick Hill and Andreas Karatzas
cc5dd0a857
[MRV1] Scope breakable cudagraphs to the piecewise path only ( #56312 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-11 19:55:34 +00:00
e77daef89e
[Model] Support DeepSeek-V4.1-Flash ( #56214 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
Signed-off-by: andyluo7 <andy.luo@amd.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: andyluo7 <andy.luo@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-11 02:11:19 -07:00
Yunzhu Lu and Isotr0py
442d36031c
[MM][CG] Enable encoder CUDA Graph for MiniCPM-V ( #42785 )
...
Signed-off-by: YunzhuLu <lucia.yunzhu@gmail.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-09-10 06:28:48 +00:00
Wentao Ye
07950d4734
[Kimi Bug] Fix kimi k3 startup cuda graph issue with recoverSSM ( #55774 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-08 10:29:53 -04:00
d3d79ffc1e
[Bugfix][Spec Decode] Capture the widest uniform decode batch by default ( #50488 )
...
Signed-off-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com >
Signed-off-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Signed-off-by: Rahul Chalamala <rahul@modal.com >
Co-authored-by: Janelle Cai <janelle.cai@modal.com >
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Devin <devin@cognition.ai >
2026-08-28 22:38:13 -07:00
Qiu Chunshuo
b3af042abd
Bugfix: use PCP slot mappings for PIECEWISE capture ( #53869 )
...
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com >
2026-08-27 05:39:36 +00:00
c74dd2d208
[Bugfix][MRV2] Dispatch uniform decode to a padded FULL cudagraph ( #53407 )
...
Signed-off-by: Xiaohu Guo <Xiaohu.Guo@amd.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: xiaohuguo2023 <149615094+xiaohuguo2023@users.noreply.github.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-08-25 06:59:38 +00:00
Shanshan Shen and Andreas Karatzas
0ad04cff1b
[ROCm][CI] Enable ViT CUDA graph tests on AMD gfx950 GPUs ( #52256 )
...
Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-08-17 09:52:02 +08:00
Isotr0py and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
e2fa28594f
[1/N] Unify multiple-path encoder cuda graph support ( #49934 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-02 10:33:23 +08:00
+13
aeeb36b1f1
[New model] Kimi K3 ( #50000 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Summer Yang <girasoleyang@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Bowen Wang <abmfy@icloud.com >
Co-authored-by: gnovack <novackgm@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com >
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-07-30 03:49:58 -07:00
Andreas Karatzas
f00efc5265
[CI] Isolate cudagraph tests in child processes ( #49510 )
...
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
2026-07-23 17:16:19 +00:00
Nick Hill and Claude Opus 4.8
1dca300653
[CI] Fix and wire encoder/manager cudagraph unit tests ( #49339 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-07-21 20:41:16 +01:00
Nick Hill and Claude Opus 4.8
de6ec294ef
[Bugfix] Fix DSA crash under breakable piecewise cudagraphs ( #49302 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-07-21 18:28:13 +01:00
Ilya Markov and Markov Ilya
f890e1dbe2
[BugFix] Set graph_pool_id before FULL CUDA graph capture in ModelRunner V2 ( #48843 )
...
Signed-off-by: Markov Ilya <markovilya19@gmail.com >
Co-authored-by: Markov Ilya <markovilya19@gmail.com >
2026-07-21 08:59:26 -04:00
251f7e478e
[Model] Add PW CUDA graph support for Inkling [2/N] ( #48822 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-07-16 10:28:56 -07:00
ad32608e24
[MM][Perf][CG] Support dual-path ViT full CUDA graph for DeepSeek-OCR ( #43586 )
...
Signed-off-by: shen-shanshan <467638484@qq.com >
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn >
2026-06-16 04:35:20 -07:00
eb28452b10
[Model] Add DiffusionGemma Support ( #45163 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Martin Kukla <martin.kukla@cantab.net >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Dipika Sikka <dsikka@redhat.com >
Co-authored-by: NickLucche <nlucches@redhat.com >
Co-authored-by: jiahanc <173873397+jiahanc@users.noreply.github.com >
Co-authored-by: Alec Kohlhoff <134344302+aleckohlhoff@users.noreply.github.com >
Co-authored-by: Porras Huang <20535584+porrashuang@users.noreply.github.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: scoootscooob <167050519+scoootscooob@users.noreply.github.com >
2026-06-11 22:17:35 -07:00
JooHo Lee
a045c7425f
[MM][CG] Profile encoder CUDA graph pool memory ( #41714 )
...
Signed-off-by: JooHo Lee <jooho414@gmail.com >
2026-06-02 12:27:34 +08:00
Weida Hong
dfe8ba7c80
Adjust design around encoder_cudagraph_forward ( #42288 )
...
Signed-off-by: Weida Hong <wdhongtw@google.com >
2026-05-29 03:02:52 +00:00
Isotr0py
f0feb15e7f
[Multimodal] Simplify ViT CUDA graph interfaces ( #41234 )
...
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
2026-05-22 22:31:00 +08:00
Weida Hong
6bb8753db1
Correcting the mock classes for MM GC tests ( #43321 )
...
Signed-off-by: Weida Hong <wdhongtw@google.com >
2026-05-22 15:21:35 +08:00
Jiangyun Zhu
8a56da3845
[Experimental] Breakable CUDA graph ( #42304 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
2026-05-16 22:04:12 +08:00
Baorun (Lauren) Mu
a7737cb4f3
[Fix] Misc Fixes in ViT CUDA Graph ( #38040 )
...
Signed-off-by: Baorun Mu <bmu@nvidia.com >
2026-05-14 23:49:06 +08:00