Feanor and Taneem Ibrahim
5d5c3d7c43
[Bugfix] Preserve padding token types for cross encoders ( #58143 )
...
Signed-off-by: zhaoye <772971548@qq.com >
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
2026-10-15 03:09:24 +00:00
dingdangmao and Codex
124b89d5b7
[Bugfix][RecoverSSM] Correct aligned state indices at exact block boundaries ( #59962 )
...
Signed-off-by: apex-mochen <2756823972@qq.com >
Co-authored-by: Codex <noreply@openai.com >
2026-10-12 02:08:14 +00:00
DimensionSTP and OpenAI Codex
10cc2f6ae2
[Bugfix] Bound GDN speculative decode query widths ( #55504 )
...
Signed-off-by: DimensionSTP <65501090+DimensionSTP@users.noreply.github.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-10-10 11:00:16 +08:00
Nick Hill
7d1d8660d6
[MRV2] Miscellaneous code structure updates ( #60754 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-10-09 17:58:09 -07:00
3d6853b32c
[Bugfix] Seed hybrid mamba state index with mamba_block_size on prefix-cache hits ( #55601 )
...
Signed-off-by: pondzikk <pondzik@gmail.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: ptorsten <1652174+ptorsten@users.noreply.github.com >
2026-10-07 17:09:50 +00:00
8a26869bca
[Bugfix] Index Mamba align state copies by request slot in MRV2 ( #60210 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Karl0007 <44745209+Karl0007@users.noreply.github.com >
Co-authored-by: Yongqin Wang <yongqinwang@roblox.com >
2026-10-07 02:32:46 +00:00
eb2901b735
[Perf][DSv4.1] Run decoder replay layers in CUDA graphs and trim in PIECEWISE graphs ( #59532 )
...
Signed-off-by: Juntian Liu <Juntianl777@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-05 22:11:45 -07:00
0eac152707
[Bugfix] Route every speculation-capable row through the speculative path so recurrent state stays consistent ( #56531 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
Co-authored-by: Cursor Agent <cursoragent@cursor.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-06 03:03:44 +04:00
e7c903609f
[Perf][MRV2] Allow FULL decode graphs for one-token prompt tails ( #58400 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
2026-09-28 12:48:52 -07:00
Nick Hill and Claude Opus 5.5
7dbd0a8d22
[Bugfix][MRV2] Treat padded prompt tails as spec-decode rows for hybrid models ( #58434 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-09-24 18:29:53 -07:00
Yejing Lai and Kunshang Ji
14f98cbe5d
[Bugfix][XPU] store the pointer raw bit pattern instead of its numeric value ( #54514 )
...
Signed-off-by: Lai, Yejing <yejing.lai@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-09-24 02:05:26 -07:00
Nick Hill
afec265bca
[MRV2] Miscellaneous code cleanup ( #57980 )
2026-09-23 01:51:52 -07:00
88aa0d287d
[Spec decode] Support variable-length decode for Kimi-K3 adaptive ver ( #52988 )
...
Signed-off-by: Albert Cheng <albecheng@nvidia.com >
Signed-off-by: Albert Cheng (Engrg-Hardware 1) <albecheng@login-bia01.bia.clusters.nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
2026-09-22 20:43:32 -07:00
hcl and mgoin
71a2392845
[Bugfix][V1] Read ModelState max_model_len from model config ( #58149 )
...
Signed-off-by: Chenglun Hu <chenglunhu@gmail.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
2026-09-22 19:45:04 +00:00
1dc2d854c1
[Feat][Model] Support encoder-side SWA-bounded replay for DeepSeek-V4.1-Flash ( #56227 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
2026-09-18 15:01:24 -07:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
ebe1dec2da
[PCP][DCP] Enable PCP+DCP on sparse-MLA models ( #56157 )
...
Signed-off-by: patryk.saffer <patryk.saffer@blc-login-1.blc1.mistralcompute.ai >
Signed-off-by: patryk.saffer <patryk.saffer@blc-login-0.blc1.mistralcompute.ai >
Signed-off-by: PatrykSaffer <patryk.saffer@mistral.ai >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: patryk.saffer <patryk.saffer@blc-login-1.blc1.mistralcompute.ai >
Co-authored-by: patryk.saffer <patryk.saffer@blc-login-0.blc1.mistralcompute.ai >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Patryk Saffer <PatrykSaffer@users.noreply.github.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-12 23:24:44 +00:00
Nick Hill
e6cb56337b
[Core] MRV2 support for fast-prefill ( #56145 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-09-10 10:36:04 -07:00
Harry Mellor and Claude Opus 5
719284fe15
[Core][Model] Unify XD-RoPE into M-RoPE and derive the channel count ( #56078 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-09 09:23:30 -07:00
3f41d102c5
[1/2][Model Runner V2] DBO support, eager mode ( #50945 )
...
Signed-off-by: specture724 <specture724@gmail.com >
Signed-off-by: Anjie Hou <149605198+specture724@users.noreply.github.com >
Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
2026-09-04 14:17:39 -07:00
Jiangyun Zhu
98ed0856f3
[Model] add GLM-5.3-Flash support ( #53906 )
2026-09-03 09:40:35 -07:00
+5
e126687a9a
[Model] Support Qwen3.8-Flash-Next ( #53896 )
...
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com >
Signed-off-by: andyluo7 <andy.luo@amd.com >
Signed-off-by: lishunyang12 <lishunyang03@gmail.com >
Signed-off-by: Harry Huang <huanghaoyan.hhy@alibaba-inc.com >
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: KungYork <firelikest@163.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: andyluo7 <43718156+andyluo7@users.noreply.github.com >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: andyluo7 <andy.luo@amd.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: lishunyang12 <lishunyang03@gmail.com >
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-31 05:57:56 +00:00
Wentao Ye
41729fc53b
[K3 Perf] Optimize k3 mamba metadata preparation, 6.6~7.6x kernel performance improvement ( #52388 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-08-25 11:00:28 -04:00
7ca336929c
[Model] Remove ten deprecated model architectures ( #53608 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Xianbao QIAN <xianbao.qian@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-24 17:06:00 -07:00
da329cc303
[Bugfix] Fix speculative decoding for short_conv (LFM2) models ( #50272 )
...
Signed-off-by: jens.luecke <jens.luecke@liquid.ai >
Co-authored-by: Tonoken3 <sakamakismile@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-22 13:34:22 +00:00
Canlin Guo and Nick Hill
455edc022b
[ModelRunnerV2] Support prompt embeds ( #42963 )
...
Signed-off-by: gcanlin <canlinguosdu@gmail.com >
Signed-off-by: Canlin Guo <961750412@qq.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-17 21:58:50 +00:00
70afdedc10
[K3] support recoverssm for K3 ( #51855 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-17 19:34:14 +08:00
Taneem Ibrahim
fdab2b10bc
[ModelRunner v2] Support Transformers pooling model ( #52425 )
...
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
2026-08-16 15:25:49 -04:00
Nick Hill
d6f17f3d53
[MRV2] Support attention-free models ( #52374 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-08-14 23:54:50 +00:00
Isotr0py and OpenAI Codex
66728feb1f
[MRV2][Multimodal] Enable encoder cuda graph for model runner v2 ( #49852 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-14 09:33:21 +00:00
7f7a32cfec
[Spec Decode] DSpark confidence-scheduled verification ( #47808 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com >
Signed-off-by: Lucas Wilkinson <wilkinson.lucas@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-08-12 12:27:48 -07:00
Guan-Ming Chiu and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
457a5f34db
[Bugfix][MRV2] Support encoder timing stats in model runner V2 ( #50020 )
...
Signed-off-by: Guan-Ming (Wesley) Chiu <105915352+guan404ming@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-12 00:00:49 +08:00
Tianyu Guo
dd856e48bb
[Bugfix][EPD][Model Runner V2] Skip gather mm embeddings for encoder only instance ( #51222 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
2026-08-06 20:40:47 -07:00
Varun Vinayak Shenoy and Nick Hill
e578de311c
[ModelRunnerV2] Fix scalar Mamba state update with int32 mappings ( #50327 )
...
Signed-off-by: Varun Shenoy <varun.vinayak.shenoy@oracle.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-03 22:05:12 +00:00
Martin Hickey
b2fb83e7ff
[Attention]: Use KVCacheSpec for AttentionMetadataBuilder type hints ( #50148 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-07-31 11:20:18 +00:00
Taneem Ibrahim
43eaefba5a
[ModelRunner V2] Enable sequence pooling for embedding and classification models ( #48791 )
...
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
2026-07-29 06:30:45 -07:00
f51193b9ae
[Kernel][Mamba] Fused-kernel support for align-mode DS-conv state migration with num_accepted_tokens > 1 ( #49291 )
...
Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com >
2026-07-29 18:54:52 +08:00
Li, Jiang
90245f4190
[Bugfix] Fix multi-modal support on CPU MRV2 ( #50073 )
...
Signed-off-by: jiang1.li <jiang1.li@intel.com >
2026-07-28 06:09:48 +00:00
Nick Hill and Benjamin Chislett
439f336212
[Core] Fix gpu<->cpu syncs in MRV2 mamba_hybrid.py ( #49736 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
2026-07-27 02:41:27 +00:00
Nick Hill and Taneem Ibrahim
fdaa0d9e59
[ModelRunner V2] Support encoder-only attention ( #49331 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
2026-07-27 00:38:44 +00:00
Nick Hill and Claude Opus 4.8
de6ec294ef
[Bugfix] Fix DSA crash under breakable piecewise cudagraphs ( #49302 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-07-21 18:28:13 +01:00
b6ff8a2f50
[Core] Add MRV2 virtual-batch PCP for MLA ( #46570 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-07-19 02:53:15 +00:00
wendadawen and Tianyu Guo
69d4f5ef63
[Bugfix][Multimodal] Fix Qwen3-Omni use_audio_in_video with mixed image/video inputs ( #46213 )
...
Signed-off-by: wendadawen <wendadawen@qq.com >
Signed-off-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn >
Co-authored-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn >
2026-07-17 08:31:16 +00:00
251f7e478e
[Model] Add PW CUDA graph support for Inkling [2/N] ( #48822 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-07-16 10:28:56 -07:00
Nick Hill and Claude Opus 4.8
ff8d3488f2
[Bugfix][MRV2] Reset num_accepted_tokens on add_request in all modes ( #48132 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-07-09 18:17:11 +00:00
Dakai An and Nick Hill
65dcde1695
[Bugfix] Fix PD disagg + MTP correctness for Qwen3.5(GDN) ( #47466 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-07 13:51:22 +00:00
Isotr0py and Nick Hill
cc1d020d01
[MRV2] Enable mm prefix bidi attention support on MRV2 ( #46942 )
...
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Signed-off-by: Isotr0py <2037008807@qq.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-05 14:45:29 +00:00
Nick Hill
3e158ae62d
[ModelRunner V2] Fix Mamba2 crash on non-spec-decode ( #47428 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-07-02 07:05:16 -07:00
zhrrr
61ab70ec3b
[Model Runner V2] support mamba hybrid models align prefix cache ( #42406 )
...
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com >
2026-06-29 14:09:16 -07:00