MOVZX
e0fe5df731
Merge branch 'main' into patches
2026-10-17 08:06:56 +00:00
Fadi Arafeh and vllm-agent
556ac15eb9
[Bugfix] Remove extra args from CPU's compute slot mapping kernel ( #61992 )
...
Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com >
Co-authored-by: vllm-agent <claw@inferact.ai >
2026-10-17 12:11:33 +08:00
71c20f73f2
[FlashInfer] Migrate autotune warmup to flashinfer.autotune_v2 managed cache ( #55671 )
...
Signed-off-by: Yang Xu <yanxu@nvidia.com >
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
Signed-off-by: YangXu <yanxu@nvidia.com >
Co-authored-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
Co-authored-by: Kimi Code <noreply@moonshot.cn >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
Co-authored-by: vllm-agent <claw@inferact.ai >
2026-10-17 02:08:28 +00:00
Ayushman Singh
de7eeaa87b
[Perf] Share Triton JIT warmup compilation pool across registered kernels ( #61422 )
2026-10-16 08:59:42 -04:00
MOVZX
be95edb7f6
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-16 17:35:42 +07:00
MOVZX and Hinata
18c06ed406
[Offloader] Stream multimodal towers from host RAM with bulk-copy
...
Add --offload-towers: place multimodal tower weights in pinned host
memory and bulk-copy each unit to the GPU only during its forward,
keeping towers out of VRAM between encodes. Unlike UVA zero-copy,
whole-tensor H2D transfers avoid re-reading operand tiles over PCIe
inside the GEMM inner loop. Tower streaming is independent of layer
group scheduling, so group_size may be 0 for tower-only offload.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-16 14:32:14 +07:00
a8be1500ba
[Bugfix][MoE] Restore TP-sharded routed-expert capture for DP=1 ( #61388 )
...
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: Ao Shen <281812371+aoshen02@users.noreply.github.com >
Signed-off-by: Ao Shen <aoshen@inferact.ai >
Co-authored-by: Ao Shen <281812371+aoshen02@users.noreply.github.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Ao Shen <aoshen@inferact.ai >
2026-10-16 00:21:24 +00:00
f8460ef236
[Kernel][DSv4.1] Bump FlashMLA to the 2026.09.30 release and its interleaved V4.1 KV cache ( #61371 )
...
Signed-off-by: juntianl <juntianl@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Kimi Code <noreply@moonshot.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-10-15 13:58:25 -07:00
4d58fa254a
[MoE] Unify DeepGEMM MegaMoE experts behind one feature-matrix layer; add NVFP4 ( #61538 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 19:02:01 +00:00
Mikko Tukiainen and Cursor
47fd76e4ce
[ROCm][Perf] Route softmax MoE top-k, including fused shared-expert sigmoid, through AITER topk_gating ( #56226 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-10-15 17:53:18 +00:00
Matthias Gehre
011fa68ca9
[ROCm][Perf] Pad fp16/bf16 GEMM weight row stride for L2 cache ( #55090 )
...
Signed-off-by: Matthias Gehre <matthias.gehre@amd.com >
2026-10-15 14:26:32 +00:00
MOVZX
76df77c04e
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:16:39 +07:00
b965ee564d
[Chore] Use upstream Transformers DeepseekV4Config ( #61392 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 01:08:08 -07:00
Haiyu Zuo and Kevin H. Luu
ff5a91a456
[Feature][Model Loader] Apply model-owned MTP shard selection to RunAI ModelStreamer ( #60759 )
...
Signed-off-by: 浩韵 <zuohaiyu.zhy@alibaba-inc.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 12:23:11 +08:00
amd-xavierwang and JartX
06265beb25
[ROCm] Enable fp8 query for Triton attention ( #60296 )
...
Signed-off-by: xawang <xawang@amd.com >
Co-authored-by: JartX <sagformas@epdcenter.es >
2026-10-14 15:03:20 -04:00
13dbf96093
[Model Loader] Apply EP weight filtering in InstantTensor ( #61167 )
...
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-10-14 09:10:10 -07:00
Ronald and aoshen02
19aba3e4d5
[Bugfix][RL] Load attention sinks through weight_loader for layerwise reload ( #57797 )
...
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-14 10:17:58 +00:00
6849ca696a
[Quantization] Add ModelOpt IQ2_XS, IQ2_XXS, and Q8_0 support via b12x ( #59604 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Signed-off-by: Luke Alonso <lalonso@gmail.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
Co-authored-by: Luke Alonso <lalonso@gmail.com >
2026-10-13 09:06:33 -07:00
Woosuk Kwon and Claude Opus 5.5
ac71b17517
[Model Loader] Keep norm layers neutral under dummy weight init ( #61323 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-13 08:54:24 -07:00
253e02b95e
[Perf] Warm up DeepGEMM MegaMoE kernels before serving ( #60468 )
...
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com >
2026-10-13 20:17:48 +08:00
Linze Shi
e6e84f350c
[Feature][Pooling] Add offline fixed-length late chunking ( #60164 )
...
Signed-off-by: Linze-Shi <linzeshi0@gmail.com >
2026-10-13 11:16:21 +00:00
Yongye Zhu and Claude Opus 5.5
a6009ea85b
[Kernel][Build] Bump DeepGEMM to the 26/09/30 release and adapt the sparse indexer ( #61149 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-12 23:24:43 -07:00
Woosuk Kwon and Claude Opus 5.5
f47792af7f
[Model Loader] Make dummy weight init dtype-aware ( #61134 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 18:33:26 -07:00
189bbfb9f4
Add MoNe experts pruning support ( #50415 )
...
Signed-off-by: Ilya Boytsov <ieboytsov@nebius.com >
Signed-off-by: Misha Goin <mgoin64@gmail.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Ilya Boytsov <ieboytsov@nebius.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 23:18:43 +00:00
092c35dd15
[Model] Support Cloudflare/clef-omni on /v1/systemone ( #61064 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-10-12 21:27:34 +08:00
Nick Hill
3fe917abb1
[Bugfix][MRV2] Fix FlashInfer attention and ReplaySSM warmups on MRV2 ( #61013 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-10-11 16:04:41 -07:00
tc-mb and Cyrus Leung
c85616efc6
[Model] Support MiniCPM-V 4.7 ( #58674 )
...
Signed-off-by: tc-mb <tianchi_cai@icloud.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-11 10:21:10 +00:00
f2376cdc4d
[Fast Start] Preload the FlashInfer autotune table on the weight cache daemon ( #60085 )
...
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-10-10 15:48:06 +00:00
SeongJun Lee and Isotr0py
699c892dd5
[Perf] Skip the M-RoPE warmup when every holder of the RoPE fuses it ( #60246 )
...
Signed-off-by: lesj0610 <lesj0610@gmail.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-10-10 06:47:39 +00:00
02aa10539f
[Bugfix][ModelLoader] Make enable_weights_track check quantized layers ( #57108 )
...
Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Shanshan Shen <467638484@qq.com >
2026-10-10 10:44:06 +08:00
02e55d2af3
[KV Cache] GLM-5.3-Flash: use the generic packed KV layout; CircularBufferSpec tail with speculative ring slots ( #57169 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-10-09 15:48:37 -04:00
Ning Xie and Simon Mo
a0f28d73a8
[sharded state loader] support pp in sharded state loader ( #52110 )
...
Signed-off-by: Andy Xie <andy.xning@gmail.com >
Co-authored-by: Simon Mo <simon.mo@hey.com >
2026-10-09 11:08:31 -07:00
MOVZX
ac305c44cb
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-09 17:41:13 +00:00
Juan Pérez de Algaba and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
3d52578e13
[Security] Do not retain Phi-4 MM audio attention matrices ( #60614 )
...
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-09 22:23:35 +08:00
Harry Mellor and Claude Opus 5.5
2c99ee9333
[Chore] Remove vendored code made redundant by Transformers 5.16.1 ( #60623 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 09:29:03 +00:00
9f3d70e081
[Quantization] Enable shared expert fusion compatibility with online shared_expert quantization (showcase: along Quark MXFP4 routed experts) ( #55686 )
...
Signed-off-by: Felix Marty <Felix.Marty@amd.com >
Signed-off-by: fxmarty-amd <felmarty@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-10-08 12:04:46 -04:00
Woosuk Kwon
194bd72be7
[Compile] Remove torch.compile-based sequence parallelism and AsyncTP ( #60517 )
2026-10-08 08:28:23 -07:00
Harry Mellor and Claude Opus 5.5
21d9758459
Remove Python 3.10 support now that it is EOL ( #60402 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 02:30:52 -07:00
Jiahan Chang (Cyrus)
816173ca9d
[Kernel] Remove WAR for LL CuTeDSL BF16 GEMM ( #60434 )
...
Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com >
2026-10-08 05:38:07 +00:00
Matt and Claude Opus 5.5
9dba339d07
[Bugfix][ROCm] Keep MLA derived weights in place on weight reload ( #60389 )
...
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-07 20:38:06 -05:00
f38c6778ce
[quantization] Add per-token NVFP4 quantization MoE support for ReLU2 ( #56740 )
...
Signed-off-by: Xuanteng Huang <xuantengh@nvidia.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-10-07 22:01:09 +00:00
e572e02d21
[UX] Skip checkpoint shards an MTP head does not load ( #60152 )
...
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-07 15:16:19 -04:00
fba462e008
[Bugfix]Fix FlashInfer warmup crash: autotune(tuning_buckets=...) must round M up, not down (Invalid MXFP8 split-K tactic with spec-decode) ( #58165 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-07 14:01:21 -04:00
c83935b300
[Bugfix][Model Loader] Support reload_weights with runai_streamer load format ( #58124 )
...
Signed-off-by: Viktor Koukouliev <vkoukouliev@nvidia.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-10-07 14:04:35 +00:00
2a54f6b625
[ROCm][Perf][GLM-5.3-Flash] Enable AITER fused shared experts on gfx950 ( #59221 )
...
Signed-off-by: Jin Tao <jin.tao@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Simon Danielsson <70206058+simondanielsson@users.noreply.github.com >
2026-10-07 20:53:40 +08:00
3403e0f176
[Bugfix] Persist FlashInfer autotune cache per rank to fix the rank-0-only cache-hit deadlock ( #57635 )
...
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com >
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
Co-authored-by: NickLucche <nicolo.lucchesi@mistral.ai >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-10-06 17:51:01 +00:00
Tianyu Guo and OpenAI Codex
8c417a2dae
[Bugfix][Model] Fix Qwen3-Omni interleaved M-RoPE boundary positions ( #59842 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-10-06 03:35:53 +00:00
MOVZX
85c31d47ad
Merge remote-tracking branch 'upstream/main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-03 22:06:10 +07:00
MOVZX
2877121b2d
Merge remote-tracking branch 'upstream/main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-03 21:59:03 +07:00
C
44198f577f
[Bugfix][Model] Fix M-RoPE offset double-count in Qwen3-Omni ( #58890 )
...
Signed-off-by: C1ves <3121985758@qq.com >
2026-10-03 09:00:51 +08:00
MOVZX
