MOVZX
e0fe5df731
Merge branch 'main' into patches
2026-10-17 08:06:56 +00:00
71c20f73f2
[FlashInfer] Migrate autotune warmup to flashinfer.autotune_v2 managed cache ( #55671 )
...
Signed-off-by: Yang Xu <yanxu@nvidia.com >
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
Signed-off-by: YangXu <yanxu@nvidia.com >
Co-authored-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
Co-authored-by: Kimi Code <noreply@moonshot.cn >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
Co-authored-by: vllm-agent <claw@inferact.ai >
2026-10-17 02:08:28 +00:00
28e92eeff2
[ROCm][Perf][Kimi-K3] Use AITER Gluon chunk_kda for KDA prefill ( #59552 )
...
Signed-off-by: Omar Muhammad <omar.muhammad@amd.com >
Signed-off-by: Nicholas Susanto <nicholas.susanto@amd.com >
Co-authored-by: Nicholas Susanto <nicholas.susanto@amd.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-16 22:58:56 +00:00
Ayushman Singh
de7eeaa87b
[Perf] Share Triton JIT warmup compilation pool across registered kernels ( #61422 )
2026-10-16 08:59:42 -04:00
MOVZX
be95edb7f6
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-16 17:35:42 +07:00
4d58fa254a
[MoE] Unify DeepGEMM MegaMoE experts behind one feature-matrix layer; add NVFP4 ( #61538 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 19:02:01 +00:00
MOVZX
76df77c04e
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:16:39 +07:00
0951fc183c
[ROCm][DSv4.1] Disambiguate ROCm attention class names ( #58689 )
...
Signed-off-by: Tuukka Sarvi <tuukka.sarvi@amd.com >
Co-authored-by: Shanshan Shen <467638484@qq.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 21:38:39 +08:00
simon-veitner-redhat and Claude
ae94e9c9e8
[Bugfix] Don't crash DeepGEMM MegaMoE warmup when get_bf16_mega_gate_config is missing ( #61657 )
...
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-14 18:58:35 +00:00
253e02b95e
[Perf] Warm up DeepGEMM MegaMoE kernels before serving ( #60468 )
...
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com >
2026-10-13 20:17:48 +08:00
Nick Hill
3fe917abb1
[Bugfix][MRV2] Fix FlashInfer attention and ReplaySSM warmups on MRV2 ( #61013 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-10-11 16:04:41 -07:00
f2376cdc4d
[Fast Start] Preload the FlashInfer autotune table on the weight cache daemon ( #60085 )
...
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-10-10 15:48:06 +00:00
Wentao Ye
3709632ff2
[Refactor] Remove dead code multiple places ( #60120 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-10-10 15:41:10 +00:00
SeongJun Lee and Isotr0py
699c892dd5
[Perf] Skip the M-RoPE warmup when every holder of the RoPE fuses it ( #60246 )
...
Signed-off-by: lesj0610 <lesj0610@gmail.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-10-10 06:47:39 +00:00
MOVZX
ac305c44cb
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-09 17:41:13 +00:00
Jiahan Chang (Cyrus)
816173ca9d
[Kernel] Remove WAR for LL CuTeDSL BF16 GEMM ( #60434 )
...
Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com >
2026-10-08 05:38:07 +00:00
fba462e008
[Bugfix]Fix FlashInfer warmup crash: autotune(tuning_buckets=...) must round M up, not down (Invalid MXFP8 split-K tactic with spec-decode) ( #58165 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-07 14:01:21 -04:00
3403e0f176
[Bugfix] Persist FlashInfer autotune cache per rank to fix the rank-0-only cache-hit deadlock ( #57635 )
...
Signed-off-by: Madhav Jivrajani <madhav.jiv@gmail.com >
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
Co-authored-by: NickLucche <nicolo.lucchesi@mistral.ai >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-10-06 17:51:01 +00:00
Chris Leonard and Shengqi Chen
cc73cca9c8
[Build] Migrate vendored DeepGEMM from pybind to TORCH_LIBRARY (abi3) ( #48962 )
...
Signed-off-by: Chris Leonard <chleonar@redhat.com >
Co-authored-by: Shengqi Chen <harry-chen@outlook.com >
2026-10-06 08:39:04 -07:00
edfccc9469
[ROCM][Perf] Add ROCM_SEGMENTED_ATTN attention-backend works across RDNA3/3.5/4 ( #59132 )
...
Signed-off-by: root <jeffaw99@hotmail.com >
Signed-off-by: big-yellow-duck <jeffaw99@hotmail.com >
Co-authored-by: Tan Pin Siang <1716735+tanpinsiang@users.noreply.github.com >
Co-authored-by: vllmellm <190700713+vllmellm@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
2026-10-06 07:53:47 +00:00
MOVZX
2877121b2d
Merge remote-tracking branch 'upstream/main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-03 21:59:03 +07:00
24df2d3fc7
[Model] Add Qwen4Exp to the Qwen GDN Triton warmup ( #56742 )
...
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com >
Co-authored-by: lesj0610 <lesj0610@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-10-02 21:38:15 +08:00
08d77cad7b
[Bugfix] Gate Kimi-K3 KDA warmup on sys.modules to skip Kimi import for non-Kimi models ( #59257 )
...
Signed-off-by: YueshenZ <YueshenZ@users.noreply.github.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: YueshenZ <YueshenZ@users.noreply.github.com >
Co-authored-by: Claude Code <noreply@anthropic.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
2026-10-01 09:25:45 +00:00
Andrii Skliar and Andrii Skliar
deb148e730
[Mamba] Add FlashInfer ReplaySSM support for MTP ( #52928 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
2026-09-30 21:41:17 +00:00
Venky and Claude Opus 5.5
42f0c17ea7
[Bugfix][PP][Spec Decode] MiniMax-M3 EAGLE3 aux-state relay at PP > 1 and per-stage FlashInfer autotune ( #57197 )
...
Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-01 01:14:24 +04:00
lesj0610
521235d6df
Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
...
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com >
2026-09-27 18:25:21 +09:00
Misha Goin and Codex
71891bdb94
[Perf] Parallelize registered CUDA Triton kernel warmup at startup ( #58582 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-24 19:52:00 -04:00
lesj0610
045467c102
Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
...
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com >
2026-09-23 00:40:48 +09:00
f92b78f6ef
[Kernel][DSV4.1] Fuse MXFP8 wo_b GEMM with sequence-parallel reduce-scatter ( #57428 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
Signed-off-by: Canlin <canlinguosdu@gmail.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
2026-09-22 01:07:17 -07:00
lesj0610
5780c0125a
Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
...
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com >
2026-09-20 09:27:57 +09:00
Jiahan Chang (Cyrus) and OpenAI Codex
4e5ffda19d
[Bugfix] Isolate supplemental FlashInfer BF16 autotuning ( #57285 )
...
Signed-off-by: jiahanc <173873397+jiahanc@users.noreply.github.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-17 07:38:58 +00:00
Lucas Wilkinson
bd2d7ae7c1
[Perf] Reuse MoE workspace for DeepGEMM warmup ( #57268 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
2026-09-17 03:01:19 -04:00
lesj0610
6c5552776b
Merge remote-tracking branch 'origin/main' into lesj/qwen4exp-bugfixes
...
Both sides edited the same import block in the Qwen4Exp NVIDIA and AMD
models and nothing else collided. This branch adds
`from ..common.mtp import make_mtp_hidden_buffer`, the helper the MTP
buffer extraction calls; upstream ec4a3a5370 widened the `..config`
import to `ATTENTION_LAYER_TYPES, QSA_LAYER_TYPE, Qwen4ExpConfig` for the
layer-type constants it introduced. The two changes are independent, so
the resolution keeps both lines. Every symbol still has a use outside the
import in both files.
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com >
2026-09-17 08:16:59 +09:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
lesj0610
a66a3e1b4e
Merge remote-tracking branch 'origin/main' into sync/lesj/qwen4exp-bugfixes
...
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com >
2026-09-14 14:36:05 +09:00
Woosuk Kwon
3b533197a9
[Perf] Parallelize mHC pre-norm JIT warmup ( #56683 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
2026-09-13 18:58:20 -07:00
lesj0610 and Claude Opus 5
3b1911c0ca
[Bugfix][Model] Fix Qwen4Exp MTP buffer placement, config parsing, and Triton warmup
...
Three initialization and config defects found serving Qwen4Exp, each with a
regression test.
**The MTP hidden buffer lands on the ambient device.** torch.empty for the
spec-decode hidden buffer takes no device, so it follows whatever device is
current rather than vllm_config.device_config.device. The allocation was
duplicated verbatim in the NVIDIA and AMD backends; it moves to
common/mtp.py so the two cannot drift, and the caller passes is_last_rank
rather than reaching for the process group from shared code.
**M-RoPE keys are reported as unrecognized.** Qwen4Exp drives M-RoPE through
mrope_section and mrope_interleaved, which transformers' rope validator does
not know for rope_type="default", so every load warns about a configuration
the model genuinely uses. Declare them at rope validation on the text config
and on the outer config that mirrors it; every other key still warns.
**transformers' layer_types spelling is unread.** Its Qwen4Exp config
canonicalizes "full_attention" entries to "qwen_sparse_attention" in memory,
so checkpoints re-exported through it serialize that name. vLLM keys QSA off
"full_attention" plus indexer_n_heads, so map the spelling back.
**Qwen4Exp is missing from the Triton warmup set.** Its GDN prefill and QSA
run on other backends, but the kernels this warmup covers -- the gated RMS
norm, the causal conv, the post-conv preparation, the decode state update --
are Triton whichever prefill kernel is chosen, so they compile on the first
prompt instead. qwen4_exp_mtp stays out: kernel_warmup passes the worker's
own model_config, which is the target model's.
tests/models/qwen4_exp/test_config.py: 15 passed.
tests/model_executor/test_qwen_triton_warmup.py: 6 passed.
ruff check, ruff format --check and mypy (Python 3.10) report what they
already report on the base commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com >
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com >
2026-09-14 07:25:01 +09:00
Kevin H. Luu
dec0b5d63f
[Bugfix] Skip Triton autotune inspection without Triton ( #56676 )
...
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com >
2026-09-13 17:05:29 +00:00
Roberto L. Castro and Codex
e52be1a62d
[9/N][warmup][DSv4] Migrate MHC TileLang kernels ( #50178 )
...
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Signed-off-by: Roberto L. Castro <38211239+LopezCastroRoberto@users.noreply.github.com >
Co-authored-by: Codex <codex@openai.com >
2026-09-12 17:25:23 -07:00
aed894c190
[6/N][warmup][DSv4] Migrate sampling, and DFlash JIT kernels ( #56323 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-12 17:24:24 -07:00
7ee8a6dd01
[watermarking] Dual-key gumbel-max watermarking for speculative decoding support ( #56122 )
...
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai >
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Signed-off-by: Tomas Ruiz <tomas.ruiz.te@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Tomas Ruiz <tomas.ruiz.te@gmail.com >
2026-09-12 18:42:01 +00:00
Ayushman Singh and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
7bcca16729
[Bugfix] Fix DeepGEMM FP8 warmup coverage ( #56452 )
...
Signed-off-by: Ayushman Singh <40520701+ayush1399@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-12 14:20:38 -04:00
Itay Alroy
658c8131c7
[Elastic EP] Reuse CUDA graphs across reconfiguration ( #54985 )
2026-09-12 10:26:39 -04:00
d43bb2f37f
[3/N] HiSparse: host-resident sparse-MLA decode hot-buffering ( #53781 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Signed-off-by: Matthew Bonanni <matthew@neuralmagic.com >
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-09-12 00:36:47 -04:00
Roberto L. Castro and Codex
120ec4ebd2
[5/N][warmup][DSv4] Migrate NVIDIA CuTeDSL attention kernels ( #53566 )
...
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Co-authored-by: Codex <codex@openai.com >
2026-09-12 00:05:59 -04:00
e77daef89e
[Model] Support DeepSeek-V4.1-Flash ( #56214 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
Signed-off-by: andyluo7 <andy.luo@amd.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: andyluo7 <andy.luo@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-11 02:11:19 -07:00
cc4210f671
[Perf] Improve BF16x3 router GEMM accuracy and make it default on sm100 ( #55899 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
2026-09-09 07:33:12 -07:00
Roberto L. Castro and Codex
6ddbab03de
[4/N][warmup][DSv4] Migrate common attention kernels ( #50176 )
...
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Co-authored-by: Codex <codex@openai.com >
2026-09-08 10:25:48 -04:00
7fa2c63796
[Bugfix] Autotune FlashInfer deferred MoE decode kernels before CUDA graph capture ( #55377 )
...
Signed-off-by: ming <ming@inferact.ai >
Co-authored-by: Kimi Code <kimi-code@moonshot.cn >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-08 04:02:23 +00:00