59 Commits
Author SHA1 Message Date
Harry HuangandThien Tran 494663940b [Qwen4Exp] Clean up HC and PLE implementations for Qwen3.8-Flash-Next (#61454)
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-18 19:38:21 +08:00
Andreas KaratzasandClaude 29345f42a6 [ROCm][Qwen4Exp] Support HC and MoE sequence parallelism on ROCm (#61471)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-15 20:11:17 -05:00
77154ca611 [Perf][Model][Qwen4Exp] QSA: decode four FP8 e4m3 bytes per register below sm_89 (#60992)
Signed-off-by: kimseunghyun <hellonypcteam@gmail.com>
Co-authored-by: kimseunghyun <hellonypcteam@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-14 13:06:16 +08:00
Stefano CastagnettaandKartikey Vyas 4e0681ed56 [Quantization] Support packed NVFP4 Qwen4Exp PLE embeddings (#56273)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Kartikey Vyas <kartikey@artificialanalysis.ai>
2026-10-14 08:24:55 +08:00
Tim002andThien Tran 96fc2da315 [Perf][Qwen4Exp] Add SM121 TP=2 merged QSA projection plans (#61437)
Signed-off-by: TimRen01 <HolaTim@hotmail.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-13 22:58:44 +00:00
bc3ddb198a [Model][Qwen4Exp] FP8 QSA main KV cache on Ampere (sm_80, sm_86) (#60602)
Signed-off-by: kimseunghyun <hellonypcteam@gmail.com>
Signed-off-by: kimseunghyun <incongnito12@gmail.com>
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: kimseunghyun <hellonypcteam@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-13 15:30:19 +08:00
Harry Huangandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> c75cb63cd2 [Qwen4Exp] support Sequence Parallelism for Qwen3.8-Flash-Next (#56322)
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-12 20:56:44 +08:00
SeongJun Leeandlesj0610 a98247ab4d [Model] Drop the 4x M-RoPE cache headroom for Qwen3-VL, Qwen3.5 and Qwen3.8 (#59853)
Signed-off-by: lesj0610 <lesj0610@users.noreply.github.com>
Signed-off-by: lesj0610 <lesj0610@gmail.com>
Co-authored-by: lesj0610 <lesj0610@users.noreply.github.com>
2026-10-10 06:51:52 +00:00
Mikko TukiainenandCursor 81c993658e [ROCm][Perf] Fuse main QK-norm/RoPE/gate and KV-cache write into the AMD QSA prepare launch (#59732)
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-09 21:14:49 +08:00
82e89e9944 [Perf][Qwen4Exp] Retune H200 M=4 merged QSA LL-GEMM plan (#60776)
Signed-off-by: zigzagcai <zigzagcai@users.noreply.github.com>
Co-authored-by: zigzagcai <zigzagcai@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-10-09 07:18:50 +00:00
Yifan QiaoandClaude Opus 5.5 240785b82c [Bugfix] Keep the GLM-5.3 kpool tail and Qwen4 QSA ring out of the null block (#59528)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 20:44:04 +00:00
Mikko TukiainenandCursor e2414ec1db [ROCm][Qwen4Exp] Use fused PLE Triton kernels on the AMD backend (#60021)
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-08 15:47:56 +00:00
d9503dc503 [Bugfix][Qwen4Exp] Derive the QSA head_dim from the unsharded head count (#59945)
Signed-off-by: lesj0610 <lesj0610@godoiksan.org>
Co-authored-by: lesj0610 <lesj0610@godoiksan.org>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-07 12:24:40 +00:00
b0b23a48d4 [Bugfix] Fix Qwen4Exp FP8 PLE pinned lookup kernel failing to compile below SM89 (#60142)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-06 01:56:44 +00:00
Mikko TukiainenandCursor d23af12cb1 [ROCm][Perf] Reach the fused QSA pre-indexer from the AMD path (#57947)
Signed-off-by: mjkvaak-amd <mjkvaak-amd@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-05 16:22:29 +00:00
Stefano CastagnettaandThien Tran 4ff028d77e [Bugfix][Qwen4Exp] Honor --kv-cache-dtype-skip-layers in QSA attention (#60023)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-05 12:13:30 +00:00
Stefano CastagnettaandThien Tran 55b80221ed [Perf][Qwen4Exp] Keep the HC up projection on the skinny GEMM path (#60027)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-05 20:09:36 +08:00
Mikko TukiainenCursormergify[bot] <37929162+mergify[bot]@users.noreply.github.com>TJian
edca360f13 [Bugfix][Qwen4Exp] Load PLE tables unquantized under Quark checkpoints (#59443)
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
2026-10-05 10:40:02 +00:00
c04c79e50d [Performance] Add SM121 TP=2 skinny-GEMM plans (#59632)
Signed-off-by: sudhanshu shukla <sudhanshu112233shukla@users.noreply.github.com>
Co-authored-by: sudhanshu shukla <sudhanshu112233shukla@users.noreply.github.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-05 18:27:51 +08:00
042ab0305c [Perf][Qwen4Exp] Merge QSA QKVG and indexer QK projections (#59533)
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: namgyu-youn <namgyu.dev@gmail.com>
2026-10-05 07:03:29 +00:00
Vadim GimpelsonandClaude 4f52fa35ef [Bugfix] Fix Qwen4Exp PLE embedding rejecting INC (AutoRound) checkpoints (#59990)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-05 01:52:06 +00:00
Harry Mellor 9e0b6fcf77 [CI] Bump Transformers version to 5.18.0 (#59621)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-10-04 00:42:35 +00:00
Stefano CastagnettaandBenjamin Chislett f590eb2448 [Perf][Qwen4Exp] Add SM121 TP=1 skinny-GEMM plans (#59753)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com>
2026-10-02 17:17:45 +00:00
Harry MellorandClaude Opus 5 e3cae8d2ac [Model] Use upstream GLM-5.3 and Qwen4-Exp configs and processor (#57387)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-02 02:25:18 +00:00
cb6458c5ce [Core] Remove deprecated mamba_cache_mode "all" (#58997)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Thomas Parnell <tpa@zurich.ibm.com>
2026-10-01 02:59:37 -07:00
Shuolei WangandThien c9578f10ab [Qwen3.8-Flash-Next] Fuse main QK-norm/RoPE/gate and KV-cache write into the QSA pre-indexer launch (#57097)
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com>
Signed-off-by: Thien <thien@inferact.ai>
Co-authored-by: Thien <thien@inferact.ai>
2026-10-01 14:21:06 +08:00
namgyu-younandThien Tran 4ed438b79c [Perf][Qwen4Exp] Add SM100 low-latency decode GEMM plans (#59214)
Signed-off-by: namgyu-youn <namgyu.dev@gmail.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-01 03:40:55 +00:00
Eldar Kurtić c32513b99d Add support for unquantized ngram in CT format (#59431) 2026-09-30 16:08:31 +00:00
Mathew Oddenandopencode+glm-5.3+vllm 91dab0eb7f [ROCm][Perf] Allocate the pinned PLE prefetch buffer lazily (#58797)
Signed-off-by: Mathew Odden <modden@redhat.com>
Co-authored-by: opencode+glm-5.3+vllm <opencode+glm-5.3+vllm@example.com>
2026-09-30 17:43:16 +08:00
cjackalandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 491f44adfa [Model] Extend device-side mm normalization to Qwen3VL/Qwen3.5/Qwen4Next (#51289)
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-29 11:44:54 +08:00
Jiangyun ZhuandOpenAI Codex 20b52e9f5b [Perf][Qwen3.8] Reduce PLE metadata construction overhead (#58114)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-28 20:46:22 +08:00
Thien Tran d6cce94fd4 [Perf][Qwen4Exp] Fuse HC down projection and SiLU on NVIDIA (#58957)
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-09-28 18:51:38 +08:00
Gabriel WuClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
a2be4d3cc7 [Bugfix][Qwen4Exp] Release the profiling KV cache held by QSA key views (#58961)
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-28 08:11:46 +00:00
Thien Tran 7a877ae34d [Qwen3.8-Flash-Next] Avoid memory fragmentation in QSA indexer logits workspace (#57105)
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-09-27 10:11:52 +08:00
267eee54bf [Qwen4Exp][ROCm] PLE n-gram table CPU offload (#57497)
Signed-off-by: Mathew Odden <modden@redhat.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: opencode+deepseek-v4-flash+vllm <opencode+deepseek-v4-flash+vllm@example.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-09-24 23:28:20 -05:00
Juntian LiuClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
48d8880d09 [Bugfix][Qwen4Exp] Keep pinned PLE prefetch ids out of the CUDA graph pool (#58489)
Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-25 10:51:50 +08:00
Harry MellorandClaude Opus 5.5 d881812500 [Chore] Use Transformers v5 names and drop redundant processor use_fast (#58550)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-24 14:14:10 +00:00
Jhao-Ting Chen 4fb767ff0a [Kernel] Resubmit PR 48666 - Gemma4 FP8 KV FA4 head dim 512 backend selection (#53175)
Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com>
2026-09-23 20:29:06 -07:00
Tyler Michael SmithOpenAI CodexOpenAI CodexMisha Goinmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
4f9a41ac94 [CI/Build][Hardware][NVIDIA] Add Rubin CUDA 13.4 nightly images (#55953)
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-17 08:05:31 -07:00
Shuolei WangandRoger Wang 7bbce752b8 [Perf][Model] Qwen4Exp QSA: sm_90 tuning table for _select_config (#57273)
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com>
Signed-off-by: Shuolei Wang <948904026@qq.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
2026-09-17 07:02:24 +00:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
semerandreClaudemergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Jiangyun Zhu
dff1bde84d [Model] Qwen4Exp: fp8_e4m3 main KV cache on the QSA path (#55557)
Signed-off-by: Andrea Semeraro <andrea.semeraro@sezione1.it>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
2026-09-16 15:07:52 +00:00
Harry Mellormergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Claude Opus 5
ec4a3a5370 [CI] Bump Transformers version to 5.17.0 (#56108)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-16 06:34:16 -07:00
3f55ad2f07 [Qwen3.8-Flash-Next] Fuse PLE residual and QSA output gate (#55309)
Signed-off-by: Canlin <canlinguosdu@gmail.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
2026-09-14 12:16:25 +00:00
Harry Huangandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 3116c5d06b [Qwen4Exp] Support UVA PLE-offload and Engram tensor parallelism (#54371)
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-09 07:32:34 -07:00
sychen52mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Isotr0py
60ad959b6f Fix block FP8 MTP in ModelOpt mixed checkpoints (#55513)
Signed-off-by: Shiyang Chen <shiychen@nvidia.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
2026-09-08 22:47:21 +00:00
Thien Tran 94e26dd3dd [Qwen3.8-Flash-Next] Support FP8 indexer cache for QSA (#54890)
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-09-07 12:15:45 +00:00
Thien Tran d9105ea800 [Qwen3.8-Flash-Next] Remove torch.compile for NVIDIA implementation (#55272)
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-09-06 21:57:56 -07:00
Jee Jee Li 199cb9b964 [Kernel] Remove unused fake implementation (#55535)
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
2026-09-06 17:54:32 -07:00
Harry Huang 28e605fb33 [Bugfix][Qwen4Exp] fix state index strides in fused PLE conv (#55375)
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
2026-09-05 14:02:20 +00:00