MOVZX and Hinata
e61dcb9143
[Mini-vLLM] SM86-only cut: models, backends, Rust, tests
...
- Keep only Qwen3.5, Qwen3.5-MTP and Qwen4Exp model paths; trim the model registry to 41 entries
- Delete idle attention backends (MLA, FlashInfer, ROCm-only, quant backends) and dead GDN CuTeDSL kernels
- Remove inc/quark quant loaders; keep compressed-tensors and Marlin
- Drop the Rust frontend, pooling entrypoints, and idle test suites
- Rebuild kernels for sm_80;sm_86 and strip the extensions
- Verified: full import sweep clean (optional deps only) and a dummy-load Qwen3.5 engine run on RTX 3090 via the Triton GDN path
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-18 19:19:56 +07:00
Wentao Ye
8e25bbca19
[Refactor] Remove dead code multiple places ( #61843 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-10-16 15:01:07 +00:00
Stefano Castagnetta and Kevin H. Luu
784d1fff0c
[MRV2][Spec Decode] Quantized draft lm_head for drafters that share the target head ( #59973 )
...
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 22:06:40 +00:00
e711723352
[ROCm] gfx1250 optimizations and model enablement ( #58800 )
...
Signed-off-by: jpvillam <juan.villamizar@amd.com >
Signed-off-by: Jaden Mathias <jaden.mathias@amd.com >
Signed-off-by: Juan Villamizar <jpvillam@amd.com >
Signed-off-by: jpvillam <jpvillam@amd.com >
Signed-off-by: Omar Muhammad <omar.muhammad@amd.com >
Signed-off-by: Daniel <danichan@amd.com >
Signed-off-by: Juan Villamizar <100237675+jpvillam-amd@users.noreply.github.com >
Co-authored-by: Jaden Mathias <jaden.mathias@amd.com >
Co-authored-by: jpvillam <jpvillam@amd.com >
Co-authored-by: Omar Muhammad <omar.muhammad@amd.com >
Co-authored-by: Daniel <danichan@amd.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Daniel Chan <Daniel.Chan@amd.com >
2026-10-15 14:51:57 -07:00
Rohan Potdar and Claude Opus 5.5
105605b8e9
[ROCm] Support DeepSeek-V4 hash routing in AITER Triton W4A16 MoE ( #60282 )
...
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 14:10:12 -05:00
4f49b55c1f
[MoE] Shape-based padding for FlashInfer TRT-LLM MXFP8 experts (gated and non-gated) ( #61293 )
...
Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-10-15 13:44:11 -04:00
984c242ede
[Quantization] Support compressed-tensors MXFP4 MoE with FP8 activations via Mxfp4MoEMethod ( #59815 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-14 13:33:35 -07:00
fe98d90a9d
[Dependency] Update Humming and remove NVFP4 scale workaround ( #61556 )
...
Signed-off-by: jinzhen.ljz <jinzhen.ljz@antgroup.com >
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-14 13:32:36 -07:00
Ashish Rao and Vadim Gimpelson
b4b7eca0ef
[Bugfix][Quantization] Clamp FP8 scales before UE8M0 rounding ( #58032 )
...
Signed-off-by: Ashish Rao <asrao@nvidia.com >
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com >
2026-10-14 13:56:42 +00:00
c9bfa574ce
[Online quantization] Extend re-quantization API to MOE (showcase NVFP4 modelopt/CT/quark to any online MOE method) ( #60276 )
...
Signed-off-by: Felix Marty <Felix.Marty@amd.com >
Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com >
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com >
Co-authored-by: Tan Pin Siang <tanpinsiang@gmail.com >
2026-10-14 15:49:37 +08:00
Stefano Castagnetta and Kartikey Vyas
4e0681ed56
[Quantization] Support packed NVFP4 Qwen4Exp PLE embeddings ( #56273 )
...
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
Co-authored-by: Kartikey Vyas <kartikey@artificialanalysis.ai >
2026-10-14 08:24:55 +08:00
yifanFengg and Claude Fable 5.1
b33cb81237
[Bugfix] Keep compressed-tensors wNa16 linear layers batch invariant ( #60524 )
...
Signed-off-by: yifanFengg <evanfengff@gmail.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-13 15:43:40 -04:00
6849ca696a
[Quantization] Add ModelOpt IQ2_XS, IQ2_XXS, and Q8_0 support via b12x ( #59604 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Signed-off-by: Luke Alonso <lalonso@gmail.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
Co-authored-by: Luke Alonso <lalonso@gmail.com >
2026-10-13 09:06:33 -07:00
e0317b28ce
[Feature] Support batch invariance on ROCm ( #52231 )
...
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-10-12 10:53:26 -05:00
9f3d70e081
[Quantization] Enable shared expert fusion compatibility with online shared_expert quantization (showcase: along Quark MXFP4 routed experts) ( #55686 )
...
Signed-off-by: Felix Marty <Felix.Marty@amd.com >
Signed-off-by: fxmarty-amd <felmarty@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-10-08 12:04:46 -04:00
JartX
994b8b77be
[Bugfix][ROCm] RDNA3 W4A16 MoE: handle the N-first CT weight layout ( #60388 )
...
Signed-off-by: JartX <sagformas@epdcenter.es >
2026-10-08 15:06:39 +00:00
stefankoncarevic and Andreas Karatzas
814301b268
[Bugfix][Quant] Read safetensors headers of checkpoints with other file names ( #60411 )
...
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-10-08 01:38:41 -05:00
f38c6778ce
[quantization] Add per-token NVFP4 quantization MoE support for ReLU2 ( #56740 )
...
Signed-off-by: Xuanteng Huang <xuantengh@nvidia.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-10-07 22:01:09 +00:00
Jinzhen Lin and Kevin H. Luu
32fbfa15e8
[Bugfix][Quantization] Accept folded NVFP4 scales for Humming A16 ( #60337 )
...
Signed-off-by: jinzhen.ljz <jinzhen.ljz@antgroup.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-07 11:04:43 -07:00
5ab4e4445b
[Perf][MoE] Skip padded work in block-FP8 DeepGEMM experts ( #59128 )
...
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-06 18:49:57 -05:00
d1f3d8b870
[Quantization] Prefer Humming before Marlin backends on SM90 ( #56997 )
...
Signed-off-by: jinzhen.ljz <jinzhen.ljz@antgroup.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-06 12:31:35 -07:00
f8d92cae41
[Quantization] Layer re-quantization for linear layers through online quantization API (MXFP8 -> FP8 PTPC showcase) ( #55684 )
...
Signed-off-by: Felix Marty <Felix.Marty@amd.com >
Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com >
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com >
Co-authored-by: Tan Pin Siang <tanpinsiang@gmail.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-10-06 11:38:18 -05:00
40c86ae1ce
[Kernel] Enable OAI Triton MXFP4 MoE on SM12x ( #58877 )
...
Signed-off-by: lazymio <mio@lazym.io >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Shengqi Chen <harry-chen@outlook.com >
2026-10-06 01:49:13 -04:00
Markus Hartikainen
b0e21b3083
[ROCm][Kimi-K3] Add opt-in gfx942 MXFP4-to-int4 conversion ( #51274 )
...
Signed-off-by: Markus Hartikainen <markus.hartikainen@amd.com >
Signed-off-by: Hartikainen <mahartik@amd.com >
2026-10-03 14:28:02 +08:00
Julien Debache
111f71a6db
[feat] FlashInfer CuteDSL MegaMoE integration ( #54049 )
...
Signed-off-by: jdebache <jdebache@nvidia.com >
2026-10-02 15:43:33 -04:00
jackLei and Misha Goin
10f6ab01a9
[Perf] Use value-only reduction for native per-token FP8 quantization ( #59800 )
...
Signed-off-by: jackLei0901 <841747318@qq.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-10-02 18:37:17 +00:00
4c4003bb5f
[Bugfix] Bind routed-experts capture to the MoE layer, not the kernel ( #59455 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-02 15:50:38 +00:00
aoshen02 and Claude Opus 5.5
c91dccc080
[Bugfix][Quark] Pass grouped-routing arguments to OCP MX monolithic kernels ( #59752 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-02 13:46:48 +00:00
Rajeev Warrier and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
30e956fad3
[Docs] Fix stale W4A16 NVFP4 default kernel in ModelOpt docs ( #59135 )
...
Signed-off-by: Rajeev Warrier <warrier.rajeev.95@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-02 13:21:43 +00:00
pmanczak and Pawel Manczak
bdfb02674f
[XPU][Kernel] Tune Triton W8A8 block-FP8 GEMM for Intel B70 ( #56063 )
...
Signed-off-by: Pawel Manczak <203671809+pmanczak@users.noreply.github.com >
Co-authored-by: Pawel Manczak <203671809+pmanczak@users.noreply.github.com >
2026-10-02 12:56:11 +01:00
e12291d733
[CPU][Zen] Add DA8W4 (W4A8) int4 support for dense and MoE layers ( #54024 )
...
Signed-off-by: R <Ganesh.R@amd.com >
Signed-off-by: Ganesh R <Ganesh.R@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-01 15:18:25 +08:00
0103a9b96f
[Quantization] Add per-token NVFP4 CuTe-DSL MoE backend ( #50030 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: S1ro1 <matej.sirovatka@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
2026-09-30 10:42:03 +00:00
Fangzhou Ai and Cursor Agent
77e8b02c8d
[ROCm][Perf] Add opt-in a4w4 (FP4 activation) MoE for DeepSeek V4.1 on AITER ( #58819 )
...
Signed-off-by: fai <fangzhouai@gmail.com >
Signed-off-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com >
Co-authored-by: Cursor Agent (Claude Sonnet 5) <cursoragent@cursor.com >
2026-09-30 15:07:44 +08:00
f4917dadc8
[Bugfix][Quantization] Stop sleep(level=2) from zeroing compressed-tensors KV scales ( #57163 )
...
Signed-off-by: AlanFokCo <alanfok2868@gmail.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-29 15:34:15 +00:00
9b29370524
[Bugfix] Fix moe_wna16 w13 zero-point shard split for 8-bit asym GPTQ MoE ( #58950 )
...
Signed-off-by: Spandan Tiwari <sptiwari@amd.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-28 23:38:28 +00:00
c01368fd64
[Quant] Use canonical N-first weight format for CT WNA16 MoE ( #52798 )
...
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com >
Co-authored-by: HDCharles <{"message":"Not Found","documentation_url":"https://docs.github.com/rest/users/emails#list-email-addresses-for-the-authenticated-user ","status":"404"}>
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-09-28 11:22:44 -06:00
xiao-llm and Cursor
5840d95284
[Bugfix][ROCm] AMD-Quark mixed-precision DeepSeek-V4.1 support ( #57071 )
...
Signed-off-by: Xiao Yu <xiao.yu.dc@outlook.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-25 20:51:18 -05:00
Yongye Zhu and Claude Opus 5.5
378504a544
[MoE] Defer the TRTLLM-Gen top-k finalize on the modular path ( #58635 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-25 15:38:29 +08:00
Samuel Kim and mgoin
f55419c43c
[Bugfix][Quantization] Fix MXFP8 startup crash on layers below mm_mxfp8 shape limits ( #54223 )
...
Signed-off-by: samuelkim7 <samuelmwkim@gmail.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
2026-09-25 02:54:11 +00:00
3652f35e7c
[Bugfix][Quantization] Refresh online NVFP4 scales before reload post-processing ( #57954 )
...
Signed-off-by: S1ro1 <matej.sirovatka@gmail.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-24 18:36:08 -07:00
Misha Goin
4f72bd0152
[Bugfix][Quantization] Give LM heads standard linear metadata ( #58444 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
2026-09-24 16:39:49 -07:00
90a9515006
[Cleanup] Remove online quantization support in fp8.py in favor of online shorthands ( #53585 )
...
Signed-off-by: Felix Marty <Felix.Marty@amd.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-24 12:48:14 -07:00
Misha Goin
cd06e81c41
[Bugfix][Quantization] Add Humming to the W4A8 (INT4xFP8) MoE oracle ( #58427 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
2026-09-24 12:37:33 -07:00
Harry Mellor and Claude Opus 5.5
d881812500
[Chore] Use Transformers v5 names and drop redundant processor use_fast ( #58550 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-09-24 14:14:10 +00:00
fxmarty-amd and mgoin
0549e8d0ab
[Quark] Remove quark-specific silent online quantization ( #51800 )
...
Signed-off-by: Felix Marty <Felix.Marty@amd.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
2026-09-23 11:35:30 -04:00
HDCharles
9f07d023d0
[Quantization] Enable humming wNaM asymmetric quant (zero_point) with compressed-tensors ( #46528 )
2026-09-23 10:23:11 -04:00
aoshen02 and Yongye Zhu
973a3be780
[Refactor][Quantization] Make FP8 and MLA weight transforms reusable pure functions ( #57732 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
2026-09-23 01:16:20 -07:00
9127170b88
[Quantization] Select per-token NVFP4 MoE backends explicitly ( #57176 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: S1ro1 <matej.sirovatka@gmail.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-23 07:30:58 +00:00
Misha Goin
25965ee583
[Kernel] Remove AllSpark INT8 W8A16 GEMM backend ( #58001 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
2026-09-22 14:07:34 -07:00
stefankoncarevic and Andreas Karatzas
72675c4706
[Bugfix][ROCm] Dispatch the QuantFP8 CUDA fallback on the class ( #58136 )
...
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-22 17:49:11 +00:00
MOVZXandHinata
Wentao Ye