MOVZX and Hinata
81074e94a3
[Mini-vLLM] Cut dead subsystems: models, entrypoints, backends, utils, tests
...
- Drop dead model modules, keeping files live imports require
- Remove anthropic, cohere, codegen entrypoints and batch STT modes
- Delete unused attention backends, MoE experts, and NPU/XPU util leaves
- Strip dead third_party trees and ROCm leaves with no CUDA path
- Remove mooncake EC connectors: dependency missing since branch base
- Re-audit verified by full import sweep and Qwen3.5 smoke test
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 00:20:48 +07:00
MOVZX and Hinata
e61dcb9143
[Mini-vLLM] SM86-only cut: models, backends, Rust, tests
...
- Keep only Qwen3.5, Qwen3.5-MTP and Qwen4Exp model paths; trim the model registry to 41 entries
- Delete idle attention backends (MLA, FlashInfer, ROCm-only, quant backends) and dead GDN CuTeDSL kernels
- Remove inc/quark quant loaders; keep compressed-tensors and Marlin
- Drop the Rust frontend, pooling entrypoints, and idle test suites
- Rebuild kernels for sm_80;sm_86 and strip the extensions
- Verified: full import sweep clean (optional deps only) and a dummy-load Qwen3.5 engine run on RTX 3090 via the Triton GDN path
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-18 19:19:56 +07:00
Wei Zhao
90d6d871d6
[Attention] Add CuTe DSL backend selection for MLA context prefill ( #62069 )
...
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
2026-10-17 01:21:55 -04:00
b965ee564d
[Chore] Use upstream Transformers DeepseekV4Config ( #61392 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 01:08:08 -07:00
xuebwang
81503550f3
[ROCm] Disable AITER FP8/FP4 BMM for MLA by default ( #53123 )
...
Signed-off-by: xuebwang-amd <xuebwang@amd.com >
2026-10-14 10:46:02 +00:00
Matthew Bonanni and Claude Opus 5.5
292da21f9d
[HiSparse] Read GPU-resident KV directly for mixed and BF16 prefills ( #60417 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-13 16:40:20 -04:00
31e90ac370
[Attention] Enable FA4 sparse MQA decode on Blackwell ( #55866 )
...
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-10-13 09:31:37 -04:00
8055293353
[ROCm][MLA] Cut ~21x host dispatch from the MLA metadata build ( #58381 )
...
Signed-off-by: lifulu <fululi12@amd.com >
Signed-off-by: lifulu <lifulu@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: lifulu <lifulu@users.noreply.github.com >
2026-10-12 13:11:20 -05:00
02e55d2af3
[KV Cache] GLM-5.3-Flash: use the generic packed KV layout; CircularBufferSpec tail with speculative ring slots ( #57169 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-10-09 15:48:37 -04:00
Matthew Bonanni and Claude Opus 5.5
4ed4c4433b
[Bugfix][HiSparse] Drop the unaccounted, write-only prefill mirror staging buffer ( #60429 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 14:00:49 +00:00
Matt and Claude Opus 5.5
9dba339d07
[Bugfix][ROCm] Keep MLA derived weights in place on weight reload ( #60389 )
...
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-07 20:38:06 -05:00
xaguilar-amd and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
9367d8b969
[Bugfix][MLA] Scope DSpark non-causal capability to builder layers ( #57992 )
...
Signed-off-by: Xavier Aguilar <xavier.aguilarfruto@amd.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-07 10:22:32 -05:00
b2863c0de9
[Kimi-K3] Use the fused K/V pack kernel on the decode-context-parallel prefill path ( #56881 )
...
Signed-off-by: Brandon Zhang <31413216+brandonfzhang@users.noreply.github.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-10-06 00:13:55 +00:00
Qiu Chunshuo and Lucas Wilkinson
8486f1ebfe
[Perf][PCP] Shard decode requests across PCP ranks ( #52162 )
...
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-10-01 23:50:13 -04:00
Varun Sundar Rabindranath
5463fe4962
[KV-Offloading][TP] : Expand replicated_layout detection to multi-group MLA ( #57652 )
2026-09-30 09:11:03 -04:00
Matthew Bonanni and Claude Opus 5.5
2798f66860
[Bugfix][HiSparse] Resolve MTP verification rows with a union residency kernel ( #59235 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 03:01:45 +00:00
Wentao Ye
c88f4824ff
[GLM5.3 Perf] Optimize glm 5.3 metadata op, 1.6~4.8x kernel level performance improvement ( #58450 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-25 13:15:18 -04:00
29468dde8b
[Bugfix][KV Cache][MLA] Align packed block strides for V3.2 sparse MLA ( #55528 )
...
Signed-off-by: lz <145014769+200lz@users.noreply.github.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-25 05:33:36 +00:00
Jhao-Ting Chen
4fb767ff0a
[Kernel] Resubmit PR 48666 - Gemma4 FP8 KV FA4 head dim 512 backend selection ( #53175 )
...
Signed-off-by: Jhao-Ting Chen <jhaotingc@nvidia.com >
2026-09-23 20:29:06 -07:00
aoshen02 and Yongye Zhu
973a3be780
[Refactor][Quantization] Make FP8 and MLA weight transforms reusable pure functions ( #57732 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
2026-09-23 01:16:20 -07:00
88aa0d287d
[Spec decode] Support variable-length decode for Kimi-K3 adaptive ver ( #52988 )
...
Signed-off-by: Albert Cheng <albecheng@nvidia.com >
Signed-off-by: Albert Cheng (Engrg-Hardware 1) <albecheng@login-bia01.bia.clusters.nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
2026-09-22 20:43:32 -07:00
1c0eee919d
[ROCm][Model][Bugfix] Enable GLM-5.2-MXFP4 on the deepseek_v32 path and fix sparse attention correctness ( #51915 )
...
Signed-off-by: Jack Hu <Jack.Hu@amd.com >
Signed-off-by: Jack Hu <jack.hu@amd.com >
Signed-off-by: Douglas Lehr <Doug.Lehr@amd.com >
Co-authored-by: James E T Smith <jamesETsmith@users.noreply.github.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
Co-authored-by: Douglas Lehr <Doug.Lehr@amd.com >
2026-09-22 18:08:51 -05:00
a369a7becc
[Perf][Attention] Reduce GLM sparse MLA preparation overhead ( #57458 )
...
Signed-off-by: Summer Yang <girasoleyang@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com >
2026-09-21 17:24:25 -07:00
Yuwen Zhou and Li, Jiang <jiang1.li@intel.com>
b346479065
[CPU] Add CPU FP8 W8A8 linear/MoE support ( #49942 )
...
Signed-off-by: yuwenzho <yuwen.zhou@intel.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-09-18 17:59:42 +08:00
Shanshan Shen
bc0f47cd03
[ROCm][Bugfix] Revert #56433 + #51692 to fix accuracy breakdown for DeepSeek-V4 ( #57132 )
...
Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com >
2026-09-16 18:58:23 +00:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
Nick Hill
f2aad6aa70
[MRV2] Buffer util simplifications ( #56888 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-09-15 20:36:51 +00:00
Ronald
e0c04c7b4d
[Bugfix][Models] Fix OpenPangu sleep mode with static sinks ( #53696 )
...
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
2026-09-14 21:05:47 +08:00
238cb2b191
[Perf][GLM-5.3-Flash] Dense/masked-MHA sparse prefill for the NoPE (256, 0, 256) layout + skip the NoPE K concat ( #55738 )
...
Signed-off-by: Jared Wen <jaredwen@inferact.ai >
Signed-off-by: Jared Wen <w13431838023@gmail.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-14 09:36:39 +00:00
ebe1dec2da
[PCP][DCP] Enable PCP+DCP on sparse-MLA models ( #56157 )
...
Signed-off-by: patryk.saffer <patryk.saffer@blc-login-1.blc1.mistralcompute.ai >
Signed-off-by: patryk.saffer <patryk.saffer@blc-login-0.blc1.mistralcompute.ai >
Signed-off-by: PatrykSaffer <patryk.saffer@mistral.ai >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: patryk.saffer <patryk.saffer@blc-login-1.blc1.mistralcompute.ai >
Co-authored-by: patryk.saffer <patryk.saffer@blc-login-0.blc1.mistralcompute.ai >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Patryk Saffer <PatrykSaffer@users.noreply.github.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-12 23:24:44 +00:00
d43bb2f37f
[3/N] HiSparse: host-resident sparse-MLA decode hot-buffering ( #53781 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Signed-off-by: Matthew Bonanni <matthew@neuralmagic.com >
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-09-12 00:36:47 -04:00
Andy Lo
dc07f1638f
[Bugfix][MLA] Read sparse model settings from text config ( #56160 )
...
Signed-off-by: Andy Lo <andy@mistral.ai >
2026-09-11 18:02:13 +00:00
6fe67cbbf3
[Spec][V2] Support MTP speculative decoding under pipeline parallelism ( #46994 )
...
Signed-off-by: Chris Eastwood <chris.eastwood@pwn4g3.dev >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-11 17:55:21 +00:00
Shanshan Shen
127143e27f
Revert "[Rocm][Kimi-k3] Fix pipeline_parallel support for the kimik3 DCP mode ( #53664 )" ( #56429 )
...
Signed-off-by: Shanshan Shen <87969357+shen-shanshan@users.noreply.github.com >
2026-09-11 09:17:26 -07:00
Wentao Ye
5fe77aecfc
[Deprecation] Deprecate items scheduled for 0.29 ( #55353 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-11 08:17:02 -07:00
Matt
b4da4d17ae
[ROCm][Bugfix] Fix AITER preshuffled FP8 block-scale kernel ( #56433 )
...
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com >
2026-09-11 11:44:57 +00:00
Kevin H. Luu and OpenAI Codex
1cf6555214
[Bugfix] Pin EPLB and MLA host-to-device transfer buffers ( #56138 )
...
Signed-off-by: khluu <khluu000@gmail.com >
Signed-off-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-09-11 01:57:47 -07:00
Jared Wen and Claude Fable 5.1
1768273c13
[Perf][GLM-5.3-Flash] Decode hot-path cleanups: strided KDA recurrent inputs, NoPE MQA query without concat, no duplicate router GEMM ( #55736 )
...
Signed-off-by: Jared Wen <jaredwen@inferact.ai >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-10 16:28:33 +08:00
Wei Zhao and OpenAI Codex
8c87c333b8
[Perf] Fix TRTLLM ragged prefill perf regression ( #55499 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-09-09 20:38:20 +00:00
d8d53f17c2
[Rocm][Kimi-k3] Add pipeline_parallel support for the kimik3 model ( #53664 )
...
Signed-off-by: haic0 <149741444+haic0@users.noreply.github.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Shanshan Shen <467638484@qq.com >
2026-09-09 07:50:24 -07:00
d29c88f162
[Core] Let SWA layers take the primary block size to avoid inflating the KV block LCM ( #53007 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-08 16:44:43 -04:00
Jee Jee Li
199cb9b964
[Kernel] Remove unused fake implementation ( #55535 )
...
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
2026-09-06 17:54:32 -07:00
Nick Hill
8277c42e4c
[Perf] Ensure async h2d copies are pinned in more places ( #55202 )
2026-09-04 17:17:01 -07:00
5690b02c03
[Quantization] Support online quantization with partially pre-quantized checkpoints ( #51392 )
...
Signed-off-by: Felix Marty <Felix.Marty@amd.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
2026-09-04 11:46:06 -07:00
Jiangyun Zhu
98ed0856f3
[Model] add GLM-5.3-Flash support ( #53906 )
2026-09-03 09:40:35 -07:00
7a977c0699
[ModelOpt] Redesign the LinearMethod classes using the generic QuantKey-driven method ( #49381 )
...
Signed-off-by: Juhi Mittal <juhim@nvidia.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
2026-09-01 16:28:56 -04:00
sychen52
7c5dc571cb
[Attention][DSA] Enable W4A16 DSA ( #51724 )
...
Signed-off-by: Shiyang Chen <shiychen@nvidia.com >
2026-09-01 12:37:47 +08:00
Roberto L. Castro and Codex
e16b5e518d
[1/N][warmup][DSv4] Migrate generic MLA metadata and indexing kernels ( #50175 )
...
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Co-authored-by: Codex <codex@openai.com >
2026-08-31 23:26:05 -04:00
Nicolò Lucchesi
7f4793eaa3
[Nixl][PD] DCP support for MLA models ( #50611 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-08-29 16:03:53 +00:00