MOVZX and Hinata
e61dcb9143
[Mini-vLLM] SM86-only cut: models, backends, Rust, tests
...
- Keep only Qwen3.5, Qwen3.5-MTP and Qwen4Exp model paths; trim the model registry to 41 entries
- Delete idle attention backends (MLA, FlashInfer, ROCm-only, quant backends) and dead GDN CuTeDSL kernels
- Remove inc/quark quant loaders; keep compressed-tensors and Marlin
- Drop the Rust frontend, pooling entrypoints, and idle test suites
- Rebuild kernels for sm_80;sm_86 and strip the extensions
- Verified: full import sweep clean (optional deps only) and a dummy-load Qwen3.5 engine run on RTX 3090 via the Triton GDN path
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-18 19:19:56 +07:00
c3de0b37e2
[Core][Spec Decode] Replace EAGLE group heuristics with explicit draft-layer tags ( #61455 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-17 08:22:33 +00:00
b965ee564d
[Chore] Use upstream Transformers DeepseekV4Config ( #61392 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-15 01:08:08 -07:00
Woosuk Kwon and Claude Opus 5.5
8530aed2b7
[Spec Decode] Remove Medusa and MLP speculator support ( #61701 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 17:29:23 -07:00
02e55d2af3
[KV Cache] GLM-5.3-Flash: use the generic packed KV layout; CircularBufferSpec tail with speculative ring slots ( #57169 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-10-09 15:48:37 -04:00
927c87b347
[PCP][DCP] Support DCP target model with non-DCP Dspark ( #56723 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-26 18:12:35 -04:00
Michael Lapshin
d421a2e8fc
[Perf] Remove CPU-GPU sync in heterogeneous vocabulary speculative decoding ( #57396 )
2026-09-22 18:34:19 -04:00
siyu and Isotr0py
c723a831a8
[Fast Start] Cache the MTP draft model in a separate daemon group ( #57312 )
...
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-09-22 03:13:38 +00:00
Cyrus Leung
c8824a1e42
[MM] Move supports_multimodal_inputs and cache out of registry ( #57913 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-09-21 07:43:07 -07:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
Nick Hill
f2aad6aa70
[MRV2] Buffer util simplifications ( #56888 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-09-15 20:36:51 +00:00
Michele Campi
d392ac836e
[Doc][Metrics] Fix spec-decode PromQL examples to use the exposed names ( #55016 )
...
Signed-off-by: Michele Campi <michele.campi@outlook.com >
2026-09-14 21:15:28 +00:00
aed894c190
[6/N][warmup][DSv4] Migrate sampling, and DFlash JIT kernels ( #56323 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-12 17:24:24 -07:00
Wentao Ye
5fe77aecfc
[Deprecation] Deprecate items scheduled for 0.29 ( #55353 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-11 08:17:02 -07:00
fe26c705f7
[Model] Add Bailing V3 VL support ( #55921 )
...
Signed-off-by: zexplorerhj <zhjoneson@163.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-10 02:46:56 -07:00
Harry Mellor and Claude Opus 5
719284fe15
[Core][Model] Unify XD-RoPE into M-RoPE and derive the channel count ( #56078 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-09 09:23:30 -07:00
Jiangyun Zhu
98ed0856f3
[Model] add GLM-5.3-Flash support ( #53906 )
2026-09-03 09:40:35 -07:00
Andreas Karatzas and OpenAI Codex
e52407ef4c
[ROCm][CI] Add MTP and other spec-decode acceptance coverage ( #53399 )
...
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-02 03:27:05 -05:00
Zihan Zhang and Claude Opus 5
d61b6e1878
[Bugfix][Spec Decode] Take the DFlash draft's RoPE layout from its own config ( #54373 )
...
Signed-off-by: SubSir <tiancaizhangdaxian@sjtu.edu.cn >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-08-31 18:13:20 -04:00
fanxingran
080a66a69c
[Bugfix][ROCm] Capture CUDA graphs on the current stream ( #53818 )
...
Signed-off-by: fanxingran <xingran.fan@amd.com >
2026-08-26 23:16:45 +08:00
danisereb
a7195188a4
Add MTP support for Nemotron VL models ( #53121 )
...
Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com >
2026-08-24 00:59:56 +00:00
Woosuk Kwon
b09bd69b5b
[Model][NVIDIA] Route DSA models to the CUDA non-compiled path ( #52861 )
2026-08-19 00:13:52 -07:00
6adad08767
Add Muse Glimmer model support ( #51655 )
...
Signed-off-by: Tiezhen Wang <tiezhen@inferact.ai >
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
Signed-off-by: zixi-qi <zixi@inferact.ai >
Co-authored-by: Beto de Paola <betodepaola@meta.com >
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
Co-authored-by: zixi-qi <zixi@inferact.ai >
2026-08-14 12:28:56 +08:00
wangxiyuan and Benjamin Chislett
355a338b8f
[BugFix][SpecDecode] Fix dspark parallel_drafting_token_id init bug ( #51602 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
2026-08-10 13:33:10 -04:00
Nick Hill
9b0afeb4f6
[Perf] Avoid some more unnecessary GPU<->CPU syncs ( #51458 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-08-08 17:13:26 -07:00
Raphaël Rialland
4a3447d200
[Bugfix][Model Runner V2] Restore multimodal draft capability detection ( #50417 )
...
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai >
2026-08-03 14:18:14 -07:00
+13
aeeb36b1f1
[New model] Kimi K3 ( #50000 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Summer Yang <girasoleyang@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Bowen Wang <abmfy@icloud.com >
Co-authored-by: gnovack <novackgm@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com >
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-07-30 03:49:58 -07:00
32e657e689
[BugFix] eagle draft max position embeddings ( #49343 )
...
Signed-off-by: JaredforReal <w13431838023@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-07-29 01:24:47 -07:00
Siddhant Bharti and Claude Fable 5
0b6aa3c47c
[Bugfix][Spec Decode] Size DFlash query buffers for cudagraph-padded batches ( #50065 )
...
Signed-off-by: siddhant-bharti <sbharti@together.ai >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-07-28 19:34:22 +00:00
Giancarlo Delfin
272abd5f48
[Tests][Spec Decode] Add gemma4 MTP acceptance rates test ( #47920 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-07-27 16:15:54 -07:00
Woosuk Kwon
213f681f81
Revert "[Perf][GLM-5.2] Blackwell decode optimizations" ( #49768 )
2026-07-24 16:04:59 -07:00
dd72658e7d
[Perf][GLM-5.2] Blackwell decode optimizations ( #48597 )
...
Signed-off-by: Peiyuan Zhou <peiyuanzhou1994@gmail.com >
Signed-off-by: zhou <zhou@zhoudeMacBook-Pro.local >
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: zhou <zhou@zhoudeMacBook-Pro.local >
Co-authored-by: stefankoncarevic <Stefan.Koncarevic@amd.com >
2026-07-23 21:36:27 -07:00
Evan Takahashi and Michael Goin
b2b8f679d0
[Bugfix][Spec Decode] Restrict embedding-width share guard to EAGLE drafts ( #47953 )
...
Signed-off-by: Evan Takahashi <evan10takahashi@gmail.com >
Co-authored-by: Michael Goin <mgoin64@gmail.com >
2026-07-21 10:53:37 -07:00
Wentao Ye and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f17be06fbe
[Perf] Optimize clamp to clamp_ ( #48143 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-16 18:41:07 -04:00
Michael Goin and Claude Opus 4.8
8bfd683901
[Spec Decode] Add kv_cache_dtype to speculative_config to control separately from target ( #48787 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
2026-07-15 23:49:23 -06:00
Michael Goin and Claude Fable 5
ecf4aa5ce2
[Bugfix] Fix FlashInfer non-causal draft attention (DFlash/DSpark) on Blackwell ( #48167 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-07-15 12:44:01 -07:00
Kunshang Ji and Harry Mellor
07f9baf756
Revert "[Platform] Replace torch.cuda.Event with torch.Event ( #47140 )" ( #47668 )
...
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-06 14:18:33 +01:00
d2ec433e37
[XPU] Fix Eagle3 initialization on XPU ( #43957 )
...
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-07-06 08:46:05 +08:00
adamkbaranowski
4c3c64fcf7
Add Laguna XS.2.1 DFlash drafter support ( #46853 )
...
Signed-off-by: Adam Baranowski <adam.baranowski@poolside.ai >
2026-07-02 18:09:27 -07:00
3af8789559
[Feature] Universal speculative decoding for heterogeneous vocabularies (TLI) ( #38174 )
...
Signed-off-by: wan-danfeng <wandanfeng0802@gmail.com >
Signed-off-by: Wonderful <wandanfeng0802@gmail.com >
Co-authored-by: Wan_DF <wonderful199082@126.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
2026-07-02 01:34:20 -07:00
Giancarlo Delfin
d63c8e9444
[BugFix][Spec Decode] Compact shared topk indices buffer after first MTP draft step ( #47238 )
2026-07-01 21:38:51 -07:00
Kunshang Ji and Harry Mellor
e840f0d3f5
[Platform] Replace torch.cuda.Event with torch.Event ( #47140 )
...
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-06-30 08:39:59 -07:00
Ilya Markov and Markov Ilya
a2abce646f
[EPLB] Mask padding in EPLB load recording ( #38128 )
...
Signed-off-by: ilmarkov <markovilya197@gmail.com >
Signed-off-by: Markov Ilya <markovilya19@gmail.com >
Co-authored-by: Markov Ilya <markovilya19@gmail.com >
2026-06-28 19:43:58 -07:00
guybd and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2396d91e93
[CPU][Spec Decode] Enable DFlash SD for CPU ( #44029 )
...
Signed-off-by: guybd <guy.boudoukh@intel.com >
Signed-off-by: Guy Boudoukh <guy.boudoukh@intel.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-25 15:32:48 +08:00
Benjamin Chislett
4c5bc41ba6
[Bugfix][Spec Decode] Fix probabilistic sampling for parallel drafting ( #45956 )
...
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com >
2026-06-24 05:36:23 +00:00
Benjamin Chislett
89bd2c14d3
[Spec Decode] Add Qwen3 architecture support for EAGLE3 ( #43132 )
...
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com >
2026-06-21 13:55:26 -07:00
Nick Hill
7df3d7dada
[Core] Ensure memory is pinned prior to async h2d copy ( #45424 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-06-20 20:02:24 -07:00
Jared Wen
ab66606993
[bugfix]Indexer init skip and MTP TopK share for iteration ( #45895 )
...
Signed-off-by: JaredforReal <w13431838023@gmail.com >
2026-06-19 09:57:51 +08:00
Jonathan Mamou and Li, Jiang <jiang1.li@intel.com>
351c72d6e5
[CPU] Skip Triton kernel monkey-patches when Triton-CPU is available ( #44991 )
...
Signed-off-by: jmamou <jonathan.mamou@intel.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-06-18 18:59:30 +08:00