MOVZX and Hinata
81074e94a3
[Mini-vLLM] Cut dead subsystems: models, entrypoints, backends, utils, tests
...
- Drop dead model modules, keeping files live imports require
- Remove anthropic, cohere, codegen entrypoints and batch STT modes
- Delete unused attention backends, MoE experts, and NPU/XPU util leaves
- Strip dead third_party trees and ROCm leaves with no CUDA path
- Remove mooncake EC connectors: dependency missing since branch base
- Re-audit verified by full import sweep and Qwen3.5 smoke test
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 00:20:48 +07:00
MOVZX
76df77c04e
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-15 21:16:39 +07:00
48f17b77bb
[Bugfix] Reject mismatched float16/bfloat16 KV cache dtype ( #60885 )
...
Signed-off-by: cipheraxat <cipheraxat@users.noreply.github.com >
Co-authored-by: cipheraxat <cipheraxat@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 08:06:59 +00:00
2cc2e68d1c
[Core][Mamba] Remove leftovers of mamba_cache_mode="all" ( #60043 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Jiangyun Zhu <zhujiangyun@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-13 00:58:59 -07:00
MOVZX
ac305c44cb
Merge branch 'main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-09 17:41:13 +00:00
d8fa463e2b
[Bugfix][Prefix caching] Fix inconsistent hybrid Mamba prefix cache boundaries ( #60533 )
...
Signed-off-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-08 18:33:48 +00:00
aditi-amd
385d86a6cf
[Attention] Add UltraQuant 4-bit KV cache backend (FlyDSL D=256) ( #57057 )
...
Signed-off-by: Aditi Rana <aditi.rana@amd.com >
2026-10-06 20:42:52 +00:00
MOVZX
2877121b2d
Merge remote-tracking branch 'upstream/main' into patches
...
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-03 21:59:03 +07:00
cb6458c5ce
[Core] Remove deprecated mamba_cache_mode "all" ( #58997 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Thomas Parnell <tpa@zurich.ibm.com >
2026-10-01 02:59:37 -07:00
Andrii Skliar and Andrii Skliar
deb148e730
[Mamba] Add FlashInfer ReplaySSM support for MTP ( #52928 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
2026-09-30 21:41:17 +00:00
Yifan Qiao and Claude Fable 5.1
4b2e1cfa7b
[Model] Decoder-side SWA bounded replay for DeepSeek-V4.1 ( #58132 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-29 08:25:46 +00:00
Jürgen Schmied and Claude
a80e41d690
[Mamba] Share RecoverSSM commit kernels and helpers across models
...
RecoverSSM's commit-plan and conv-window compaction kernels are model independent.
Move them from models/kimi_k3 into model_executor/layers/mamba/ops/recoverssm.py
(Kimi-K3 keeps its private names as import aliases; kernel source unchanged), add
mamba/recoverssm_utils.py (activation predicate, argument validation) and the
generic spec-row/request mapping helpers to v1/attention/backends/recoverssm_metadata.py.
CacheConfig.use_kda_recoverssm becomes use_recoverssm: the switch now selects RecoverSSM
for any model whose recurrent layers implement it, not only KDA.
No functional change for Kimi-K3.
Signed-off-by: Jürgen Schmied <juergenschmied70@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-27 00:07:04 +02:00
Louie Tsai and Harry Mellor
6dc34b6334
[CPU] Add device-memory-utilization CLI alias ( #56547 )
...
Signed-off-by: louie-tsai <louie.tsai@intel.com >
Signed-off-by: Louie Tsai <louie.tsai@intel.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-09-22 17:37:15 +08:00
roikoren755 and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
49ee12d742
[Misc] Rename --enable-mamba-fine-grained-prefix-cache ( #53945 follow-up) ( #57382 )
...
Signed-off-by: Roi Koren <roik@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-20 09:50:34 -07:00
1dc2d854c1
[Feat][Model] Support encoder-side SWA-bounded replay for DeepSeek-V4.1-Flash ( #56227 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
2026-09-18 15:01:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
f37c550bf6
[Feature][Frontend] Expose effective attention block size for DCP ( #56538 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Signed-off-by: Julien Darve <jdarve@NVIDIA.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-09-16 08:29:43 +00:00
Wentao Ye
5fe77aecfc
[Deprecation] Deprecate items scheduled for 0.29 ( #55353 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-11 08:17:02 -07:00
Nick Hill
e6cb56337b
[Core] MRV2 support for fast-prefill ( #56145 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-09-10 10:36:04 -07:00
263c4ff95f
[Bugfix][Spec Decode] Cache the Mamba state at the block-grid position of EAGLE resume ( #53945 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Signed-off-by: Adam Shaver <ashaver@nvidia.com >
Signed-off-by: akshaver <168006157+akshaver@users.noreply.github.com >
Co-authored-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: roikoren755 <26850796+roikoren755@users.noreply.github.com >
2026-09-08 09:59:58 -04:00
Nick Hill
e55b93f296
[Core] Deprecate "all" mamba cache mode ( #55041 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-09-03 08:56:59 -07:00
Andrii Skliar and Andrii Skliar
adebc41b7e
[Mamba] Add FlashInfer ReplaySSM backend ( #52506 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Signed-off-by: Andrii Skliar <andreyws96@gmail.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
2026-09-01 10:26:43 -04:00
sychen52
7c5dc571cb
[Attention][DSA] Enable W4A16 DSA ( #51724 )
...
Signed-off-by: Shiyang Chen <shiychen@nvidia.com >
2026-09-01 12:37:47 +08:00
Lucas Wilkinson
8bdc70ec7b
[6/N][KV-Cache Layout Refactor] Standardize KV cache layout ( #51718 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
2026-08-21 17:47:49 -07:00
Tyler Michael Smith and OpenAI Codex
017e9f4448
Promote prefix_cache_retention_interval to an argument and change the default to 0 ( #52216 )
...
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-17 13:20:50 +00:00
70afdedc10
[K3] support recoverssm for K3 ( #51855 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-17 19:34:14 +08:00
Wei-Ming Chen
1c94e8dc7d
Add NVFP4 KV 4-over-6 scale search ( #45187 )
...
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com >
2026-08-07 13:22:27 -07:00
Jiangyun Zhu and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f9c74b4b9c
[Mamba] enable prefix cache by default ( #50991 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-04 15:20:51 +00:00
wangxiyuan
dd11df04f3
[Misc] Remove deprecated calculate_kv_scales runtime KV scale calculation ( #49389 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2026-08-03 11:15:13 -04:00
Harry Mellor
3e74c60b9c
[Docs] Use gen-files for generated docs content ( #49587 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-25 14:43:21 +00:00
866fea2b99
[Kernel] ReplaySSM: cache SSM inputs for faster Mamba2 standard decode ( #48018 )
...
Signed-off-by: Johnny-Liou <a897111@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-07-24 09:39:49 -07:00
Jiangyun Zhu and Yifan Qiao
481e481be7
[2/N][Core] support partial prefix cache hit for hybrid model ( #46384 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-07-12 05:37:51 +00:00
Nicolò Lucchesi
d79855eaac
[Docs] kv_sharing_fast_prefill correction ( #47044 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-07-08 01:17:46 -07:00
Nils Matteson and Nils Matteson
d6875196ad
[Bugfix] Exclude kv_cache_memory_bytes from CacheConfig.compute_hash ( #47356 )
...
Signed-off-by: Nils Matteson <nils@thaw.sh >
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com >
Co-authored-by: Nils Matteson <nils@thaw.sh >
2026-07-07 10:46:51 -07:00
sychen52 and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
67ff0ae30f
Support nvfp4 kv with kv-cache-dtype-skip-layers sliding_window ( #42890 )
...
Signed-off-by: Shiyang Chen <shiychen@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-04 02:29:13 +00:00
JartX and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
70749fdcca
[Feature] Triton INT4 per-token-head KV cache quantization ( #40835 )
...
Signed-off-by: JartX <sagformas@epdcenter.es >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-24 10:21:25 +00:00
Ethan Feng
b7f9b6ab27
[Metrics] Add group-aware KV cache capacity to vllm:cache_config_info ( #42206 )
...
The startup log already reports the correct group-aware KV cache capacity for
hybrid models, but Prometheus did not expose matching info in 'vllm:cache_config_info`.
This PR adds kv_cache_size_tokens and kv_cache_max_concurrency.
Signed-off-by: Ethan Feng <ethan.fengch@gmail.com >
2026-06-12 11:49:44 +00:00
Harry Mellor
2c2c966669
Validate against some config fields being set to 0 ( #43794 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-05-27 21:14:49 +00:00
Qi Zhou
504a26ce2b
Support bf16 for mamba ssm cache ( #41680 )
...
Signed-off-by: Qi Zhou <qizzzh@google.com >
2026-05-16 17:54:58 -07:00
+7
4d51588e23
[Feat] DeepSeek V4 Rebased ( #40860 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Signed-off-by: qizixi <zixi@inferact.ai >
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <yongye@inferact.ai >
Co-authored-by: Simon Mo <simon@inferact.ai >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: youkaichao <youkaichao@gmail.com >
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com >
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com >
Co-authored-by: khluu <khluu000@gmail.com >
Co-authored-by: qizixi <zixi@inferact.ai >
Co-authored-by: Zhewen Li <zhewenli@inferact.ai >
2026-04-26 18:31:08 -07:00
Matthew Bonanni and Tyler Michael Smith
96a85c5750
[Startup][UX] Enable CUDAGraph memory profiling by default ( #38284 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
2026-04-21 18:16:59 -04:00
sychen52
6b2b7bd0eb
Add nvfp4 support to reshape_and_cache_flash ( #37332 )
...
Signed-off-by: Shiyang Chen <shiychen@nvidia.com >
2026-04-17 07:28:00 -07:00
Li, Jiang
d02421a7db
[CPU] Refactor CPU affinity and memory management ( #39781 )
...
Signed-off-by: jiang1.li <jiang1.li@intel.com >
2026-04-17 21:01:08 +08:00
f4b42df048
[Attention Backend] TurboQuant: 2-bit KV cache compression with 4x capacity ( #38479 )
...
Signed-off-by: vibhavagarwal5 <vibhavagarwal5@gmail.com >
Signed-off-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: Xinyu Chen <xinyu1.chen@intel.com >
Co-authored-by: Michael Goin <mgoin64@gmail.com >
2026-04-14 19:57:13 -07:00
roikoren755
ecf5ff7ce3
[Mamba] Flashinfer selective_state_update ( #36162 )
...
Signed-off-by: Roi Koren <roik@nvidia.com >
2026-04-14 15:10:58 -04:00
2ce3d0ce36
[Feature] KV cache per-token-head INT8/FP8 quantization ( #38378 )
...
Signed-off-by: JartX <sagformas@epdcenter.es >
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
Co-authored-by: yangyang4991 <yangyang4991@gmail.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
Co-authored-by: Isotr0py <2037008807@qq.com >
2026-04-02 08:13:26 -04:00
Matthew Bonanni
116f4be405
[1/N][Cleanup] Standardize on use of is_quantized_kv_cache ( #38659 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
2026-04-01 04:08:01 +00:00
3b1dbaad4e
[HMA]Fix corner case when hybrid page_size can not be evenly divided issue (blk_size=64,tp=4) ( #37467 )
...
Signed-off-by: Chendi Xue <chendi.xue@intel.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Signed-off-by: Chendi.Xue <chendi.xue@intel.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com >
2026-03-30 16:47:30 +00:00
roikoren755
8e6293e838
[Mamba] Add stochastic rounding support ( #35753 )
...
Signed-off-by: Roi Koren <roik@nvidia.com >
2026-03-30 12:33:49 -04:00
Harry Mellor and gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
21d2b53f88
Remove need for explicit \n in docstring lists for --help formatting ( #38350 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-03-27 08:38:00 -07:00