MOVZX and Hinata
5aa2ac64f8
[Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
...
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.
- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
render routes, and vllm/config/speech_to_text.py. GenerationTask is
generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
Multi, Offloading and Example connectors stay; production uses the
CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
arch gate, declarations and op registrations. The CUDA build never
defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
fail with "The model ... does not exist. Valid aliases: ...".
Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 21:06:19 +07:00
c3de0b37e2
[Core][Spec Decode] Replace EAGLE group heuristics with explicit draft-layer tags ( #61455 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-17 08:22:33 +00:00
Yan Ma
f147949be8
[XPU] limit FA kernel block size to multipleOf 64 ( #57297 )
...
Signed-off-by: Yan Ma <yan.ma@intel.com >
2026-10-12 10:25:32 +08:00
02e55d2af3
[KV Cache] GLM-5.3-Flash: use the generic packed KV layout; CircularBufferSpec tail with speculative ring slots ( #57169 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-10-09 15:48:37 -04:00
fd928e5f3c
[Bugfix] Check batch-invariance support for every attention backend ( #60271 )
...
Signed-off-by: yifanFengg <evanfengff@gmail.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-10-08 11:19:57 -04:00
Harry Mellor
9e0b6fcf77
[CI] Bump Transformers version to 5.18.0 ( #59621 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-10-04 00:42:35 +00:00
f7999d2e44
[Bugfix][HiSparse] Size the KV cache from the groups HiSparse allocates ( #59450 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-10-02 04:11:55 +00:00
dd3bc9c72f
[Perf][MRV2] Reuse Mamba/GDN metadata across KV cache groups ( #58762 )
...
Signed-off-by: Luca Motz <luca.motz@icloud.com >
Co-authored-by: Xin Yang <xyangx@amazon.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-09-28 04:36:58 +00:00
a4eb3f25d6
[Spec Decode] Enable Gemma4 DSpark adaptive verification with FlashInfer ( #57263 )
...
Signed-off-by: zixi-qi <zixi@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-26 17:01:17 +08:00
waizuichougou
a1bf8ac12d
[Bugfix][MRV2] Match fast-prefill padding to active LoRA batches ( #56456 )
...
Signed-off-by: waizuichougou <2082431897@qq.com >
Signed-off-by: waizuichougou <105572299+waizuichougou@users.noreply.github.com >
2026-09-18 22:13:23 +00:00
Juntian Liu and Woosuk Kwon
bf13ecc2f7
[Perf][MRV2] Share token-to-request mappings across KV cache groups ( #57102 )
...
Signed-off-by: Juntian Liu <Juntianl777@gmail.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
2026-09-18 03:14:36 +00:00
Roberto L. Castro and Codex
186a1e6222
[Warmup] Gemma 4 de-JITification ( #55768 )
...
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-13 16:47:21 -04:00
ebe1dec2da
[PCP][DCP] Enable PCP+DCP on sparse-MLA models ( #56157 )
...
Signed-off-by: patryk.saffer <patryk.saffer@blc-login-1.blc1.mistralcompute.ai >
Signed-off-by: patryk.saffer <patryk.saffer@blc-login-0.blc1.mistralcompute.ai >
Signed-off-by: PatrykSaffer <patryk.saffer@mistral.ai >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: patryk.saffer <patryk.saffer@blc-login-1.blc1.mistralcompute.ai >
Co-authored-by: patryk.saffer <patryk.saffer@blc-login-0.blc1.mistralcompute.ai >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Patryk Saffer <PatrykSaffer@users.noreply.github.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-12 23:24:44 +00:00
d43bb2f37f
[3/N] HiSparse: host-resident sparse-MLA decode hot-buffering ( #53781 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Signed-off-by: Matthew Bonanni <matthew@neuralmagic.com >
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-09-12 00:36:47 -04:00
Nick Hill
e6cb56337b
[Core] MRV2 support for fast-prefill ( #56145 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-09-10 10:36:04 -07:00
3f41d102c5
[1/2][Model Runner V2] DBO support, eager mode ( #50945 )
...
Signed-off-by: specture724 <specture724@gmail.com >
Signed-off-by: Anjie Hou <149605198+specture724@users.noreply.github.com >
Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
2026-09-04 14:17:39 -07:00
Andrii Skliar and Andrii Skliar
adebc41b7e
[Mamba] Add FlashInfer ReplaySSM backend ( #52506 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Signed-off-by: Andrii Skliar <andreyws96@gmail.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
2026-09-01 10:26:43 -04:00
Ronald and aoshen02
479eeb32d2
[Bugfix][MRV2] Isolate sleep-mode KV allocations ( #53508 )
...
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-08-27 15:08:28 -07:00
Misha Goin
7de96050c5
[Spec Decode] Enable adaptive DSpark on SM100 sparse MLA ( #52783 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
2026-08-25 13:31:15 -04:00
Lucas Wilkinson
8bdc70ec7b
[6/N][KV-Cache Layout Refactor] Standardize KV cache layout ( #51718 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
2026-08-21 17:47:49 -07:00
Lucas Wilkinson and Claude
57bd0ed441
[5/N][KV-Cache Layout Refactor] Backend-published KV packing via customize_spec ( #51704 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-08-14 09:47:55 -04:00
7f7a32cfec
[Spec Decode] DSpark confidence-scheduled verification ( #47808 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com >
Signed-off-by: Lucas Wilkinson <wilkinson.lucas@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-08-12 12:27:48 -07:00
stefankoncarevic
3e372c5ff2
[Bugfix][ROCm] Give KV-first attention blocks their own page in hybrid models ( #51837 )
...
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com >
2026-08-11 16:52:30 -04:00
Soila Kavulya and Claude
aef85aed5d
[Bugfix][TurboQuant] Add KV quant mode for turboquant ( #50533 )
...
Signed-off-by: Soila Kavulya <soila.p.kavulya@intel.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-07-31 17:42:32 +00:00
Martin Hickey
b2fb83e7ff
[Attention]: Use KVCacheSpec for AttentionMetadataBuilder type hints ( #50148 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-07-31 11:20:18 +00:00
Nick Hill and Taneem Ibrahim
fdaa0d9e59
[ModelRunner V2] Support encoder-only attention ( #49331 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
2026-07-27 00:38:44 +00:00
6700813f86
[3/N][KV-Cache Layout Refactor] Standardize Mamba cache; drop get_transfer_cache_regions ( #44456 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-07-21 09:16:15 +00:00
b6ff8a2f50
[Core] Add MRV2 virtual-batch PCP for MLA ( #46570 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-07-19 02:53:15 +00:00
Lucas Wilkinson and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
04d553f390
[Misc] Use meta tensor for KV cache stride calculation ( #47316 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-10 23:24:59 -04:00
Lucas Wilkinson and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
9c18e90f6c
[BugFix] Fix packed HND KV cache reshape for FlashAttention ( #47314 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-10 23:22:39 -04:00
Michael Goin
08dfd68610
[Model] Add LongCat-Flash-Lite (n-gram embedding) ( #47857 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
2026-07-10 07:17:50 -07:00
Lucas Wilkinson and Codex
e08a915146
[Bugfix] Preserve tensor causal metadata for grouped attention ( #48135 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Codex <codex@openai.com >
2026-07-09 17:57:53 +00:00
Michael Goin and Claude Opus 4.8
0d12618e98
[Spec Decode] Support hybrid (SWA + full attention) DFlash drafters ( #47914 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-07-08 11:12:45 -04:00
Andreas Karatzas
2f71b2bd9f
[ROCm] Align mixed encoder-decoder KV cache views in V2 runner ( #47685 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-07-07 12:09:22 +08:00
Isotr0py and Nick Hill
cc1d020d01
[MRV2] Enable mm prefix bidi attention support on MRV2 ( #46942 )
...
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Signed-off-by: Isotr0py <2037008807@qq.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-05 14:45:29 +00:00
Lucas Wilkinson
fa4321de3d
[Bugfix][TurboQuant] Preserve KV cache dtype in backend shape ( #47609 )
2026-07-05 08:20:48 +00:00
sychen52 and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
67ff0ae30f
Support nvfp4 kv with kv-cache-dtype-skip-layers sliding_window ( #42890 )
...
Signed-off-by: Shiyang Chen <shiychen@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-04 02:29:13 +00:00
c6741b2ad4
[Model] Support Unlimited OCR ( #46564 )
...
Signed-off-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn >
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-06-27 23:09:18 -07:00
2cac89f9da
[Spec Decode] Support mixed KV page sizes for DFlash ( #45181 )
...
Signed-off-by: Alex Steiner <asteiner@nvidia.com >
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-06-21 22:45:14 +08:00
01192139bf
[DSv4] Pack KV caches into contiguous per-block allocations for DeepSeek V4 ( #44577 )
...
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-06-19 12:55:42 -04:00
b8336c3c7c
[Bugfix][V1] Split V2 model-runner attention groups on num_heads_q ( #45564 )
...
Signed-off-by: Roger Wang <hey@rogerw.io >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-06-14 21:49:46 -07:00
Benjamin Chislett and Giancarlo Delfin
0bae1d3848
[MRV2][Spec Decode] DFlash ( #44586 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com >
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com >
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-06-10 08:47:46 -07:00
qizixi and Claude Opus 4.8
d07ad0693b
[Bugfix] Use storage_block_size in KV cache reshape for compressed specs (DeepSeek V4) ( #43988 )
...
Signed-off-by: zixi-qi <zixi@inferact.ai >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-05-29 11:14:25 -07:00
Mengqing Cao and Nick Hill
1223732dda
[ModelRunnerV2][Hybrid model] Support kernel block size in hybrid model ( #38831 )
...
Signed-off-by: MengqingCao <cmq0113@163.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Signed-off-by: Mengqing Cao <cmq0113@163.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-05-28 00:55:55 +00:00
7e33081cee
[Attention] Make FlexAttention and FlashAttention use num-blocks first layouts ( #42095 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com >
2026-05-26 19:55:56 -07:00
Nick Hill
47d4407d7c
[Model Runner V2] Support sharing kv cache layers ( #35045 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-05-22 22:18:23 +00:00
Nicolò Lucchesi and Nick Hill
fba010dd74
[Bugfix][MRV2] Fix KVCache tensor explicit kernel_block_size dim ( #42766 )
...
Signed-off-by: NickLucche <nlucches@redhat.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-05-18 20:25:41 -07:00
zhrrr
7a08b34fbf
[Model Runner V2] support qwen35 / mamba hybrid model ( #35520 )
...
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com >
2026-05-07 09:31:05 -07:00
+7
4d51588e23
[Feat] DeepSeek V4 Rebased ( #40860 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Signed-off-by: qizixi <zixi@inferact.ai >
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <yongye@inferact.ai >
Co-authored-by: Simon Mo <simon@inferact.ai >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: youkaichao <youkaichao@gmail.com >
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com >
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com >
Co-authored-by: khluu <khluu000@gmail.com >
Co-authored-by: qizixi <zixi@inferact.ai >
Co-authored-by: Zhewen Li <zhewenli@inferact.ai >
2026-04-26 18:31:08 -07:00
Nick Hill
fe85a92e86
[Core] Avoid seq_lens_cpu GPU->CPU sync ( #40654 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-04-24 00:35:55 +00:00