MOVZX and Hinata
5f3c7bd8ab
[Mini-VLLM] Remove the ROCm device path, debug routes and arch-gated tests
...
- Delete csrc/rocm, hipify, the _rocm_C extension and the setup.py rocm
target path; drop the rocm requirement files
- Delete the aiter, rdna3, skinny-gemm and ROCm fused-MoE kernel modules
and retarget the linear kernel registries away from them
- Reduce platforms/rocm.py to the GCN architecture predicates that live
code still imports, and delete the RocmPlatform class and plugin
- Drop the VLLM_SERVER_DEV_MODE debug surface: reset, collective_rpc,
server_info and sleep routes plus the log_error_stack flag
- Delete lm-format-enforcer, lark and otel-semantic-conventions-ai pins;
the live server process maps none of them
- Delete tests that only build architectures this fork no longer
registers, and test directories left without any test file
- Restore the tests/v1/kv_connector mock helpers that scheduler tests
import at module level
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 19:31:07 +07:00
02e55d2af3
[KV Cache] GLM-5.3-Flash: use the generic packed KV layout; CircularBufferSpec tail with speculative ring slots ( #57169 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-10-09 15:48:37 -04:00
Woosuk Kwon
194bd72be7
[Compile] Remove torch.compile-based sequence parallelism and AsyncTP ( #60517 )
2026-10-08 08:28:23 -07:00
yhcheong (vllmellm)
273e099693
[Bugfix][ROCm] Select page-aligned kernel blocks for pooled indexers so the block table addresses storage pages (GLM-5.3-Flash) ( #59412 )
...
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com >
2026-10-08 14:44:58 +00:00
f5ed3d9c03
[Fast Start] Charge daemon-held weights against gpu_memory_utilization ( #57298 )
...
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Xun Sun <UNIDY2002@outlook.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Xun Sun <UNIDY2002@outlook.com >
2026-09-29 17:58:35 +00:00
e7c903609f
[Perf][MRV2] Allow FULL decode graphs for one-token prompt tails ( #58400 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
2026-09-28 12:48:52 -07:00
004e37ece2
[Misc] Name each backend and its kernel block sizes in block-size errors ( #58557 )
...
Signed-off-by: Eugenio "Jay" Zuccarelli <11176606+jayzuccarelli@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-28 14:33:18 +08:00
stefankoncarevic and Andreas Karatzas
9ac3b7c69f
[ROCm][CI] Mirror the DSv4-Flash disaggregated DP EP group on MI355 ( #58558 )
...
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-24 18:38:25 -05:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
d43bb2f37f
[3/N] HiSparse: host-resident sparse-MLA decode hot-buffering ( #53781 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Signed-off-by: Matthew Bonanni <matthew@neuralmagic.com >
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-09-12 00:36:47 -04:00
Jiangyun Zhu
98ed0856f3
[Model] add GLM-5.3-Flash support ( #53906 )
2026-09-03 09:40:35 -07:00
Andrii Skliar and Andrii Skliar
adebc41b7e
[Mamba] Add FlashInfer ReplaySSM backend ( #52506 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Signed-off-by: Andrii Skliar <andreyws96@gmail.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
2026-09-01 10:26:43 -04:00
Nick Hill and Kimi Code
96242aa50d
[Bugfix][MRV2] Release layer-bound KV cache memory in shutdown() ( #54246 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Kimi Code <noreply@moonshot.ai >
2026-08-28 21:27:09 +00:00
Nick Hill and Kimi Code
70732942c6
[Bugfix][ROCm] Pre-allocate wvSplitKrc static workspaces before KV init ( #54247 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Kimi Code <noreply@moonshot.ai >
2026-08-28 14:22:53 -07:00
Avinash Paul and avininjamay8
8d301f075b
[Bugfix][ROCm][Disagg] Fix MoRIIO shared KV memory region registration ( #53698 )
...
Signed-off-by: avininjamay8 <avpaul@amd.com >
Co-authored-by: avininjamay8 <avpaul@amd.com >
2026-08-27 00:10:57 +08:00
Lucas Wilkinson
8bdc70ec7b
[6/N][KV-Cache Layout Refactor] Standardize KV cache layout ( #51718 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
2026-08-21 17:47:49 -07:00
wangxiyuan
73b83949f7
[Platform] Add check_runner_kv_caches_multi_layer ( #51633 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2026-08-13 19:25:07 +00:00
Lucas Wilkinson and OpenAI Codex
79f3183f86
[Bugfix] Bound KV block zeroing launch geometry ( #52058 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-13 03:42:26 +00:00
0a94d85a66
[Bugfix][MRV2] Require all requests to be decoding for uniform-decode dispatch ( #51865 )
...
Signed-off-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com >
Co-authored-by: Janelle Cai <janelle.cai@modal.com >
2026-08-11 17:15:06 -07:00
Michael Goin and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
e3fe212eaf
[Bugfix] Generalize KV block zeroing to AttentionSpec ( #51749 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-11 12:13:09 -04:00
Guan-Ming Chiu and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
457a5f34db
[Bugfix][MRV2] Support encoder timing stats in model runner V2 ( #50020 )
...
Signed-off-by: Guan-Ming (Wesley) Chiu <105915352+guan404ming@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-12 00:00:49 +08:00
Yizhou
a311916a29
[MRV2][Spec] Fuse AR speculator multi-step decodes back into one CUDA graph ( #46849 )
...
Signed-off-by: Yizhou Liu <liu_yizhou@outlook.com >
2026-08-11 15:06:22 +08:00
wangxian001 and Nick Hill
d6af803f43
[Bugfix] Fix packed KV block zeroing stride ( #50276 )
...
Signed-off-by: wangxian001 <120719093+wangxian001@users.noreply.github.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-06 13:49:06 -07:00
Tyler Michael Smith and OpenAI Codex
12292d94b2
[CI] Detect and fail evals on when NaNs appear in logits ( #50323 )
...
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-04 23:02:58 +00:00
Matthew Bonanni
a0cd2b69b3
[Bugfix] Universally align block table width to 128 tokens ( #50302 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
2026-07-31 08:52:17 -07:00
Nick Hill
0d0504b54c
[Core] Warm up runner-owned Triton kernels before the first request ( #49903 )
2026-07-28 07:58:02 -07:00
Elvir Crnčević and Claude Opus 4.6
972848f276
[Bugfix] Support non-uniform page sizes in KVBlockZeroer ( #49704 )
...
Signed-off-by: Elvir Crncevic <elvircrn@gmail.com >
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com >
2026-07-24 13:38:59 -07:00
Nick Hill and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
521aa80f71
[Core] Simplify KVBlockZeroer index tensor handling ( #48399 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-23 18:12:46 +08:00
6700813f86
[3/N][KV-Cache Layout Refactor] Standardize Mamba cache; drop get_transfer_cache_regions ( #44456 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-07-21 09:16:15 +00:00
Jiangyun Zhu and Yifan Qiao
481e481be7
[2/N][Core] support partial prefix cache hit for hybrid model ( #46384 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-07-12 05:37:51 +00:00
Benjamin Chislett and Wentao Ye
1cd75b3dd4
[Bugfix] Fix race condition in KVBlockZeroer ( #48085 )
...
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com >
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-07-09 09:18:19 +00:00
zhrrr
8a9eb40808
[Model Runner V2] Support zeroing freshly allocated KV blocks for hybrid + fp8 KVCache ( #43990 )
...
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com >
2026-06-02 05:56:53 +00:00
Nick Hill
47d4407d7c
[Model Runner V2] Support sharing kv cache layers ( #35045 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-05-22 22:18:23 +00:00
32e45636e3
[torch.compile]: Disable Sequence Parallelism (SP) for piecewise compilation ( #38373 )
...
Signed-off-by: SouthWest7 <am1ao@qq.com >
Signed-off-by: Xinan Miao <1403572259@qq.com >
Co-authored-by: SouthWest7 <am1ao@qq.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Wang Xingran <72983099+wangxingran222@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-26 17:44:42 +00:00
f4b42df048
[Attention Backend] TurboQuant: 2-bit KV cache compression with 4x capacity ( #38479 )
...
Signed-off-by: vibhavagarwal5 <vibhavagarwal5@gmail.com >
Signed-off-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: Xinyu Chen <xinyu1.chen@intel.com >
Co-authored-by: Michael Goin <mgoin64@gmail.com >
2026-04-14 19:57:13 -07:00
Wentao Ye
c59a132f96
[V0 Deprecation] Refactor kv cache from list to element ( #37487 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-03-23 20:10:11 -07:00
Wentao Ye
0d81a1fe61
[V0 Deprecation] Deprecate virtual engine ( #37195 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-03-18 14:30:14 -07:00
Kunshang Ji
d157216093
[BUGFIX][Mamba] Use uint64 for address in KVBlockZeroer ( #37197 )
...
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com >
2026-03-16 21:39:56 +01:00
Nicolò Lucchesi
098d844731
[NIXL][1/N] Refactor kernel_block_size detection ( #35752 )
...
Signed-off-by: NickLucche <nlucches@redhat.com >
2026-03-11 01:11:23 -07:00
Vadim Gimpelson
4ff8c3c8f9
[BUGFIX][Mamba][Qwen3.5] Zero freed SSM cache blocks on GPU ( #35219 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
2026-03-10 03:32:20 -07:00
Lucas Wilkinson and Matthew Bonanni
28ef9ba399
[BugFix] Add support for MTP num_speculative_tokens > 1 with sparse MLA ( #34552 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-03-03 07:21:57 -08:00
Nicolò Lucchesi
cabdaa7619
[Misc] Move GPUModelRunner.prepare_kernel_block_sizes to utils ( #35400 )
...
Signed-off-by: NickLucche <nlucches@redhat.com >
2026-02-27 11:42:51 +08:00
Cyrus Leung
d7e17aaacd
[Refactor] Move profiling methods to MM budget ( #33559 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-02-02 23:27:00 +08:00
Wentao Ye
010ec0c30e
[Deprecation] Deprecate seed_everything and scatter_mm_placeholders in v0.15 ( #33362 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-01-31 02:54:16 +00:00
Cyrus Leung
c87eac18f7
[Refactor] Move MM item count validation outside of processor ( #33396 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-01-30 09:27:31 +00:00
Matthew Bonanni
a608b4c6c2
[5/N][Attention] Finish eliminating vllm/attention folder ( #32064 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
2026-01-27 10:02:51 -05:00
Tomas Ruiz
4a5299c93f
feat: spec decode with draft models ( #24322 )
...
Signed-off-by: Tomas Ruiz <tomas.ruiz.te@gmail.com >
2026-01-19 16:05:46 -05:00
Cyrus Leung
cbbae38f93
[2/N] Move cache factories to MM registry ( #32382 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-01-15 01:02:30 -08:00
Matthew Bonanni
98f60e5acb
[6/N][Attention] Move utils to more appropriate locations ( #32215 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
2026-01-13 05:38:52 -08:00
Matthew Bonanni
2612ba9285
[1/N][Attention] Restructure attention: move files ( #31916 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
2026-01-09 13:10:24 -08:00