iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
23,342 Commits 3 Branches 0 Tags
sm86
Commit Graph
136 Commits
Author SHA1 Message Date
MOVZXandHinata 5f3c7bd8ab [Mini-VLLM] Remove the ROCm device path, debug routes and arch-gated tests 
- Delete csrc/rocm, hipify, the _rocm_C extension and the setup.py rocm
  target path; drop the rocm requirement files
- Delete the aiter, rdna3, skinny-gemm and ROCm fused-MoE kernel modules
  and retarget the linear kernel registries away from them
- Reduce platforms/rocm.py to the GCN architecture predicates that live
  code still imports, and delete the RocmPlatform class and plugin
- Drop the VLLM_SERVER_DEV_MODE debug surface: reset, collective_rpc,
  server_info and sleep routes plus the log_error_stack flag
- Delete lm-format-enforcer, lark and otel-semantic-conventions-ai pins;
  the live server process maps none of them
- Delete tests that only build architectures this fork no longer
  registers, and test directories left without any test file
- Restore the tests/v1/kv_connector mock helpers that scheduler tests
  import at module level

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-19 19:31:07 +07:00
MOVZXandHinata 81074e94a3 [Mini-vLLM] Cut dead subsystems: models, entrypoints, backends, utils, tests 
- Drop dead model modules, keeping files live imports require
- Remove anthropic, cohere, codegen entrypoints and batch STT modes
- Delete unused attention backends, MoE experts, and NPU/XPU util leaves
- Strip dead third_party trees and ROCm leaves with no CUDA path
- Remove mooncake EC connectors: dependency missing since branch base
- Re-audit verified by full import sweep and Qwen3.5 smoke test

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-19 00:20:48 +07:00
Woosuk KwonandClaude Opus 5.5 4a89316efa [Core] Drop the Ray driver-node GPU pin left over from the legacy executor (#61713) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-15 16:00:18 -07:00
Nick Hill c059bee812 [MRV2] Switch CI tests still running on MRV1 unnecessarily (#61014) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-10-11 09:57:07 -07:00
aoshen02Claude Fable 5.1indianspeedster
 		a6f43e57d6 [ROCm] Enable the cuMem CUDA-graph pool offload on ROCm (#59523) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: indianspeedster <cspandey016@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: indianspeedster <cspandey016@gmail.com>
 2026-10-09 08:29:55 +00:00
MattandClaude Opus 5.5 5dd4a628dd [ROCm][Bugfix] Map sleep-mode cuMem memory for the host (#60367) 
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-10-08 04:26:12 +00:00
aoshen02andClaude Opus 5.5 5e56e9af2a [Feature] Release the CUDA graph pool on sleep (#59160) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 01:13:10 +00:00
aoshen02andClaude Opus 5.5 8e58ac22af [Feature] Offload KV-init runtime state on sleep (#59158) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-02 09:49:21 +00:00
aoshen02andClaude Opus 5.5 866f02db8a [Feature] Release WorkspaceManager scratch on sleep (#59156) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-02 02:14:37 +00:00
AlanFokCoClaude Opus 5.5aoshen02
 		f4917dadc8 [Bugfix][Quantization] Stop sleep(level=2) from zeroing compressed-tensors KV scales (#57163) 
Signed-off-by: AlanFokCo <alanfok2868@gmail.com>
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
 2026-09-29 15:34:15 +00:00
Thang NguyenClaude Opus 5.5Andreas KaratzasCodex
 		da1aaec31f [CI] Split (H200 MIG/MI300) Basic Correctness into named jobs (#57054) 
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-25 17:05:33 -07:00
mkunreddMahesh KunreddiClaudeAndreas KaratzasCodexAndreas Karatzas
 		4013f3ad6c [ROCm] Fix CI runtime and tests for MI355 DPX (#58244) 
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Mahesh Kunreddi <mahesh.kunreddi@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-09-24 23:15:11 -05:00
Yan MaandKunshang Ji 4991f97669 [XPU] sleep mode: fix KV cache release test (#57485) 
Signed-off-by: Yan Ma <yan.ma@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-09-18 15:49:17 +00:00
Cyrus Leung dbf4b89cf2 [Misc] Remove unnecessary Transformers version guards (#57556) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-09-18 15:01:35 +00:00
Dakai AnAlanFokCoaoshen02OpenAI Codex
 		092bdd6d57 [Frontend][Core] Add release_kv_cache_memory() API (#44890) 
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: AlanFokCo <alanfok2868@gmail.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com>
Co-authored-by: AlanFokCo <alanfok2868@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-09-18 07:12:12 +08:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 16:44:29 +00:00
Oxana KorzhandClaude Opus 5 6cddad414e [CI][Test] Deflake test_mem.py sleep-mode asserts via allocator bookeeping (#54312) 
Signed-off-by: Oxana Korzh <okorzh@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-29 12:05:17 -07:00
Ronaldandaoshen02 16d6c376bc [Bugfix][Models] Register sleep-managed runtime buffers (#53507) 
Signed-off-by: Ronald1995 <ronaldautomobile@163.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
 2026-08-29 10:32:24 +08:00
Ronaldandaoshen02 479eeb32d2 [Bugfix][MRV2] Isolate sleep-mode KV allocations (#53508) 
Signed-off-by: Ronald1995 <ronaldautomobile@163.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
 2026-08-27 15:08:28 -07:00
ray24777Claude Opus 5Isotr0pymergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		61d4f56635 [Offloader] Offload submodules that make_layers never reaches (#53120) 
Signed-off-by: ray24777 <103923677+ray24777@users.noreply.github.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-08-26 18:37:00 +08:00
Dakai AnAlanFokCoOpenAI Codex
 		9409f59e09 [Core] Add CuMemAllocator.discard() for tag-selective GPU memory release (#52514) 
Signed-off-by: AlanFokCo <alanfok2868@gmail.com>
Signed-off-by: Dakai An <dakaian108@gmail.com>
Co-authored-by: AlanFokCo <alanfok2868@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-16 23:15:14 +08:00
Nick Hill 9b0afeb4f6 [Perf] Avoid some more unnecessary GPU<->CPU syncs (#51458) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-08-08 17:13:26 -07:00
Wentao Ye c39076feff [CI Test] Add specific unit test for mrv2 offloading (#51440) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-08-07 22:49:45 +00:00
Wentao Ye a671679e9f [MRv2 Feature] MR v2 weight offloading support (#51413) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-08-07 13:20:21 -06:00
Silen NaihinDakai AnJiangyun Zhu
 		b50fdebce0 [Bugfix] Fix level-2 sleep/wake/reload with enable_lora=True (#39935) 
Signed-off-by: Silen Naihin <silen.naihin@gmail.com>
Co-authored-by: Dakai An <dakaian108@gmail.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
 2026-08-06 09:13:10 +08:00
Tyler Michael SmithandOpenAI Codex 12292d94b2 [CI] Detect and fail evals on when NaNs appear in logits (#50323) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-04 23:02:58 +00:00
Andreas Karatzas 165ed33327 [CI][ROCm] Stabilize LLM GC teardown check (#50340) 
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
 2026-07-30 18:42:03 +08:00
Andreas Karatzas 3ad85e0de4 [CI][AMD] Configure MI300 tests for native execution without DinD (#48387) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-07-15 02:14:16 +00:00
Kunshang Ji fb42e5219e [Platform] Replace torch.cuda.mem_get_info with torch.accelerator.get_memory_info (#44825) 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
 2026-06-30 14:39:52 +08:00
Andreas KaratzasMatthew WongMatt
 		3f5a1e1733 [ROCm][CI] Expand basic correctness target suites (#46573) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>
Co-authored-by: Matthew Wong <Matthew.Wong2@amd.com>
Co-authored-by: Matt <156021403+mawong-amd@users.noreply.github.com>
 2026-06-25 12:18:57 +08:00
Yan Ma 3bb46975bd [XPU][Feature] transparent sleep mode support for XPU platform (#37149) 
Signed-off-by: Yan Ma <yan.ma@intel.com>
 2026-06-07 10:45:31 +08:00
wliao2 3abf858443 [Test] Refactor hard coded device string in test files under compile/quantization/models/model_executor folders (#38901) 
Signed-off-by: Liao, Wei <wei.liao@intel.com>
 2026-04-15 11:02:35 +08:00
Harry Mellor 65986db6ba Make Gemma and Gemma 2 accept inputs_embeds like Gemma 3 (#36787) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-11 18:12:43 +00:00
Joe RundeTravis JohnsonNick Hill
 		6f0dd93801 [Core] Remove busy loop from idle buffer readers (#28053) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Travis Johnson <tsjohnso@us.ibm.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-03-04 07:44:20 +00:00
Ming YangandMichael Goin 6831650c40 [offloader] v2: Hide weight onloading latency via prefetching (#29941) 
Signed-off-by: Ming Yang <minos.future@gmail.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
 2026-02-25 17:20:59 -08:00
Wei ZhaoandMichael Goin 59d53066d8 [Feature] Support CPU Offloading without Pytorch Pinned Memory that leads to doubled allocation (#32993) 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
 2026-02-13 08:11:26 -08:00
Ning Xie c907d22158 [refactor] refactor memory constants usage (#31865) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2026-01-07 18:37:31 +00:00
Matthew Bonanni 7eb6cb6c18 [Attention] Update tests to remove deprecated env vars (#30563) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2025-12-17 09:49:59 -08:00
Charlie Fu 6af70e11a0 [ROCm][CI] Fix test_max_len.py for Rocm (#29916) 
Signed-off-by: charlifu <charlifu@amd.com>
Signed-off-by: Charlie Fu <Charlie.Fu@amd.com>
 2025-12-08 16:58:30 -05:00
Charlie Fu 2c22c4ca2d [ROCm][CI] Increase the memory threshold for test_deep_sleep_fp8_kvcache (#30104) 
Signed-off-by: charlifu <charlifu@amd.com>
 2025-12-05 04:51:44 +00:00
VensenandTJian 66b5840287 [Bugfix][sleepmode][fp8 kv cache]: Fix FP8 KV cache + sleep(level=2) gibberish output (#28783) 
Signed-off-by: vensen <vensenmu@gmail.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
 2025-11-30 14:24:25 +08:00
Roger Wang 0ff70821c9 [Core] Deprecate xformers (#29262) 
Signed-off-by: Roger Wang <hey@rogerw.io>
 2025-11-24 04:18:55 +00:00
ℍ𝕠𝕝𝕝𝕠𝕨 𝕄𝕒𝕟tjtanaakliuae
 		4ca5cd5740 [Core][AMD] Migrate fully transparent sleep mode to ROCm platform (#12695) 
Signed-off-by: Hollow Man <hollowman@opensuse.org>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: kliuae <kuanfu.liu@embeddedllm.com>
 2025-11-12 15:24:12 -08:00
Sumanth R Hegde 4917002523 [Fix] Skip record_sleep_state logic in PrometheusStatsLogger if not in dev mode (#27789) 
Signed-off-by: SumanthRH <sumanthrh99@gmail.com>
 2025-10-30 19:26:27 +00:00
Nick Hill 647214f3d5 [V0 Deprecation] Remove V0 executors (#27142) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-10-21 11:09:37 -07:00
iAmir97iAmir97gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 		1d165d6d85 [Chore] Separate out vllm.utils.mem_utils (#27143) 
Signed-off-by: iAmir97 <Amir.balwel@embeddedllm.com>
Signed-off-by: iAmir97 <71513472+iAmir97@users.noreply.github.com>
Co-authored-by: iAmir97 <Amir.balwel@embeddedllm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-10-18 10:06:59 +00:00
Tahsin TunanNick HillHarry Mellor
 		43721bc67f [CI] Replace large models with tiny alternatives in tests (#24057) 
Signed-off-by: Tahsin Tunan <tahsintunan@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-16 15:51:27 +01:00
Cyrus Leung 1e4ecca1d0 [V0 Deprecation] Remove VLLM_USE_V1 from tests (#26341) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-10-07 15:42:31 +00:00
Harry Mellor d6953beb91 Convert formatting to use ruff instead of yapf + isort (#26247) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-05 07:06:22 -07:00