iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
20,845 Commits 3 Branches 0 Tags
78300cdabffbaaffe7625ed34412ec2aec75535c
Commit Graph
264 Commits
Author SHA1 Message Date
Derek WeitzelandClaude Fable 5.1 859dd39512 [Bugfix][Core] Wait for the previous PP tensor sends before the next forward pass (#54962) 
Signed-off-by: Derek Weitzel <djw8605@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-03 08:01:36 +00:00
aoshen02aoshen02Claude Fable 5
 		4ac452ad98 [Core] Release NCCL communicator memory in sleep mode (#51485) 
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-09-01 14:31:16 +08:00
AI-Infra Rookie c92b29a1d4 [Bugfix] Make metadata send non-blocking in GroupCoordinator.isend_tensor_dict (#49274) 
Signed-off-by: z00897177 <zhouxinyi6@huawei.com>
 2026-08-30 23:51:25 +00:00
stefankoncarevic 9236159bb6 [ROCm][CI] Keep startup profiling from aborting when free memory grows (#53591) 
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
 2026-08-28 00:02:40 -07:00
Ronaldandaoshen02 479eeb32d2 [Bugfix][MRV2] Isolate sleep-mode KV allocations (#53508) 
Signed-off-by: Ronald1995 <ronaldautomobile@163.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
 2026-08-27 15:08:28 -07:00
aoshen02 17da48596c [Bugfix][DP] Synchronize the device on pause completion (#52914) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
 2026-08-26 09:50:33 -07:00
aoshen02OpenAI CodexOpenAI CodexRoger Wang
 		ee5ae2ae70 [RL] Add rank-local IPC weight updates (#52497) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-08-26 15:36:57 +08:00
Wentao Ye f59bb0bd8c [Refactor] Refactor batch invariance folder (#53619) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-08-24 14:31:36 -07:00
Tianyu Guo 7ca49fbe4b [Refactor][Model Runner V2][Multimodal] Move the encoder-only path out of the shared runner (#53176) 
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
 2026-08-22 05:11:44 +00:00
Lucas Wilkinson 8bdc70ec7b [6/N][KV-Cache Layout Refactor] Standardize KV cache layout (#51718) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-08-21 17:47:49 -07:00
Itay Alroy a2257f95b7 [Elastic EP] Reduce eager-mode reconfiguration downtime (#51885) 
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
 2026-08-19 02:13:06 +00:00
Gabriel Wu 97388c44f9 [Bugfix] Make DSV4 sparse MLA work end-to-end for plain decode, MTP, and DSpark (#51538) 2026-08-15 07:01:32 -07:00
Connor CarpenterandNick Hill f473870ecf [Rust Frontend][gRPC] Add RL lifecycle control (#51316) 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-08-14 11:32:39 -07:00
鐘天楽 a367cdbca0 [Profiler] Add minimal Triton Proton profiling backend (#48789) 
Signed-off-by: Luossu <zhluosuu@outlook.com>
 2026-08-11 14:51:57 -07:00
Silen NaihinDakai AnJiangyun Zhu
 		b50fdebce0 [Bugfix] Fix level-2 sleep/wake/reload with enable_lora=True (#39935) 
Signed-off-by: Silen Naihin <silen.naihin@gmail.com>
Co-authored-by: Dakai An <dakaian108@gmail.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
 2026-08-06 09:13:10 +08:00
Nick HillandClaude 7635a9002b [Core] Explicitly manage torch CPU threads in workers (#49919) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-08-04 10:06:38 -07:00
aoshen02aoshen02vx120Claude Opus 5Roger Wang
 		b09688a6e7 [Bugfix][Spec Decode] Preserve draft buffers across level-2 sleep (#49774) 
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Signed-off-by: vx120 <893600387@qq.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: vx120 <893600387@qq.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-07-28 00:16:21 -07:00
Schwinn Saereesitthipitak b5b61c622c [Core][Distributed] Add process-checkpoint lifecycle hooks for communicators (starting with Flashinfer) (#46877) 
Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
 2026-07-26 14:47:50 -04:00
fangyuchuzWaNg3a798347923TianZhuo205150940a798347923w00689259zWaNg3yzchang-plusJade Zheng
 		0b0bd2b5f6 [Feature] Add fault tolerance framework (simplified) for DP+EP external LB deployments (#44428) 
Signed-off-by: fangyuchu <fangyuchu@qq.com>
Signed-off-by: a798347923 <2645302020@qq.com>
Signed-off-by: TianZhuo <2770730562@qq.com>
Signed-off-by: a798347923 <39047817+a798347923@users.noreply.github.com>
Signed-off-by: 205150940 <112750056+205150940@users.noreply.github.com>
Signed-off-by: w00689259 <wangzhuo66@huawei.com>
Signed-off-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com>
Signed-off-by: zWaNg3 <389750525@qq.com>
Signed-off-by: yzchang-plus <1078477584@qq.com>
Signed-off-by: Jade Zheng <zheng.shoujian@outlook.com>
Co-authored-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com>
Co-authored-by: a798347923 <2645302020@qq.com>
Co-authored-by: TianZhuo <2770730562@qq.com>
Co-authored-by: 205150940 <112750056+205150940@users.noreply.github.com>
Co-authored-by: a798347923 <39047817+a798347923@users.noreply.github.com>
Co-authored-by: w00689259 <wangzhuo66@huawei.com>
Co-authored-by: zWaNg3 <389750525@qq.com>
Co-authored-by: yzchang-plus <1078477584@qq.com>
Co-authored-by: Jade Zheng <zheng.shoujian@outlook.com>
 2026-07-25 11:49:09 -04:00
Isotr0py 1750e443f2 [Misc] Move PyNvVideoCodec stuff out of gpu worker (#49322) 
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
 2026-07-22 10:11:56 +08:00
Matthew BonannihaosdentClaude Opus 4.7Justin Wood
 		58b2012aa2 [copy of #45208] CuMem slept-L1 fragmentation accounting (#49208) 
Signed-off-by: haosdent <haosdent@gmail.com>
Signed-off-by: Justin Wood <justin.m.wood@me.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: haosdent <haosdent@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Justin Wood <jwood@me.com>
 2026-07-20 23:08:11 +00:00
aoshen02 9459fc6471 [Bugfix][RL] Set vLLM config during weight reload (#45989) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
 2026-07-20 15:02:56 +08:00
devalshahamdandDeval Shah fcd2255d16 [Hardware][GPU] Profiler config additional to increase it scope and annotation details (#37524) 
Signed-off-by: devalshahamd <deval.shah@amd.com>
Signed-off-by: Deval Shah <devashah@amd.com>
Signed-off-by: Deval Shah <deval.shah@amd.com>
Co-authored-by: Deval Shah <devashah@amd.com>
 2026-07-17 13:38:59 -07:00
peizhang56ClaudeAndreas Karatzas
 		05eed72aec [ROCm] Re-enable cudagraph memory profiling, captured on the current stream (#48526) 
Signed-off-by: pei.zhang <pei.zhang@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-07-15 10:03:48 -05:00
Rohan PotdarandClaude Opus 4.8 d973cce3ca Re-disable CUDA graph memory profiling on ROCm (#48440) 
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-13 03:59:20 +00:00
vx120crp0128aoshen02Claude Opus 4.8
 		fc1c548093 Runtime Draft Weight Update for Speculative Decoding (#46725) 
Signed-off-by: vx120 <893600387@qq.com>
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: crp0128 <191679376@qq.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-11 22:51:53 -07:00
peizhang56ClaudeAndreas Karatzas
 		88e5e2c57b [CI/Build][AMD] Fix ROCm OOM in eagle_correctness_heavy by reserving CUDA graph memory (#47366) 
Signed-off-by: pei.zhang <pei.zhang@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-07-10 02:14:38 +00:00
Micah Williamson 766469a4c4 [ROCm] Revert Part of [ROCm] Fix pooling startup workspace lock #47912 (#48154) 
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
 2026-07-09 20:34:24 +00:00
Andreas Karatzas 26831949b4 [ROCm] Fix pooling startup workspace lock (#47912) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-07-08 17:59:50 -05:00
Nils MattesonNils Mattesonmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		e97c3cb303 [Core] Persist and reuse the memory-profiling result across boots (opt-in) (#47388) 
Signed-off-by: Nils Matteson <nils@thaw.sh>
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
Co-authored-by: Nils Matteson <nils@thaw.sh>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-08 00:53:02 +00:00
Nils MattesonNils Mattesonmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		dd0d74cd92 [Doc] Surface the --kv-cache-memory suggestion at INFO and document fast-startup knobs (#47374) 
Signed-off-by: Nils Matteson <nils@thaw.sh>
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
Co-authored-by: Nils Matteson <nils@thaw.sh>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-07 15:05:07 -07:00
Robin c74e751824 [Doc] Fix grammatically incorrect error message in gpu_worker and xpu_worker (#36715) 
Signed-off-by: Hongbin10 <jdmjdm1998@163.com>
 2026-07-07 17:03:06 +00:00
Juan Pérez de Algabaandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> ba22152096 fix(security): block request-level GPU video backend selection withou… (#47259) 
Signed-off-by: jperezde <jperezde@redhat.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-06 02:36:49 -07:00
Aaron HaoandClaude Sonnet 4.6 77a9c5ae28 Weight sync refactor + move sparse nccl engine (#44353) 
Signed-off-by: hao-aaron <ahao@anyscale.com>
Signed-off-by: haoaaron <ahao@anyscale.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-07-01 01:25:19 -07:00
Nils Matteson 93d8f834dd [Core] Pluggable sleep-mode backend abstraction (RFC #34303) (#44074) 2026-06-30 22:00:53 -07:00
Chaojun Zhang 8cc242335d [XPU] Optimize XPU worker shutdown logic to prevent resource leak (#46433) 
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
 2026-06-30 15:27:21 +08:00
Kunshang Ji fb42e5219e [Platform] Replace torch.cuda.mem_get_info with torch.accelerator.get_memory_info (#44825) 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
 2026-06-30 14:39:52 +08:00
Andreas KaratzasandCodex 4eb227992a [ROCm][CI] Make memory sampling less racy in tests and sleep mode (#45490) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Signed-off-by: Codex <codex@example.invalid>
Co-authored-by: Codex <codex@example.invalid>
 2026-06-29 14:26:41 -07:00
Brandon PelfreyandRoger Wang af16446bf3 Vram semaphore infra (#44465) 
Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-06-26 17:32:51 -07:00
Nick Hill 4e07ca2c92 [Core] Add VLLM_GPU_SYNC_CHECK env var (#44800) 2026-06-26 08:24:33 -07:00
Roberto L. Castro d86c66c981 [Feat] Add runtime monitor for post-warmup CuTeDSL compilation (#46167) 2026-06-23 23:33:17 +00:00
peizhang56CursorMatthew WongAndreas Karatzas
 		68afd78897 [Bugfix][ROCm] Fix cumem sleep and teardown (#46203) 
Signed-off-by: pei.zhang <pei.zhang@amd.com>
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Matthew Wong <Matthew.Wong2@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-06-24 02:45:31 +08:00
Tyler Michael SmithClaudeCodexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>kourosh hakhamaneshi
 		ebfbcfe46a Stop setting CUDA_VISIBLE_DEVICES internally in vLLM, add device_ids arg (#45026) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: kourosh hakhamaneshi <kouroshHakha@users.noreply.github.com>
 2026-06-20 13:38:10 -07:00
Thien Tran b2cfae777d Add Triton recompile detection (#45631) 
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
 2026-06-16 18:25:28 +08:00
Yan Ma 3bb46975bd [XPU][Feature] transparent sleep mode support for XPU platform (#37149) 
Signed-off-by: Yan Ma <yan.ma@intel.com>
 2026-06-07 10:45:31 +08:00
qizixi 96229fa99e [KVConnector][1/N] PP-aware handshake aggregation and intermediate-PP output plumbing (#43720) 
Signed-off-by: zixi-qi <zixi@inferact.ai>
 2026-06-04 22:04:19 -07:00
Tyler Michael Smith 4cc78c9d5d [Core] Freeze garbage collector in workers after model initialization (#44363) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-06-04 08:39:04 -07:00
Wei ZhaoandWei Zhao 2427094152 [Feature] Support EPLB for DeepSeek v4 Mega Moe (#43339) 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Wei Zhao (Engrg-Hardware 1) <weizha@login-lyris01.lyris.clusters.nvidia.com>
 2026-06-02 10:56:44 -07:00
Siddharth BedekarandOpenAI Codex 266b9d9c64 [Frontend][Core] Add sparse NCCL weight transfer support for in-place updates (#40096) 
Signed-off-by: Siddharth Bedekar <bedeksid@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-06-01 15:37:30 -04:00
omerpaz95 811d805195 [EC Connector] Add shutdown API to EC Connector. (#42423) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
 2026-05-28 12:28:01 +00:00