iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
19,362 Commits 3 Branches 0 Tags
f1899b2ffdf87598350c416a074c8c291c845902
Commit Graph
248 Commits
Author SHA1 Message Date
aoshen02aoshen02vx120Claude Opus 5Roger Wang
 		b09688a6e7 [Bugfix][Spec Decode] Preserve draft buffers across level-2 sleep (#49774) 
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Signed-off-by: vx120 <893600387@qq.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: vx120 <893600387@qq.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-07-28 00:16:21 -07:00
Schwinn Saereesitthipitak b5b61c622c [Core][Distributed] Add process-checkpoint lifecycle hooks for communicators (starting with Flashinfer) (#46877) 
Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
 2026-07-26 14:47:50 -04:00
fangyuchuzWaNg3a798347923TianZhuo205150940a798347923w00689259zWaNg3yzchang-plusJade Zheng
 		0b0bd2b5f6 [Feature] Add fault tolerance framework (simplified) for DP+EP external LB deployments (#44428) 
Signed-off-by: fangyuchu <fangyuchu@qq.com>
Signed-off-by: a798347923 <2645302020@qq.com>
Signed-off-by: TianZhuo <2770730562@qq.com>
Signed-off-by: a798347923 <39047817+a798347923@users.noreply.github.com>
Signed-off-by: 205150940 <112750056+205150940@users.noreply.github.com>
Signed-off-by: w00689259 <wangzhuo66@huawei.com>
Signed-off-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com>
Signed-off-by: zWaNg3 <389750525@qq.com>
Signed-off-by: yzchang-plus <1078477584@qq.com>
Signed-off-by: Jade Zheng <zheng.shoujian@outlook.com>
Co-authored-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com>
Co-authored-by: a798347923 <2645302020@qq.com>
Co-authored-by: TianZhuo <2770730562@qq.com>
Co-authored-by: 205150940 <112750056+205150940@users.noreply.github.com>
Co-authored-by: a798347923 <39047817+a798347923@users.noreply.github.com>
Co-authored-by: w00689259 <wangzhuo66@huawei.com>
Co-authored-by: zWaNg3 <389750525@qq.com>
Co-authored-by: yzchang-plus <1078477584@qq.com>
Co-authored-by: Jade Zheng <zheng.shoujian@outlook.com>
 2026-07-25 11:49:09 -04:00
Isotr0py 1750e443f2 [Misc] Move PyNvVideoCodec stuff out of gpu worker (#49322) 
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
 2026-07-22 10:11:56 +08:00
Matthew BonannihaosdentClaude Opus 4.7Justin Wood
 		58b2012aa2 [copy of #45208] CuMem slept-L1 fragmentation accounting (#49208) 
Signed-off-by: haosdent <haosdent@gmail.com>
Signed-off-by: Justin Wood <justin.m.wood@me.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: haosdent <haosdent@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Justin Wood <jwood@me.com>
 2026-07-20 23:08:11 +00:00
aoshen02 9459fc6471 [Bugfix][RL] Set vLLM config during weight reload (#45989) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
 2026-07-20 15:02:56 +08:00
devalshahamdandDeval Shah fcd2255d16 [Hardware][GPU] Profiler config additional to increase it scope and annotation details (#37524) 
Signed-off-by: devalshahamd <deval.shah@amd.com>
Signed-off-by: Deval Shah <devashah@amd.com>
Signed-off-by: Deval Shah <deval.shah@amd.com>
Co-authored-by: Deval Shah <devashah@amd.com>
 2026-07-17 13:38:59 -07:00
peizhang56ClaudeAndreas Karatzas
 		05eed72aec [ROCm] Re-enable cudagraph memory profiling, captured on the current stream (#48526) 
Signed-off-by: pei.zhang <pei.zhang@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-07-15 10:03:48 -05:00
Rohan PotdarandClaude Opus 4.8 d973cce3ca Re-disable CUDA graph memory profiling on ROCm (#48440) 
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-13 03:59:20 +00:00
vx120crp0128aoshen02Claude Opus 4.8
 		fc1c548093 Runtime Draft Weight Update for Speculative Decoding (#46725) 
Signed-off-by: vx120 <893600387@qq.com>
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: crp0128 <191679376@qq.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-11 22:51:53 -07:00
peizhang56ClaudeAndreas Karatzas
 		88e5e2c57b [CI/Build][AMD] Fix ROCm OOM in eagle_correctness_heavy by reserving CUDA graph memory (#47366) 
Signed-off-by: pei.zhang <pei.zhang@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-07-10 02:14:38 +00:00
Micah Williamson 766469a4c4 [ROCm] Revert Part of [ROCm] Fix pooling startup workspace lock #47912 (#48154) 
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
 2026-07-09 20:34:24 +00:00
Andreas Karatzas 26831949b4 [ROCm] Fix pooling startup workspace lock (#47912) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-07-08 17:59:50 -05:00
Nils MattesonNils Mattesonmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		e97c3cb303 [Core] Persist and reuse the memory-profiling result across boots (opt-in) (#47388) 
Signed-off-by: Nils Matteson <nils@thaw.sh>
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
Co-authored-by: Nils Matteson <nils@thaw.sh>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-08 00:53:02 +00:00
Nils MattesonNils Mattesonmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		dd0d74cd92 [Doc] Surface the --kv-cache-memory suggestion at INFO and document fast-startup knobs (#47374) 
Signed-off-by: Nils Matteson <nils@thaw.sh>
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
Co-authored-by: Nils Matteson <nils@thaw.sh>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-07 15:05:07 -07:00
Robin c74e751824 [Doc] Fix grammatically incorrect error message in gpu_worker and xpu_worker (#36715) 
Signed-off-by: Hongbin10 <jdmjdm1998@163.com>
 2026-07-07 17:03:06 +00:00
Juan Pérez de Algabaandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> ba22152096 fix(security): block request-level GPU video backend selection withou… (#47259) 
Signed-off-by: jperezde <jperezde@redhat.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-06 02:36:49 -07:00
Aaron HaoandClaude Sonnet 4.6 77a9c5ae28 Weight sync refactor + move sparse nccl engine (#44353) 
Signed-off-by: hao-aaron <ahao@anyscale.com>
Signed-off-by: haoaaron <ahao@anyscale.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-07-01 01:25:19 -07:00
Nils Matteson 93d8f834dd [Core] Pluggable sleep-mode backend abstraction (RFC #34303) (#44074) 2026-06-30 22:00:53 -07:00
Chaojun Zhang 8cc242335d [XPU] Optimize XPU worker shutdown logic to prevent resource leak (#46433) 
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
 2026-06-30 15:27:21 +08:00
Kunshang Ji fb42e5219e [Platform] Replace torch.cuda.mem_get_info with torch.accelerator.get_memory_info (#44825) 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
 2026-06-30 14:39:52 +08:00
Andreas KaratzasandCodex 4eb227992a [ROCm][CI] Make memory sampling less racy in tests and sleep mode (#45490) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Signed-off-by: Codex <codex@example.invalid>
Co-authored-by: Codex <codex@example.invalid>
 2026-06-29 14:26:41 -07:00
Brandon PelfreyandRoger Wang af16446bf3 Vram semaphore infra (#44465) 
Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-06-26 17:32:51 -07:00
Nick Hill 4e07ca2c92 [Core] Add VLLM_GPU_SYNC_CHECK env var (#44800) 2026-06-26 08:24:33 -07:00
Roberto L. Castro d86c66c981 [Feat] Add runtime monitor for post-warmup CuTeDSL compilation (#46167) 2026-06-23 23:33:17 +00:00
peizhang56CursorMatthew WongAndreas Karatzas
 		68afd78897 [Bugfix][ROCm] Fix cumem sleep and teardown (#46203) 
Signed-off-by: pei.zhang <pei.zhang@amd.com>
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Matthew Wong <Matthew.Wong2@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-06-24 02:45:31 +08:00
Tyler Michael SmithClaudeCodexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>kourosh hakhamaneshi
 		ebfbcfe46a Stop setting CUDA_VISIBLE_DEVICES internally in vLLM, add device_ids arg (#45026) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: kourosh hakhamaneshi <kouroshHakha@users.noreply.github.com>
 2026-06-20 13:38:10 -07:00
Thien Tran b2cfae777d Add Triton recompile detection (#45631) 
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
 2026-06-16 18:25:28 +08:00
Yan Ma 3bb46975bd [XPU][Feature] transparent sleep mode support for XPU platform (#37149) 
Signed-off-by: Yan Ma <yan.ma@intel.com>
 2026-06-07 10:45:31 +08:00
qizixi 96229fa99e [KVConnector][1/N] PP-aware handshake aggregation and intermediate-PP output plumbing (#43720) 
Signed-off-by: zixi-qi <zixi@inferact.ai>
 2026-06-04 22:04:19 -07:00
Tyler Michael Smith 4cc78c9d5d [Core] Freeze garbage collector in workers after model initialization (#44363) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-06-04 08:39:04 -07:00
Wei ZhaoandWei Zhao 2427094152 [Feature] Support EPLB for DeepSeek v4 Mega Moe (#43339) 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Wei Zhao (Engrg-Hardware 1) <weizha@login-lyris01.lyris.clusters.nvidia.com>
 2026-06-02 10:56:44 -07:00
Siddharth BedekarandOpenAI Codex 266b9d9c64 [Frontend][Core] Add sparse NCCL weight transfer support for in-place updates (#40096) 
Signed-off-by: Siddharth Bedekar <bedeksid@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-06-01 15:37:30 -04:00
omerpaz95 811d805195 [EC Connector] Add shutdown API to EC Connector. (#42423) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
 2026-05-28 12:28:01 +00:00
Sumanth R Hegdeandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 3cb83c9592 Add model to WeightTransferEngine.__init__ (#42922) 
Signed-off-by: SumanthRH <sumanthrh99@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-22 17:52:15 -07:00
Kebe 19cf334207 [Feature] Support manually enabling the cumem allocator (#33648) 
Signed-off-by: Kebe <mail@kebe7jun.com>
 2026-05-20 08:58:30 -04:00
Wentao YeandNick Hill ae4f59f0ec [Model Runner v2] Oracle for model runner v2 - qwen3 dense model by default [1/N] (#39337) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-05-14 10:02:33 -07:00
Sumanth R Hegde e3b65a5ba0 [feat] Add explicit /start_weight_update and /finish_weight_update APIs for weight transfer (#39212) 2026-05-08 18:03:33 -07:00
Chaojun ZhangandKunshang Ji 1c58876618 [XPU] Disable CUDA graph memory estimate on XPU platform (#41344) 
Signed-off-by: chaojun-zhang <chaojun.zhang@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-05-06 16:38:18 +08:00
Artem PerevedentsevandJiangyun Zhu 8b9ea2f881 [Feature] Add Triton kernel JIT compilation monitor for inference (#40137) 
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
 2026-05-05 14:08:57 +04:00
Matthew Bonanni f03d82efdd [UX][Bugfix] Fix OOM by setting PyTorch max_split_size_mb during model loading (#41268) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-04-30 07:46:54 -07:00
Wentao YeandNick Hill 51fda1ba44 [Model Runner v2] Fix block table IMA issue (#40648) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-04-29 08:30:33 -07:00
Andreas Karatzas 95995bbef8 [ROCm][Engine] Fix GPU memory leaks in engine shutdown and test workaround for async KV prefix cache reset (#38503) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-04-25 05:25:20 +00:00
Srreyansh SethiMatthew Bonannimergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		b7a2605020 [Bugfix] Make Attention Backend Auto-Selection Batch-Invariance-Aware (#40193) 
Signed-off-by: Srreyansh Sethi <srreyansh.sethi@gmail.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-04-23 14:57:03 +00:00
Wentao Yeandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 8f87eb4622 [Refactor] Clean up log once scope="local" (#40540) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-04-22 16:42:43 -04:00
Matthew BonanniandTyler Michael Smith 96a85c5750 [Startup][UX] Enable CUDAGraph memory profiling by default (#38284) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-04-21 18:16:59 -04:00
Matthew Bonanni ab5666eb7c [UX] Bump version in CG memory profiling log message (#40465) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-04-21 15:26:06 +00:00
Chaojun ZhangandKunshang Ji 4f4713f96e [XPU] [torch.compile] Skipping CUDA graph memory estimation to avoid startup errors. (#39977) 
Signed-off-by: chaojun-zhang <chaojun.zhang@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-04-20 13:04:39 +08:00
Lucas Kabela c08f3b2a62 Measure encoder compile time seperate from llm backbone (#39240) 
Signed-off-by: Lucas Kabela <lucaskabela@meta.com>
 2026-04-14 10:52:49 -04:00
Matthew Bonanni 116f4be405 [1/N][Cleanup] Standardize on use of is_quantized_kv_cache (#38659) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-04-01 04:08:01 +00:00