iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
21,722 Commits 3 Branches 0 Tags
382970ee6ca490aeaaaf4e32c53695b581ff61ba
Commit Graph
276 Commits
Author SHA1 Message Date
Hongxin Xuaoshen02OpenAI CodexRoger WangNick HillCodex
 		0b7f11a1ee [AuxOutput] Add block-keyed storage for routed-expert outputs (#45635) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-21 10:25:05 +00:00
czhu-cohere 3918f3c5a3 [Profiler] Unify platform-aware torch profiling (#57460) 2026-09-20 19:17:23 -07:00
Dakai AnAlanFokCoaoshen02OpenAI Codex
 		092bdd6d57 [Frontend][Core] Add release_kv_cache_memory() API (#44890) 
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: AlanFokCo <alanfok2868@gmail.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com>
Co-authored-by: AlanFokCo <alanfok2868@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-09-18 07:12:12 +08:00
Alexander Leeaoshen02Claude Opus 5aoshen02mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		9e1cd262cb [Bugfix] Use DP index for dense DP weight updates and EC CPU region (#56950) 
Signed-off-by: lxy-alexander <lxy_alexander@outlook.com>
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-17 22:44:36 +08:00
Matt 2d8a5c5741 [ROCm][Bugfix] Reduce CUDA graph divergences (#57229) 
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>
 2026-09-17 05:12:13 +00:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 16:44:29 +00:00
Taneem Ibrahimandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> ba2ae9f239 [MRV2] Run pooling post processing on non-final PP ranks (#56666) 
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-14 16:11:11 -07:00
鐘天楽andNick Hill 0a5747d410 [Profiler] Add Proton CUDA graph attribution for MRV2 (#51084) 
Signed-off-by: Luossu <zhluosuu@outlook.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-09-14 19:02:07 +00:00
Lucas WilkinsonandOpenAI Codex 7fe8fc803d [NIXL][PCP][DCP] Expose PCP producer KV shards as transfer ranks (#56645) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-09-13 20:23:15 +00:00
Teng MaCursorTianyu GuoZhou zihengjiangkuaixue123OpenAI Codex
 		6fbb00b188 [EPD] Add ECMooncakeConnector for encoder cache over Mooncake TransferEngine (#41567) 
Signed-off-by: Teng Ma <sima.mt@alibaba-inc.com>
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Tianyu Guo <guoty@inferact.ai>
Co-authored-by: Zhou ziheng <jiaranran2@gmail.com>
Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-09-07 06:21:43 +00:00
aoshen02aoshen02Nick Hill
 		685074cd61 [Bugfix][V2] Warm up kernels before capturing CUDA graphs (#55341) 
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-09-04 22:44:18 +00:00
Derek WeitzelandClaude Fable 5.1 859dd39512 [Bugfix][Core] Wait for the previous PP tensor sends before the next forward pass (#54962) 
Signed-off-by: Derek Weitzel <djw8605@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-03 08:01:36 +00:00
aoshen02aoshen02Claude Fable 5
 		4ac452ad98 [Core] Release NCCL communicator memory in sleep mode (#51485) 
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-09-01 14:31:16 +08:00
AI-Infra Rookie c92b29a1d4 [Bugfix] Make metadata send non-blocking in GroupCoordinator.isend_tensor_dict (#49274) 
Signed-off-by: z00897177 <zhouxinyi6@huawei.com>
 2026-08-30 23:51:25 +00:00
stefankoncarevic 9236159bb6 [ROCm][CI] Keep startup profiling from aborting when free memory grows (#53591) 
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
 2026-08-28 00:02:40 -07:00
Ronaldandaoshen02 479eeb32d2 [Bugfix][MRV2] Isolate sleep-mode KV allocations (#53508) 
Signed-off-by: Ronald1995 <ronaldautomobile@163.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
 2026-08-27 15:08:28 -07:00
aoshen02 17da48596c [Bugfix][DP] Synchronize the device on pause completion (#52914) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
 2026-08-26 09:50:33 -07:00
aoshen02OpenAI CodexOpenAI CodexRoger Wang
 		ee5ae2ae70 [RL] Add rank-local IPC weight updates (#52497) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-08-26 15:36:57 +08:00
Wentao Ye f59bb0bd8c [Refactor] Refactor batch invariance folder (#53619) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-08-24 14:31:36 -07:00
Tianyu Guo 7ca49fbe4b [Refactor][Model Runner V2][Multimodal] Move the encoder-only path out of the shared runner (#53176) 
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
 2026-08-22 05:11:44 +00:00
Lucas Wilkinson 8bdc70ec7b [6/N][KV-Cache Layout Refactor] Standardize KV cache layout (#51718) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-08-21 17:47:49 -07:00
Itay Alroy a2257f95b7 [Elastic EP] Reduce eager-mode reconfiguration downtime (#51885) 
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
 2026-08-19 02:13:06 +00:00
Gabriel Wu 97388c44f9 [Bugfix] Make DSV4 sparse MLA work end-to-end for plain decode, MTP, and DSpark (#51538) 2026-08-15 07:01:32 -07:00
Connor CarpenterandNick Hill f473870ecf [Rust Frontend][gRPC] Add RL lifecycle control (#51316) 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-08-14 11:32:39 -07:00
鐘天楽 a367cdbca0 [Profiler] Add minimal Triton Proton profiling backend (#48789) 
Signed-off-by: Luossu <zhluosuu@outlook.com>
 2026-08-11 14:51:57 -07:00
Silen NaihinDakai AnJiangyun Zhu
 		b50fdebce0 [Bugfix] Fix level-2 sleep/wake/reload with enable_lora=True (#39935) 
Signed-off-by: Silen Naihin <silen.naihin@gmail.com>
Co-authored-by: Dakai An <dakaian108@gmail.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
 2026-08-06 09:13:10 +08:00
Nick HillandClaude 7635a9002b [Core] Explicitly manage torch CPU threads in workers (#49919) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-08-04 10:06:38 -07:00
aoshen02aoshen02vx120Claude Opus 5Roger Wang
 		b09688a6e7 [Bugfix][Spec Decode] Preserve draft buffers across level-2 sleep (#49774) 
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Signed-off-by: vx120 <893600387@qq.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: vx120 <893600387@qq.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-07-28 00:16:21 -07:00
Schwinn Saereesitthipitak b5b61c622c [Core][Distributed] Add process-checkpoint lifecycle hooks for communicators (starting with Flashinfer) (#46877) 
Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
 2026-07-26 14:47:50 -04:00
fangyuchuzWaNg3a798347923TianZhuo205150940a798347923w00689259zWaNg3yzchang-plusJade Zheng
 		0b0bd2b5f6 [Feature] Add fault tolerance framework (simplified) for DP+EP external LB deployments (#44428) 
Signed-off-by: fangyuchu <fangyuchu@qq.com>
Signed-off-by: a798347923 <2645302020@qq.com>
Signed-off-by: TianZhuo <2770730562@qq.com>
Signed-off-by: a798347923 <39047817+a798347923@users.noreply.github.com>
Signed-off-by: 205150940 <112750056+205150940@users.noreply.github.com>
Signed-off-by: w00689259 <wangzhuo66@huawei.com>
Signed-off-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com>
Signed-off-by: zWaNg3 <389750525@qq.com>
Signed-off-by: yzchang-plus <1078477584@qq.com>
Signed-off-by: Jade Zheng <zheng.shoujian@outlook.com>
Co-authored-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com>
Co-authored-by: a798347923 <2645302020@qq.com>
Co-authored-by: TianZhuo <2770730562@qq.com>
Co-authored-by: 205150940 <112750056+205150940@users.noreply.github.com>
Co-authored-by: a798347923 <39047817+a798347923@users.noreply.github.com>
Co-authored-by: w00689259 <wangzhuo66@huawei.com>
Co-authored-by: zWaNg3 <389750525@qq.com>
Co-authored-by: yzchang-plus <1078477584@qq.com>
Co-authored-by: Jade Zheng <zheng.shoujian@outlook.com>
 2026-07-25 11:49:09 -04:00
Isotr0py 1750e443f2 [Misc] Move PyNvVideoCodec stuff out of gpu worker (#49322) 
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
 2026-07-22 10:11:56 +08:00
Matthew BonannihaosdentClaude Opus 4.7Justin Wood
 		58b2012aa2 [copy of #45208] CuMem slept-L1 fragmentation accounting (#49208) 
Signed-off-by: haosdent <haosdent@gmail.com>
Signed-off-by: Justin Wood <justin.m.wood@me.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: haosdent <haosdent@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Justin Wood <jwood@me.com>
 2026-07-20 23:08:11 +00:00
aoshen02 9459fc6471 [Bugfix][RL] Set vLLM config during weight reload (#45989) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
 2026-07-20 15:02:56 +08:00
devalshahamdandDeval Shah fcd2255d16 [Hardware][GPU] Profiler config additional to increase it scope and annotation details (#37524) 
Signed-off-by: devalshahamd <deval.shah@amd.com>
Signed-off-by: Deval Shah <devashah@amd.com>
Signed-off-by: Deval Shah <deval.shah@amd.com>
Co-authored-by: Deval Shah <devashah@amd.com>
 2026-07-17 13:38:59 -07:00
peizhang56ClaudeAndreas Karatzas
 		05eed72aec [ROCm] Re-enable cudagraph memory profiling, captured on the current stream (#48526) 
Signed-off-by: pei.zhang <pei.zhang@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-07-15 10:03:48 -05:00
Rohan PotdarandClaude Opus 4.8 d973cce3ca Re-disable CUDA graph memory profiling on ROCm (#48440) 
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-13 03:59:20 +00:00
vx120crp0128aoshen02Claude Opus 4.8
 		fc1c548093 Runtime Draft Weight Update for Speculative Decoding (#46725) 
Signed-off-by: vx120 <893600387@qq.com>
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: crp0128 <191679376@qq.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-11 22:51:53 -07:00
peizhang56ClaudeAndreas Karatzas
 		88e5e2c57b [CI/Build][AMD] Fix ROCm OOM in eagle_correctness_heavy by reserving CUDA graph memory (#47366) 
Signed-off-by: pei.zhang <pei.zhang@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-07-10 02:14:38 +00:00
Micah Williamson 766469a4c4 [ROCm] Revert Part of [ROCm] Fix pooling startup workspace lock #47912 (#48154) 
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
 2026-07-09 20:34:24 +00:00
Andreas Karatzas 26831949b4 [ROCm] Fix pooling startup workspace lock (#47912) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-07-08 17:59:50 -05:00
Nils MattesonNils Mattesonmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		e97c3cb303 [Core] Persist and reuse the memory-profiling result across boots (opt-in) (#47388) 
Signed-off-by: Nils Matteson <nils@thaw.sh>
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
Co-authored-by: Nils Matteson <nils@thaw.sh>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-08 00:53:02 +00:00
Nils MattesonNils Mattesonmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		dd0d74cd92 [Doc] Surface the --kv-cache-memory suggestion at INFO and document fast-startup knobs (#47374) 
Signed-off-by: Nils Matteson <nils@thaw.sh>
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
Co-authored-by: Nils Matteson <nils@thaw.sh>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-07 15:05:07 -07:00
Robin c74e751824 [Doc] Fix grammatically incorrect error message in gpu_worker and xpu_worker (#36715) 
Signed-off-by: Hongbin10 <jdmjdm1998@163.com>
 2026-07-07 17:03:06 +00:00
Juan Pérez de Algabaandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> ba22152096 fix(security): block request-level GPU video backend selection withou… (#47259) 
Signed-off-by: jperezde <jperezde@redhat.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-06 02:36:49 -07:00
Aaron HaoandClaude Sonnet 4.6 77a9c5ae28 Weight sync refactor + move sparse nccl engine (#44353) 
Signed-off-by: hao-aaron <ahao@anyscale.com>
Signed-off-by: haoaaron <ahao@anyscale.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-07-01 01:25:19 -07:00
Nils Matteson 93d8f834dd [Core] Pluggable sleep-mode backend abstraction (RFC #34303) (#44074) 2026-06-30 22:00:53 -07:00
Chaojun Zhang 8cc242335d [XPU] Optimize XPU worker shutdown logic to prevent resource leak (#46433) 
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
 2026-06-30 15:27:21 +08:00
Kunshang Ji fb42e5219e [Platform] Replace torch.cuda.mem_get_info with torch.accelerator.get_memory_info (#44825) 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
 2026-06-30 14:39:52 +08:00
Andreas KaratzasandCodex 4eb227992a [ROCm][CI] Make memory sampling less racy in tests and sleep mode (#45490) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Signed-off-by: Codex <codex@example.invalid>
Co-authored-by: Codex <codex@example.invalid>
 2026-06-29 14:26:41 -07:00