307 Commits
Author SHA1 Message Date
Stefano CastagnettaandKevin H. Luu 784d1fff0c [MRV2][Spec Decode] Quantized draft lm_head for drafters that share the target head (#59973)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-15 22:06:40 +00:00
Laith SakkaandNick Hill 9cede34a67 [Bugfix] Create cuBLAS handles before memory profiling to avoid OOM (#61044)
Signed-off-by: Laith Sakka <lsakka@meta.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-10-15 08:59:22 -04:00
Sherif Waly be4ce2e505 [Bugfix] Share PCP-aware local GPU indexing for NUMA binding (#61237) 2026-10-14 10:54:03 +02:00
3ccc0a90b6 [KV Connector] Support sleep mode with MooncakeConnector over RDMA (#59625)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Dakai An <dakaian108@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Dakai An <dakaian108@gmail.com>
Co-authored-by: Dakai An <77474977+andakai@users.noreply.github.com>
2026-10-14 04:18:47 +00:00
Almog TavorandItay Alroy 87e7dce3a7 [MRV2] Keep warmup out of the EPLB load stats (#56716)
Signed-off-by: almogtavor <almogtavor@gmail.com>
Co-authored-by: Itay Alroy <75032521+itayalroy@users.noreply.github.com>
2026-10-14 00:00:48 +00:00
aoshen02andCodex 239a3fa3a8 [Sleep] Retain draft-private weights with existing parameter selector (#61046)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
2026-10-11 23:30:56 +08:00
aoshen02 bd5bbce6e4 [RL] Release weight-transfer communicators without stopping inference (#61060)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: fh3 <a@b>
2026-10-11 21:15:14 +08:00
Xun Sun 26a488ad93 [Fast Start][Bugfix] Snapshot GPU memory after external weights are ready (#60966) 2026-10-11 07:10:04 +00:00
Nick Hill f8471164bf [MRV2] Detatch weight-reload logic from MRV1 (#60937) 2026-10-10 21:15:05 -07:00
Misha GoinandClaude Opus 5.5 29e4d2d42e [Frontend][Build] Add vllm download-kernels for FlashInfer precompiled kernels (#58765)
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 21:22:35 -07:00
Wei ZhaoWei ZhaoCodexzjy0516Claude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
d8fa463e2b [Bugfix][Prefix caching] Fix inconsistent hybrid Mamba prefix cache boundaries (#60533)
Signed-off-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-08 18:33:48 +00:00
Woosuk Kwon 194bd72be7 [Compile] Remove torch.compile-based sequence parallelism and AsyncTP (#60517) 2026-10-08 08:28:23 -07:00
czhu-cohereandaoshen02 6fddc9b21e [Profiler] Add per-session profiling controls to Python and Rust frontends (#57875)
Signed-off-by: czhu <conway.zhu@cohere.com>
Signed-off-by: Conway Zhu <conway.zhu@cohere.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-10-06 05:21:19 +00:00
aoshen02andClaude Opus 5.5 d09ff77637 [Bugfix] Give each data-parallel engine its own global RNG streams (#59788)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 17:38:18 +00:00
Thiago Mota MartinsandFangzhou Ai 92044241a0 [Bugfix][ROCm] Preserve config during GPU memory profiling (#58014)
Co-authored-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com>
2026-10-04 22:28:33 -07:00
aoshen02andClaude Opus 5.5 8e58ac22af [Feature] Offload KV-init runtime state on sleep (#59158)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 09:49:21 +00:00
aoshen02andClaude Opus 5.5 866f02db8a [Feature] Release WorkspaceManager scratch on sleep (#59156)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 02:14:37 +00:00
Robert ShawRobert ShawClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Nick Hill
e5e38ba9b7 [Model Runner V2] Support randomized dummy inputs (#58411)
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-10-01 00:52:26 -07:00
aoshen02andClaude Opus 5.5 643bbef55c [Refactor] Share workspace and model runner init between GPU and XPU workers (#59200)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 14:29:12 +00:00
42d9f8ccdb support rl feature : weight checker (#51350)
Signed-off-by: shiyuan680 <yangcheng104@huawei.com>
Signed-off-by: Ronald1995 <ronaldautomobile@163.com>
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: shiyuan680 <yangcheng104@huawei.com>
Co-authored-by: Ronald1995 <ronaldautomobile@163.com>
Co-authored-by: DeepSeek Harness <noreply@deepseek.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-09-30 03:13:43 +00:00
f5ed3d9c03 [Fast Start] Charge daemon-held weights against gpu_memory_utilization (#57298)
Signed-off-by: liusy58 <mg21330037@smail.nju.edu.cn>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Xun Sun <UNIDY2002@outlook.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Xun Sun <UNIDY2002@outlook.com>
2026-09-29 17:58:35 +00:00
eb0f2ca37f [mooncake] support CUSTOM_MEM_POOL in vllm (#49300)
Signed-off-by: bruce.xu <bruce.xb@alibaba-inc.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: bruce.xu <bruce.xb@alibaba-inc.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-27 00:26:05 +00:00
6491f481a7 [Bugfix] Stop allocator fragmentation from shrinking the KV cache during memory profiling (#58430)
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com>
Signed-off-by: Robert Shaw <robertgshaw2-redhat@h100-01.nemg-001.lab.rdu2.dc.redhat.com>
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Robert Shaw <robertgshaw2-redhat@h100-01.nemg-001.lab.rdu2.dc.redhat.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-25 12:16:27 -04:00
Nick HillandKimi Code e30559b58f [Core] Skip JIT monitor when JIT warmup is disabled (#58590)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
2026-09-24 12:08:16 -07:00
Nick Hill 09fe178dba Revert "[DSpark] Support pipeline-parallel targets in aggregated serving (#56956)" (#58484) 2026-09-24 08:48:36 -07:00
liuzhenwei dcfc17e0b1 [XPU] enable XPU GRAPH by default (#51600)
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
2026-09-24 19:17:25 +08:00
Lucas WilkinsonQiuChunshuoClaude Fable 5.1CodexRobert ShawNick Hillmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
e6dc16cebd [PCP] Support prefill context parallelism with data parallelism (#57075)
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: QiuChunshuo <qiuchunshuo@huawei.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-23 22:32:36 -04:00
157bcb7c48 [DSpark] Support pipeline-parallel targets in aggregated serving (#56956)
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
2026-09-23 04:52:28 -07:00
94f4170df3 [Bugfix] Set worker runtime threads before profiling and compilation (#55891)
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-22 21:12:20 -05:00
pengyihangandmgoin 415e11c6f1 [Bugfix][V1] Honor enable_jit_warmup for V2 kernel warmup (#55146)
Co-authored-by: mgoin <mgoin64@gmail.com>
2026-09-22 15:34:31 -04:00
639461eda4 [RL][Sleep] Retain frozen weights across level-2 sleep (#57891)
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-22 12:36:55 +08:00
0b7f11a1ee [AuxOutput] Add block-keyed storage for routed-expert outputs (#45635)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-21 10:25:05 +00:00
czhu-cohere 3918f3c5a3 [Profiler] Unify platform-aware torch profiling (#57460) 2026-09-20 19:17:23 -07:00
092bdd6d57 [Frontend][Core] Add release_kv_cache_memory() API (#44890)
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: AlanFokCo <alanfok2868@gmail.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com>
Co-authored-by: AlanFokCo <alanfok2868@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-09-18 07:12:12 +08:00
Alexander Leeaoshen02Claude Opus 5aoshen02mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
9e1cd262cb [Bugfix] Use DP index for dense DP weight updates and EC CPU region (#56950)
Signed-off-by: lxy-alexander <lxy_alexander@outlook.com>
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-17 22:44:36 +08:00
Matt 2d8a5c5741 [ROCm][Bugfix] Reduce CUDA graph divergences (#57229)
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>
2026-09-17 05:12:13 +00:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
Taneem Ibrahimandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> ba2ae9f239 [MRV2] Run pooling post processing on non-final PP ranks (#56666)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-14 16:11:11 -07:00
鐘天楽andNick Hill 0a5747d410 [Profiler] Add Proton CUDA graph attribution for MRV2 (#51084)
Signed-off-by: Luossu <zhluosuu@outlook.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-14 19:02:07 +00:00
Lucas WilkinsonandOpenAI Codex 7fe8fc803d [NIXL][PCP][DCP] Expose PCP producer KV shards as transfer ranks (#56645)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-09-13 20:23:15 +00:00
6fbb00b188 [EPD] Add ECMooncakeConnector for encoder cache over Mooncake TransferEngine (#41567)
Signed-off-by: Teng Ma <sima.mt@alibaba-inc.com>
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Tianyu Guo <guoty@inferact.ai>
Co-authored-by: Zhou ziheng <jiaranran2@gmail.com>
Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-07 06:21:43 +00:00
685074cd61 [Bugfix][V2] Warm up kernels before capturing CUDA graphs (#55341)
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-04 22:44:18 +00:00
Derek WeitzelandClaude Fable 5.1 859dd39512 [Bugfix][Core] Wait for the previous PP tensor sends before the next forward pass (#54962)
Signed-off-by: Derek Weitzel <djw8605@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-03 08:01:36 +00:00
4ac452ad98 [Core] Release NCCL communicator memory in sleep mode (#51485)
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-01 14:31:16 +08:00
AI-Infra Rookie c92b29a1d4 [Bugfix] Make metadata send non-blocking in GroupCoordinator.isend_tensor_dict (#49274)
Signed-off-by: z00897177 <zhouxinyi6@huawei.com>
2026-08-30 23:51:25 +00:00
stefankoncarevic 9236159bb6 [ROCm][CI] Keep startup profiling from aborting when free memory grows (#53591)
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
2026-08-28 00:02:40 -07:00
Ronaldandaoshen02 479eeb32d2 [Bugfix][MRV2] Isolate sleep-mode KV allocations (#53508)
Signed-off-by: Ronald1995 <ronaldautomobile@163.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-08-27 15:08:28 -07:00
aoshen02 17da48596c [Bugfix][DP] Synchronize the device on pause completion (#52914)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
2026-08-26 09:50:33 -07:00
ee5ae2ae70 [RL] Add rank-local IPC weight updates (#52497)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
2026-08-26 15:36:57 +08:00