|
fc1c548093
|
Runtime Draft Weight Update for Speculative Decoding (#46725)
Signed-off-by: vx120 <893600387@qq.com>
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: crp0128 <191679376@qq.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
2026-07-11 22:51:53 -07:00 |
|
|
Jiangyun ZhuandYifan Qiao
|
481e481be7
|
[2/N][Core] support partial prefix cache hit for hybrid model (#46384)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
|
2026-07-12 05:37:51 +00:00 |
|
|
|
51878e5b6e
|
[2/N][KV-Cache Layout Refactor] Pack K/V into the content dim across attention backends (#44455)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
|
2026-07-11 11:11:16 -04:00 |
|
|
Yan XuandJingyi Yang
|
95ed0feaa5
|
DCP supports hybrid attention (#40996)
Signed-off-by: YanXu <yancey.yx@alibaba-inc.com>
Signed-off-by: Jingyi Yang <girasoleyang@gmail.com>
Co-authored-by: Jingyi Yang <girasoleyang@gmail.com>
|
2026-07-09 21:34:45 -07:00 |
|
|
|
2d814a0082
|
[kv_offload] Emit tier-owned BlockStored events from FS/OBJ secondary tiers (#47923)
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-07-10 06:17:23 +03:00 |
|
|
|
2ded1b24e7
|
[KV Connector][Mooncake] Apply SWA lookup mask before hashing/key build (#47317)
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude <noreply@anthropic.com>
|
2026-07-09 19:51:23 +00:00 |
|
|
weishu
|
2285cfca46
|
[KVConnector] MultiConnector: give every sub-connector the request's real blocks in update_state_after_alloc (#46865)
Signed-off-by: deng451e <838677410@qq.com>
|
2026-07-09 11:10:39 -07:00 |
|
|
qizixiandNick Hill
|
7802c20c4e
|
[KVConnector][NIXL] Support pipeline-parallel prefill in push mode (#45880)
Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
|
2026-07-08 16:49:23 -07:00 |
|
|
Rishabh Saini
|
2f3f441f84
|
fix: include topic frame in KV events replay response (#45177)
Signed-off-by: RishabhSaini <rishabhsaini01@gmail.com>
|
2026-07-07 14:48:23 -04:00 |
|
|
|
c46ced1ee3
|
[kv_offload] Establish tier-owned KV event handling (#46544)
Signed-off-by: Change72 <changg@nvidia.com>
Signed-off-by: Chang Guo <changg@nvidia.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-07-07 16:55:20 +03:00 |
|
|
|
3354dba381
|
[Bugfix][KV offload] Store interior chunk-boundary blocks under MTP/Eagle (#46972)
Signed-off-by: Mikhail Kostryukov <mike@triptrack.net>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-07-07 13:16:52 +03:00 |
|
|
|
e040899a00
|
[KV Offloading] Add basic offloading metrics (#45958)
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Signed-off-by: Srinivas Krovvidi <194645829+Srinivasoo7@users.noreply.github.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-07-07 09:26:28 +03:00 |
|
|
jialoop-gitandClaude Opus 4.8
|
a4f019fa25
|
fix(distributed): propagate distributed_timeout_seconds to NCCL device groups (#45159)
Signed-off-by: jialoop-git <joane8913456@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
2026-07-07 02:52:51 +00:00 |
|
|
Kunshang JiandHarry Mellor
|
07f9baf756
|
Revert "[Platform] Replace torch.cuda.Event with torch.Event (#47140)" (#47668)
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
|
2026-07-06 14:18:33 +01:00 |
|
|
BadrBasowid
|
740f379fae
|
[ROCm][AITER] Directly Implement AITER Custom All-reduce in CudaCommunicator (#46065)
Signed-off-by: BadrBasowid <badr.basowid@gmail.com>
|
2026-07-06 12:16:32 +00:00 |
|
|
Wentao YeandCanlin Guo
|
b6cc46ec3b
|
[Feature] Support sequence parallel without the need for DP, 1.9%~5.0% E2E Throughput Improvement (#47070)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Co-authored-by: Canlin Guo <canlinguosdu@gmail.com>
|
2026-07-05 05:41:30 -07:00 |
|
|
Aaron HaoandClaude Sonnet 4.6
|
77a9c5ae28
|
Weight sync refactor + move sparse nccl engine (#44353)
Signed-off-by: hao-aaron <ahao@anyscale.com>
Signed-off-by: haoaaron <ahao@anyscale.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
2026-07-01 01:25:19 -07:00 |
|
|
Woosuk KwonandClaude Opus 4.8
|
a264e41975
|
[Distributed] Default FlashInfer allreduce to mnnvl on single node (#47219)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
2026-06-30 18:35:56 -07:00 |
|
|
Woosuk Kwonandsnordmann
|
953bba488d
|
[PERF] Extend NCCL symmetric memory to AllGather and ReduceScatter (#46703)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: snordmann <snordmann@nvidia.com>
|
2026-06-30 11:38:18 -07:00 |
|
|
|
245888ff77
|
[Feature] Detect all2all peer fault with fault tolerance backend and prevent corrupted output (#43637)
Signed-off-by: fangyuchu <fangyuchu@qq.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-06-30 09:00:25 -07:00 |
|
|
Kunshang JiandHarry Mellor
|
e840f0d3f5
|
[Platform] Replace torch.cuda.Event with torch.Event (#47140)
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
|
2026-06-30 08:39:59 -07:00 |
|
|
|
d8f483dc30
|
[Spec Decode] Fix hidden-state extraction block size for hybrid verifiers (#46301)
Signed-off-by: Igor Margulis <igor.margulis@intel.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
|
2026-06-30 08:19:51 -07:00 |
|
|
Arsalan Shakil
|
00ebf19cca
|
[Bugfix][Quant] Raise actionable error instead of bare assert for group-size/TP mismatch (#46230) (#46236)
Signed-off-by: Arsalan Shakil <shakil.arsalan@yahoo.com>
|
2026-06-30 14:57:14 +00:00 |
|
|
Ronen SchafferandOr Ozeri
|
0fc2512094
|
[KV Offload] Pass ScheduleEndContext to on_schedule_end hook (#46450)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-06-30 17:07:12 +03:00 |
|
|
Dakai An
|
0feca7ffa8
|
PD disagg with Mooncake Connector: GDN support (Qwen3.5) and MLA support (Deepseek-V4-Flash) (#46807)
|
2026-06-29 23:29:04 -07:00 |
|
|
weishu
|
77654d080c
|
[KVTransfer] MultiConnector: merge kv_transfer_params dicts across connectors (#46777)
Signed-off-by: deng451e <838677410@qq.com>
|
2026-06-30 00:25:05 +00:00 |
|
|
Ilya MarkovandMarkov Ilya
|
a2abce646f
|
[EPLB] Mask padding in EPLB load recording (#38128)
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Markov Ilya <markovilya19@gmail.com>
|
2026-06-28 19:43:58 -07:00 |
|
|
Wei Zhao
|
95528527ea
|
[Bugfix][Mooncake] Fix Mooncake lookup prefixes with DCP > 1 (#46855)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
|
2026-06-28 14:36:23 -07:00 |
|
|
Varun Sundar RabindranathandVarun Sundar Rabindranath
|
798185d438
|
[KV-Offloading] Fix tensors_per_block stride (#46888)
Signed-off-by: <>
Co-authored-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com>
|
2026-06-27 21:01:45 -04:00 |
|
|
haoyangli0109
|
1744adc256
|
[ROCM] [Communication] Add INT3 quantization method for quickreduce (#45666)
Signed-off-by: Haoyang Li <lihaoyang0109@gmail.com>
|
2026-06-25 15:14:15 +00:00 |
|
|
Asaf Gardinandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
15be78732b
|
[NIXL][Mamba] Add Mamba1 support to NIXL P/D disaggregation (#45019)
Signed-off-by: Josephasafg <ajgard7@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-06-25 05:50:41 -07:00 |
|
|
|
9e88e969c0
|
[Perf][KVConnector][Mooncake] Parallelize KV load with a receive-thread pool (#45971)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-06-24 18:25:12 -07:00 |
|
|
Yifan Qiao
|
b69816043a
|
[Bugfix][MooncakeStore] track resumed requests via scheduler's resumed_req_ids (#46595)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
|
2026-06-24 23:50:56 +00:00 |
|
|
Lucas Wilkinson
|
e7df232288
|
[KV Offload] Gate packed HMA KV cache on cross-layer config (#46252)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
|
2026-06-24 11:55:30 -04:00 |
|
|
Ronen SchafferandOr Ozeri
|
bb61177e49
|
[KV Offloading] Replace bool|None lookup return with LookupResult enum (#46363)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-06-24 18:06:08 +03:00 |
|
|
Nicolò Lucchesi
|
a2cb08b3d5
|
[Misc][PD] Disable bidirectional xfer mode for NixlPushConnector (#46473)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
|
2026-06-24 21:14:05 +08:00 |
|
|
Rui YinandOr Ozeri
|
cf9fd6457e
|
Fix KV offload request-finished lifecycle contract (#46284)
Signed-off-by: test test <2260891073@qq.com>
Signed-off-by: Rui Yin <2260891073@qq.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-06-24 15:42:40 +03:00 |
|
|
Martin Hickey
|
f237e16b41
|
[KV Offload] Replace OffloadingHandler with OffloadingWorker (#45053)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
|
2026-06-24 14:44:24 +03:00 |
|
|
Wei ZhaoandYifan Qiao
|
d20dbf921b
|
[Mooncake] Only check and store new KV cache range (#46412)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
|
2026-06-24 03:10:50 -07:00 |
|
|
Wei Zhao
|
05a0caba91
|
[Mooncake] Optimize lookup pool key string construction (#46188)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
|
2026-06-24 11:51:53 +08:00 |
|
|
|
68afd78897
|
[Bugfix][ROCm] Fix cumem sleep and teardown (#46203)
Signed-off-by: pei.zhang <pei.zhang@amd.com>
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Matthew Wong <Matthew.Wong2@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
|
2026-06-24 02:45:31 +08:00 |
|
|
Andreas Karatzas
|
ceae5bcbda
|
[ROCm][CI] Fix nixl tests (#45219)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
|
2026-06-23 13:11:40 -05:00 |
|
|
|
d32575a2d2
|
[ROCm][P/D] Support MoRIIO heterogeneous TP fan-in (#46332)
Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com>
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: Hongxia Yang <hongxia.yang@amd.com>
Co-authored-by: Jun Kang Chow <junkangchow@gmail.com>
Co-authored-by: Chun Fang <chun.fang@amd.com>
Co-authored-by: TianDi101 <ditian12@amd.com>
Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
|
2026-06-23 10:33:23 +00:00 |
|
|
|
7e47fb72b5
|
[ROCm][P/D] Fix MoRIIO WRITE mode for mixed KV layouts (#46290)
Signed-off-by: Tan Pin Siang <tanpinsiang@gmail.com>
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: Hongxia Yang <hongxia.yang@amd.com>
Co-authored-by: Jun Kang Chow <junkangchow@gmail.com>
Co-authored-by: Chun Fang <chun.fang@amd.com>
Co-authored-by: TianDi101 <ditian12@amd.com>
Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
|
2026-06-23 12:12:51 +08:00 |
|
|
Dao007foreverandClaude Opus 4.8
|
3ce15fd574
|
[v1][kvconnector] DecodeBenchConnector: fill list/tuple (Mamba/KDA) KV caches (#45080)
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
2026-06-22 11:54:00 -07:00 |
|
|
Ilya MarkovandMarkov Ilya
|
ac614587f5
|
[EPLB] Enable nixl eplb communicator for elastic ep (#45013)
Signed-off-by: Markov Ilya <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Markov Ilya <markovilya19@gmail.com>
|
2026-06-22 10:54:08 -07:00 |
|
|
Change72andClaude
|
a9f7b2d41c
|
[feature][kv_offload] Self-describing KV events for OffloadingConnector (#43468)
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: Claude <noreply@anthropic.com>
|
2026-06-22 07:27:46 +00:00 |
|
|
Srinivas Krovvidiandsrinivas_oo7
|
c441ad1c07
|
[KV Offloading] Add labeled metrics support (#45957)
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
|
2026-06-21 18:04:01 +00:00 |
|
|
|
b91b7726e0
|
[ROCm][P/D] Support MiniMax-M3 mixed KV layouts in MoRIIO READ mode (#46039)
Signed-off-by: Jun Kang Chow <junkangchow@gmail.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: Hongxia Yang <hongxia.yang@amd.com>
Co-authored-by: Tan Pin Siang <tanpinsiang@gmail.com>
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: Chun Fang <chun.fang@amd.com>
Co-authored-by: TianDi101 <ditian12@amd.com>
Co-authored-by: functionstackx <47992694+functionstackx@users.noreply.github.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-06-21 12:55:19 +00:00 |
|
|
Palaiologos1453
|
d3ad8e8bcd
|
[Bugfix] Defer offload reads while transfers are pending (#46231)
Signed-off-by: test test <2260891073@qq.com>
|
2026-06-21 14:30:13 +03:00 |
|