85b3a7264b
[Bugfix][Model Runner V2] Order uniform decodes first so spec decodes aren't misclassified as prefills ( #47381 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-09 14:26:27 +01:00
Benjamin Chislett and Wentao Ye
1cd75b3dd4
[Bugfix] Fix race condition in KVBlockZeroer ( #48085 )
...
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com >
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-07-09 09:18:19 +00:00
qizixi and Nick Hill
7802c20c4e
[KVConnector][NIXL] Support pipeline-parallel prefill in push mode ( #45880 )
...
Signed-off-by: zixi-qi <zixi@inferact.ai >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-08 16:49:23 -07:00
Nick Hill
a5d19cbb95
[Core] Move MRV1 late_interaction_runner.py out of MRV2 subtree ( #48014 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-07-08 18:30:11 +00:00
cd0de48d08
[Bugfix][V1] Free out-of-window blocks on the processed-token basis under async scheduling ( #47728 )
...
Signed-off-by: Saddss <28726669061@qq.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Saddss <28726669061@qq.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-08 13:34:19 +01:00
Walter Beller-Morales
dd127d82ed
[Core][Engine] only materialize tokens when thinking budget is in req ( #47053 )
...
Signed-off-by: walterbm <walter.beller.morales@gmail.com >
2026-07-07 21:02:38 -06:00
Posedge_Lin and Or Ozeri
0ca6eee743
[Core] Pass request context to CPU offload cache policy touch ( #47744 )
...
Signed-off-by: jacklin78911-collab <jacklin78911@gmail.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-07-08 05:56:25 +03:00
e97c3cb303
[Core] Persist and reuse the memory-profiling result across boots (opt-in) ( #47388 )
...
Signed-off-by: Nils Matteson <nils@thaw.sh >
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com >
Co-authored-by: Nils Matteson <nils@thaw.sh >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-08 00:53:02 +00:00
Sahil Kadadekar and Claude Fable 5
55da232db6
[Bugfix] Pad Mamba page size instead of scaling block_size in unify_kv_cache_spec_page_size ( #45207 )
...
Signed-off-by: Sahil170595 <147995121+Sahil170595@users.noreply.github.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-07-07 22:01:34 +00:00
c46ced1ee3
[kv_offload] Establish tier-owned KV event handling ( #46544 )
...
Signed-off-by: Change72 <changg@nvidia.com >
Signed-off-by: Chang Guo <changg@nvidia.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-07-07 16:55:20 +03:00
Dakai An and Nick Hill
65dcde1695
[Bugfix] Fix PD disagg + MTP correctness for Qwen3.5(GDN) ( #47466 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-07 13:51:22 +00:00
Pierangelo Di Pilato and Or Ozeri
65a7b46284
[KV-Offloading] Support workload identity for objectstore secondary tier ( #47063 )
...
Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-07-07 16:30:16 +03:00
93e2ab7111
Disable dynamic speculative decoding when DP is enabled ( #45963 )
...
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
2026-07-07 12:52:41 +00:00
Harry Mellor
dd94484577
Bump Transformers version to 5.10.4 ( #41359 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-07 05:13:28 -07:00
Guan-Ming Chiu
ed051fab54
[Bugfix] Reject sampling params unsupported by diffusion models ( #45418 )
...
Signed-off-by: Guan-Ming (Wesley) Chiu <105915352+guan404ming@users.noreply.github.com >
2026-07-07 11:25:36 +00:00
3354dba381
[Bugfix][KV offload] Store interior chunk-boundary blocks under MTP/Eagle ( #46972 )
...
Signed-off-by: Mikhail Kostryukov <mike@triptrack.net >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-07-07 13:16:52 +03:00
cbb5f045be
[ROCm][CI] Refresh ROCm base images when docker rocm_base changes ( #46904 )
...
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com >
Signed-off-by: Codex <codex@example.invalid >
Signed-off-by: Micah Williamson <micah.williamson@amd.com >
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com >
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: Rohan138 <rohanpotdar138@gmail.com >
Co-authored-by: Codex <codex@example.invalid >
Co-authored-by: Micah Williamson <micah.williamson@amd.com >
Co-authored-by: Matthew Wong <Matthew.Wong2@amd.com >
2026-07-07 03:10:50 -07:00
Kyung Sub Lee (Daniel) and Claude Fable 5
c5b66233b2
[Bugfix][Spec Decode] Skip uniform spec-decode padding for diffusion models ( #47464 )
...
Signed-off-by: kl527 <kl527@cornell.edu >
Signed-off-by: Kyung Sub Lee (Daniel) <66861800+kl527@users.noreply.github.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-07-07 09:25:12 +00:00
e040899a00
[KV Offloading] Add basic offloading metrics ( #45958 )
...
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com >
Signed-off-by: Srinivas Krovvidi <194645829+Srinivasoo7@users.noreply.github.com >
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com >
Co-authored-by: Or Ozeri <or@ozery.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-07-07 09:26:28 +03:00
Ting SUN and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
373eb314af
[Bugfix][Core] Fix num_output_placeholders underflow with async scheduling + spec decode ( #46066 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-06 13:50:38 +00:00
Kunshang Ji and Harry Mellor
07f9baf756
Revert "[Platform] Replace torch.cuda.Event with torch.Event ( #47140 )" ( #47668 )
...
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-06 14:18:33 +01:00
BadrBasowid
740f379fae
[ROCm][AITER] Directly Implement AITER Custom All-reduce in CudaCommunicator ( #46065 )
...
Signed-off-by: BadrBasowid <badr.basowid@gmail.com >
2026-07-06 12:16:32 +00:00
Andreas Karatzas
8f0e75e16b
[ROCm][CI] Adding nixl multiconn ( #47481 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-07-06 15:04:58 +08:00
Liangliang Ma and Kunshang Ji
69715823df
[Test][XPU] Skip fork in kv_sharing_fast_prefill test on XPU ( #47406 )
...
Signed-off-by: Ma, Liangliang <liangliang.ma@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-07-06 11:32:26 +08:00
alex101-ops and zc02384840
f2aaf59151
[Feature] Support MTP speculative decoding for Bailing hybrid models ( #44880 )
...
Signed-off-by: zc02384840 <zc02384840@antgroup.com >
Co-authored-by: zc02384840 <zc02384840@antgroup.com >
2026-07-06 10:38:50 +08:00
78a04c208d
[XPU] Fix CUDA API shims breaking Torch Dynamo during AOT compile ( #43092 )
...
Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-07-06 08:29:20 +08:00
Harry Mellor
f1445f6dbd
[CI] Bump huggingface-hub from v1.10.2 to v1.22.0 ( #47551 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-04 07:45:45 -07:00
yue.yu and Benjamin Chislett
e7c9df9449
[Bugfix][Structured Output][Spec Decode] Constrain bitmask and trim grammar advance at the reasoning boundary ( #44297 )
...
Signed-off-by: Allen.Yu <yuyue0225sc@163.com >
Signed-off-by: yue.yu <yuyue0225sc@163.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
2026-07-04 09:08:45 +00:00
07516fda67
[MRV2][SD] Make Dynamic SD comatible with Full Cuda Graphs ( #45953 )
...
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-03 23:58:27 -04:00
adamkbaranowski
4c3c64fcf7
Add Laguna XS.2.1 DFlash drafter support ( #46853 )
...
Signed-off-by: Adam Baranowski <adam.baranowski@poolside.ai >
2026-07-02 18:09:27 -07:00
Andreas Karatzas
442ccc6098
[ROCm][CI] Adding extract hs 2gpu ( #47482 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-07-02 17:59:38 -07:00
Dan Blanaru and Matthew Bonanni
d29125c085
Xqa decode kernels ( #43232 )
...
Signed-off-by: Dan Blanaru <48605845+DanBlanaru@users.noreply.github.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-07-02 12:32:05 -07:00
Nick Hill
e392bf7a68
[BugFix][MRV2] Ensure all req slots are accounted for when scheduling ( #46974 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-07-02 10:19:24 -07:00
Woosuk Kwon and Claude Fable 5
a47f38f825
[Bugfix][Model Runner V2][Spec Decode] Fix int32 offset overflow in block verification kernels ( #47383 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-07-02 07:32:38 -07:00
Wentao Ye and Nick Hill
a2f713002d
[ModelRunner V2] Enable by default for all dense models ( #44443 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-02 18:48:57 +08:00
3af8789559
[Feature] Universal speculative decoding for heterogeneous vocabularies (TLI) ( #38174 )
...
Signed-off-by: wan-danfeng <wandanfeng0802@gmail.com >
Signed-off-by: Wonderful <wandanfeng0802@gmail.com >
Co-authored-by: Wan_DF <wonderful199082@126.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
2026-07-02 01:34:20 -07:00
peizhang56 and Claude
7fe7fa9cda
[CI][Bugfix] Rerun test_engine_log_metrics_ray on Ray GCS startup timeout ( #47208 )
...
Signed-off-by: pei.zhang <pei.zhang@amd.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-07-01 21:32:09 -05:00
Yiliu Dong and Nick Hill
00eb7cefa3
[Bugfix] Prevent padding placeholders from reaching embeddings ( #47029 )
...
Signed-off-by: qianlihuang <91178480+qianlihuang@users.noreply.github.com >
Signed-off-by: Yiliu Dong <91178480+qianlihuang@users.noreply.github.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-01 09:26:03 -07:00
f5a8d73377
[Spec Decode] DSpark ( #46995 )
...
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com >
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai >
Co-authored-by: mgoin <mgoin64@gmail.com >
2026-07-01 08:30:24 -07:00
Nick Hill
f1cf6b0086
[CI] Fix segfault in tracing test ( #47299 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-07-01 14:00:37 +00:00
Harry Mellor
e7d0fcbc09
[CI] Fix various failures on main ( #47197 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-01 10:35:34 +01:00
Aaron Hao and Claude Sonnet 4.6
77a9c5ae28
Weight sync refactor + move sparse nccl engine ( #44353 )
...
Signed-off-by: hao-aaron <ahao@anyscale.com >
Signed-off-by: haoaaron <ahao@anyscale.com >
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-07-01 01:25:19 -07:00
Nils Matteson
a461070d1c
[Core] Make sleep-mode backend capability flags communicator-agnostic ( #47243 )
2026-07-01 07:17:44 +00:00
Nils Matteson
93d8f834dd
[Core] Pluggable sleep-mode backend abstraction (RFC #34303 ) ( #44074 )
2026-06-30 22:00:53 -07:00
c5200d3565
[Attention][DSA] support dcp for FLASHINFER_MLA_SPARSE ( #46076 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Jingyi Yang <girasoleyang@gmail.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: GirasoleY <girasoleyang@gmail.com >
Co-authored-by: Jingyi Yang <girasoleyang@gmail.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-07-01 00:32:20 -04:00
Ting SUN
ac521f6237
[Bugfix][Structured Outputs] Reject degenerate structured_outputs that crash EngineCore ( #45346 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
2026-06-30 22:41:33 +00:00
Andreas Karatzas
c8f9c156a5
[ROCm][V1][MLA] Clone prefill backend state per metadata builder ( #46993 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-06-30 11:43:54 -07:00
727971f1c1
Add Medusa speculative decoding e2e test ( #41396 )
...
Signed-off-by: Anshika Ojha <anshikao@nvidia.com >
Signed-off-by: Rishi Puri <riship@nvidia.com >
Signed-off-by: Rishi Puri <puririshi98@berkeley.edu >
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
Co-authored-by: Anshika Ojha <215760622+ojhaanshika@users.noreply.github.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Stefano Castagnetta <scastagnetta@nvidia.com >
2026-06-30 18:02:22 +00:00
liuzhenwei
7a341fa109
[XPU] Support ZE_AFFINITY_MASK passthrough in xpu_disagg_acc_test ( #47105 )
...
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com >
2026-06-30 17:06:12 +00:00
Kunshang Ji and Harry Mellor
e840f0d3f5
[Platform] Replace torch.cuda.Event with torch.Event ( #47140 )
...
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-06-30 08:39:59 -07:00