Commit Graph
1555 Commits
Author SHA1 Message Date
85b3a7264b [Bugfix][Model Runner V2] Order uniform decodes first so spec decodes aren't misclassified as prefills (#47381)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-07-09 14:26:27 +01:00
Benjamin ChislettandWentao Ye 1cd75b3dd4 [Bugfix] Fix race condition in KVBlockZeroer (#48085)
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-07-09 09:18:19 +00:00
qizixiandNick Hill 7802c20c4e [KVConnector][NIXL] Support pipeline-parallel prefill in push mode (#45880)
Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-07-08 16:49:23 -07:00
Nick Hill a5d19cbb95 [Core] Move MRV1 late_interaction_runner.py out of MRV2 subtree (#48014)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-07-08 18:30:11 +00:00
cd0de48d08 [Bugfix][V1] Free out-of-window blocks on the processed-token basis under async scheduling (#47728)
Signed-off-by: Saddss <28726669061@qq.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Saddss <28726669061@qq.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-07-08 13:34:19 +01:00
Walter Beller-Morales dd127d82ed [Core][Engine] only materialize tokens when thinking budget is in req (#47053)
Signed-off-by: walterbm <walter.beller.morales@gmail.com>
2026-07-07 21:02:38 -06:00
Posedge_LinandOr Ozeri 0ca6eee743 [Core] Pass request context to CPU offload cache policy touch (#47744)
Signed-off-by: jacklin78911-collab <jacklin78911@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-08 05:56:25 +03:00
Nils MattesonNils Mattesonmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
e97c3cb303 [Core] Persist and reuse the memory-profiling result across boots (opt-in) (#47388)
Signed-off-by: Nils Matteson <nils@thaw.sh>
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
Co-authored-by: Nils Matteson <nils@thaw.sh>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-08 00:53:02 +00:00
Sahil KadadekarandClaude Fable 5 55da232db6 [Bugfix] Pad Mamba page size instead of scaling block_size in unify_kv_cache_spec_page_size (#45207)
Signed-off-by: Sahil170595 <147995121+Sahil170595@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 22:01:34 +00:00
c46ced1ee3 [kv_offload] Establish tier-owned KV event handling (#46544)
Signed-off-by: Change72 <changg@nvidia.com>
Signed-off-by: Chang Guo <changg@nvidia.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-07 16:55:20 +03:00
Dakai AnandNick Hill 65dcde1695 [Bugfix] Fix PD disagg + MTP correctness for Qwen3.5(GDN) (#47466)
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-07-07 13:51:22 +00:00
Pierangelo Di PilatoandOr Ozeri 65a7b46284 [KV-Offloading] Support workload identity for objectstore secondary tier (#47063)
Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-07 16:30:16 +03:00
Tyler Michael Smithmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Claude Opus 4.6
93e2ab7111 Disable dynamic speculative decoding when DP is enabled (#45963)
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-07 12:52:41 +00:00
Harry Mellor dd94484577 Bump Transformers version to 5.10.4 (#41359)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-07-07 05:13:28 -07:00
Guan-Ming Chiu ed051fab54 [Bugfix] Reject sampling params unsupported by diffusion models (#45418)
Signed-off-by: Guan-Ming (Wesley) Chiu <105915352+guan404ming@users.noreply.github.com>
2026-07-07 11:25:36 +00:00
3354dba381 [Bugfix][KV offload] Store interior chunk-boundary blocks under MTP/Eagle (#46972)
Signed-off-by: Mikhail Kostryukov <mike@triptrack.net>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-07 13:16:52 +03:00
cbb5f045be [ROCm][CI] Refresh ROCm base images when docker rocm_base changes (#46904)
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Signed-off-by: Codex <codex@example.invalid>
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Rohan138 <rohanpotdar138@gmail.com>
Co-authored-by: Codex <codex@example.invalid>
Co-authored-by: Micah Williamson <micah.williamson@amd.com>
Co-authored-by: Matthew Wong <Matthew.Wong2@amd.com>
2026-07-07 03:10:50 -07:00
Kyung Sub Lee (Daniel)andClaude Fable 5 c5b66233b2 [Bugfix][Spec Decode] Skip uniform spec-decode padding for diffusion models (#47464)
Signed-off-by: kl527 <kl527@cornell.edu>
Signed-off-by: Kyung Sub Lee (Daniel) <66861800+kl527@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 09:25:12 +00:00
e040899a00 [KV Offloading] Add basic offloading metrics (#45958)
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Signed-off-by: Srinivas Krovvidi <194645829+Srinivasoo7@users.noreply.github.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-07 09:26:28 +03:00
Ting SUNandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 373eb314af [Bugfix][Core] Fix num_output_placeholders underflow with async scheduling + spec decode (#46066)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-06 13:50:38 +00:00
Kunshang JiandHarry Mellor 07f9baf756 Revert "[Platform] Replace torch.cuda.Event with torch.Event (#47140)" (#47668)
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-07-06 14:18:33 +01:00
BadrBasowid 740f379fae [ROCm][AITER] Directly Implement AITER Custom All-reduce in CudaCommunicator (#46065)
Signed-off-by: BadrBasowid <badr.basowid@gmail.com>
2026-07-06 12:16:32 +00:00
Andreas Karatzas 8f0e75e16b [ROCm][CI] Adding nixl multiconn (#47481)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
2026-07-06 15:04:58 +08:00
Liangliang MaandKunshang Ji 69715823df [Test][XPU] Skip fork in kv_sharing_fast_prefill test on XPU (#47406)
Signed-off-by: Ma, Liangliang <liangliang.ma@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-07-06 11:32:26 +08:00
alex101-opsandzc02384840 f2aaf59151 [Feature] Support MTP speculative decoding for Bailing hybrid models (#44880)
Signed-off-by: zc02384840 <zc02384840@antgroup.com>
Co-authored-by: zc02384840 <zc02384840@antgroup.com>
2026-07-06 10:38:50 +08:00
Łukasz Ślusarczykmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Kunshang Ji
78a04c208d [XPU] Fix CUDA API shims breaking Torch Dynamo during AOT compile (#43092)
Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-07-06 08:29:20 +08:00
Harry Mellor f1445f6dbd [CI] Bump huggingface-hub from v1.10.2 to v1.22.0 (#47551)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-07-04 07:45:45 -07:00
yue.yuandBenjamin Chislett e7c9df9449 [Bugfix][Structured Output][Spec Decode] Constrain bitmask and trim grammar advance at the reasoning boundary (#44297)
Signed-off-by: Allen.Yu <yuyue0225sc@163.com>
Signed-off-by: yue.yu <yuyue0225sc@163.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
2026-07-04 09:08:45 +00:00
07516fda67 [MRV2][SD] Make Dynamic SD comatible with Full Cuda Graphs (#45953)
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-07-03 23:58:27 -04:00
adamkbaranowski 4c3c64fcf7 Add Laguna XS.2.1 DFlash drafter support (#46853)
Signed-off-by: Adam Baranowski <adam.baranowski@poolside.ai>
2026-07-02 18:09:27 -07:00
Andreas Karatzas 442ccc6098 [ROCm][CI] Adding extract hs 2gpu (#47482)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
2026-07-02 17:59:38 -07:00
Dan BlanaruandMatthew Bonanni d29125c085 Xqa decode kernels (#43232)
Signed-off-by: Dan Blanaru <48605845+DanBlanaru@users.noreply.github.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
2026-07-02 12:32:05 -07:00
Nick Hill e392bf7a68 [BugFix][MRV2] Ensure all req slots are accounted for when scheduling (#46974)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-07-02 10:19:24 -07:00
Woosuk KwonandClaude Fable 5 a47f38f825 [Bugfix][Model Runner V2][Spec Decode] Fix int32 offset overflow in block verification kernels (#47383)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-02 07:32:38 -07:00
Wentao YeandNick Hill a2f713002d [ModelRunner V2] Enable by default for all dense models (#44443)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-07-02 18:48:57 +08:00
3af8789559 [Feature] Universal speculative decoding for heterogeneous vocabularies (TLI) (#38174)
Signed-off-by: wan-danfeng <wandanfeng0802@gmail.com>
Signed-off-by: Wonderful <wandanfeng0802@gmail.com>
Co-authored-by: Wan_DF <wonderful199082@126.com>
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com>
2026-07-02 01:34:20 -07:00
peizhang56andClaude 7fe7fa9cda [CI][Bugfix] Rerun test_engine_log_metrics_ray on Ray GCS startup timeout (#47208)
Signed-off-by: pei.zhang <pei.zhang@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-07-01 21:32:09 -05:00
Yiliu DongandNick Hill 00eb7cefa3 [Bugfix] Prevent padding placeholders from reaching embeddings (#47029)
Signed-off-by: qianlihuang <91178480+qianlihuang@users.noreply.github.com>
Signed-off-by: Yiliu Dong <91178480+qianlihuang@users.noreply.github.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-07-01 09:26:03 -07:00
f5a8d73377 [Spec Decode] DSpark (#46995)
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: mgoin <mgoin64@gmail.com>
2026-07-01 08:30:24 -07:00
Nick Hill f1cf6b0086 [CI] Fix segfault in tracing test (#47299)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-07-01 14:00:37 +00:00
Harry Mellor e7d0fcbc09 [CI] Fix various failures on main (#47197)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-07-01 10:35:34 +01:00
Aaron HaoandClaude Sonnet 4.6 77a9c5ae28 Weight sync refactor + move sparse nccl engine (#44353)
Signed-off-by: hao-aaron <ahao@anyscale.com>
Signed-off-by: haoaaron <ahao@anyscale.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-01 01:25:19 -07:00
Nils Matteson a461070d1c [Core] Make sleep-mode backend capability flags communicator-agnostic (#47243) 2026-07-01 07:17:44 +00:00
Nils Matteson 93d8f834dd [Core] Pluggable sleep-mode backend abstraction (RFC #34303) (#44074) 2026-06-30 22:00:53 -07:00
c5200d3565 [Attention][DSA] support dcp for FLASHINFER_MLA_SPARSE (#46076)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: Jingyi Yang <girasoleyang@gmail.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: GirasoleY <girasoleyang@gmail.com>
Co-authored-by: Jingyi Yang <girasoleyang@gmail.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-01 00:32:20 -04:00
Ting SUN ac521f6237 [Bugfix][Structured Outputs] Reject degenerate structured_outputs that crash EngineCore (#45346)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
2026-06-30 22:41:33 +00:00
Andreas Karatzas c8f9c156a5 [ROCm][V1][MLA] Clone prefill backend state per metadata builder (#46993)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
2026-06-30 11:43:54 -07:00
Rishi PuriAnshika OjhaClaudegemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Stefano Castagnetta
727971f1c1 Add Medusa speculative decoding e2e test (#41396)
Signed-off-by: Anshika Ojha <anshikao@nvidia.com>
Signed-off-by: Rishi Puri <riship@nvidia.com>
Signed-off-by: Rishi Puri <puririshi98@berkeley.edu>
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: Anshika Ojha <215760622+ojhaanshika@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Stefano Castagnetta <scastagnetta@nvidia.com>
2026-06-30 18:02:22 +00:00
liuzhenwei 7a341fa109 [XPU] Support ZE_AFFINITY_MASK passthrough in xpu_disagg_acc_test (#47105)
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
2026-06-30 17:06:12 +00:00
Kunshang JiandHarry Mellor e840f0d3f5 [Platform] Replace torch.cuda.Event with torch.Event (#47140)
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-06-30 08:39:59 -07:00