Commit Graph
1694 Commits
Author SHA1 Message Date
Nick HillandAndreas Karatzas 6453fc0b8c [Bugfix] Don't reuse engine core payload buffer while zmq is sending it (#50053)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-07-28 16:09:28 +00:00
30217b0e80 [Bugfix][KV Offload][P2P] Scope serve state to fetch rounds (#49877)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <Itay.etelis@gmail.com>
2026-07-28 18:01:55 +03:00
Nick Hill 0d0504b54c [Core] Warm up runner-owned Triton kernels before the first request (#49903) 2026-07-28 07:58:02 -07:00
Jonguk CheongandOpenAI Codex 1e81853afc [Bugfix][KV Offload] Keep Mamba block span unscaled under DCP (#49964)
Signed-off-by: Jonguk Cheong <jdal3031@snu.ac.kr>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-07-28 17:41:55 +03:00
b6cbba8bc8 [Bugfix][Kernel] Fix batch invariance in RMSNorm kernels by pinning block size (#48391)
Signed-off-by: oops-oom <73481342@qq.com>
Signed-off-by: oops-oom <liubin8905@vip.qq.com>
Co-authored-by: oops-oom <73481342@qq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
2026-07-28 22:24:02 +08:00
Nick HillandClaude Opus 4.8 94100b5915 [CI] Wire untethered test files into CI jobs (#49340)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-28 14:02:31 +00:00
Nick HillandClaude Fable 5 601fa9a74e [KV Connector] Support NIXL heterogeneous P/D block sizes for hybrid models (#49612)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 06:45:02 -07:00
Liangliang Maandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 88402a41c4 [Test] Skip ROCm AITER MLA prefill tests on non-ROCm platforms (#49945)
Signed-off-by: Liangliang-Ma <liangliang.ma@intel.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-28 16:26:49 +08:00
MINJUN GILandOpenAI Codex 52c3c4a42f [Bugfix][KV Offload][OBJ] Preserve job completion during cleanup (#49947)
Signed-off-by: MINJUN GIL <alswnsrlf12@naver.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-07-28 07:10:52 +03:00
a8f296083f [KV Offload] Make compact secondary identity TP-independent (#49858)
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: GPT-5.6 Sol <noreply@openai.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-28 07:10:20 +03:00
Varun Sundar RabindranathandVarun Sundar Rabindranath d18ed2304a [KV-offload][FS] : Batch store/load_block in C (#49152)
Signed-off-by: <>
Co-authored-by: Varun Sundar Rabindranath <varun-sundar-rabindranath@h100-01.nemg-001.lab.rdu2.dc.redhat.com>
2026-07-28 11:21:53 +08:00
Giancarlo Delfin 272abd5f48 [Tests][Spec Decode] Add gemma4 MTP acceptance rates test (#47920)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
2026-07-27 16:15:54 -07:00
Rishi PuriandClaude Opus 4.8 fd10e8946d [Test] Regression test for hybrid-Mamba eagle cache-peek in Mooncake connector (#43559) (#48361)
Signed-off-by: Rishi Puri <riship@nvidia.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-27 19:03:58 +00:00
Matthew Bonanniandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> bf2b45b5d6 [Attention] Integrate FlashAttention 4 SM100 headdim 256 support (#42669)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-07-27 18:25:50 +00:00
Connor CarpenterandNick Hill e3c2fc3b3c [Rust Frontend][gRPC] Add server and model discovery (#49491)
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-07-27 09:53:27 -07:00
Nicolò Lucchesi 2b465b2c42 [Misc][PD] Nixl cleanup get_backend_aware_kv_block_len and virtually_split_kv_in_blocks (#49988)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-07-27 18:51:37 +02:00
Nick Hillandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 59a6b0411d [Core] Fix internal LB load-balancing (#49204)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-27 14:00:46 +00:00
Ronen Schaffer 77cba0259f [KV Offloading] Per-request tier filtering with TierFilter/TierMatcher (#48123)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-07-27 13:29:57 +03:00
Andreas KaratzasandOpenAI Codex 394beb633b [Bugfix][ROCm] Use batch DMA for CPU KV cache loads (#49843)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-07-27 02:18:01 -07:00
liuzhenwei afc94523c9 [XPU][CI] Use platform device in InputBatch V2 test (#49939)
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
2026-07-27 15:28:02 +08:00
Xiaochang WuandCopilot 8061dc26bd [Bugfix] Normalize sparse MLA warmup compression ratios (#49392)
Signed-off-by: Wu, Xiaochang <xiaochang.wu@intel.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-07-27 07:02:48 +00:00
Dao007foreverandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> d742856610 [3/N][Core][KV Connector] Support reliable partial-tail KV offload for sub-block prompts (#49502)
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-26 23:22:17 -07:00
Andreas Karatzas 5f89a03dcb [CI] Explicitly tear down speculative decode runners (#49910)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
2026-07-27 05:32:46 +00:00
JasonandOr Ozeri 53397fbfac [Bugfix][KV Offload][P2P] Fix EngineCore crash reconnecting to a reaped peer (#49823)
Signed-off-by: Jason Yao <wsyjh8@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-27 08:07:00 +03:00
Walter Beller-Moralesandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 8040ef2426 [Frontend] expose stream_interval as req sampling param (#49754)
Signed-off-by: walterbm <walter.beller.morales@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-27 11:30:37 +08:00
Andreas Karatzas ac87549cbd [CI][ROCm] Reduce V1 attention test runtime (#49916)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
2026-07-27 11:12:45 +08:00
limeward ffc4f08c8e [Core][KV-transfer] MoRIIO: heterogeneous TP<->DP prefill/decode read routing (#46116)
Signed-off-by: Edwin Lim <edwin.lim@mangoboost.io>
2026-07-27 02:10:00 +00:00
Nick Hill 50aa830482 [BugFix][MRV2] Don't create dummy requests longer than max_model_len (#49751)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-07-27 02:04:20 +00:00
da3a252fd1 [KVOffload][P2P] Generic P2P secondary tier: peer lookup and serving via ParentManager (#48021)
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: liranschour <liranschour@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-26 11:45:47 +03:00
Chang GuoandOpenAI Codex 7eca0e1a64 [KV Offload] Deduplicate replicated MLA KV in the shared CPU region (#48906)
Signed-off-by: Change72 <changg@nvidia.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-07-26 08:22:30 +03:00
Jonguk CheongandOr Ozeri 7a29a3c54c [Bugfix][KV Offload] Namespace persistent cache by model runner (#49440)
Signed-off-by: Jonguk Cheong <jdal3031@snu.ac.kr>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-26 08:21:52 +03:00
Rui YinandOr Ozeri d30b1ecd1b [Bugfix][KV Offloading] Defer request finalization until final store (#49671)
Signed-off-by: Rui Yin <2260891073@qq.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-07-25 20:58:09 +00:00
0b0bd2b5f6 [Feature] Add fault tolerance framework (simplified) for DP+EP external LB deployments (#44428)
Signed-off-by: fangyuchu <fangyuchu@qq.com>
Signed-off-by: a798347923 <2645302020@qq.com>
Signed-off-by: TianZhuo <2770730562@qq.com>
Signed-off-by: a798347923 <39047817+a798347923@users.noreply.github.com>
Signed-off-by: 205150940 <112750056+205150940@users.noreply.github.com>
Signed-off-by: w00689259 <wangzhuo66@huawei.com>
Signed-off-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com>
Signed-off-by: zWaNg3 <389750525@qq.com>
Signed-off-by: yzchang-plus <1078477584@qq.com>
Signed-off-by: Jade Zheng <zheng.shoujian@outlook.com>
Co-authored-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com>
Co-authored-by: a798347923 <2645302020@qq.com>
Co-authored-by: TianZhuo <2770730562@qq.com>
Co-authored-by: 205150940 <112750056+205150940@users.noreply.github.com>
Co-authored-by: a798347923 <39047817+a798347923@users.noreply.github.com>
Co-authored-by: w00689259 <wangzhuo66@huawei.com>
Co-authored-by: zWaNg3 <389750525@qq.com>
Co-authored-by: yzchang-plus <1078477584@qq.com>
Co-authored-by: Jade Zheng <zheng.shoujian@outlook.com>
2026-07-25 11:49:09 -04:00
Harry MellorandClaude Opus 5 0b1a8bb1f6 [Bugfix][CI] Fix stale Mooncake lookup expectation broken by a merge race (#49802)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 03:10:26 -07:00
0ba2aa35a8 Stabilize GPU memory teardown between ROCm CI tests (#49242)
Signed-off-by: aarushjain29 <Aarushi.Jain2@amd.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-07-25 04:21:36 +00:00
Yifan QiaoandClaude 70052fb924 [Bugfix][KV Connector][Mooncake] Keep TP-sharded Mamba state out of the KV-head dedup (#49499)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude <noreply@anthropic.com>
2026-07-24 21:02:22 -07:00
liuzhenweiandKunshang Ji 6a1acac3fe [BUGFIX] Fix log capture in KV test (#49655)
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-07-25 00:36:53 +00:00
Harry Mellor 89f6aa3a9e [KV Offload][CI] Fall back to buffered I/O without O_DIRECT; fix flaky api-server test (#49734)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-07-24 13:56:57 -07:00
Elvir CrnčevićandClaude Opus 4.6 972848f276 [Bugfix] Support non-uniform page sizes in KVBlockZeroer (#49704)
Signed-off-by: Elvir Crncevic <elvircrn@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-24 13:38:59 -07:00
Johnny-LiouClaude Opus 4.8mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>tomeras91Cyrus Leung
866fea2b99 [Kernel] ReplaySSM: cache SSM inputs for faster Mamba2 standard decode (#48018)
Signed-off-by: Johnny-Liou <a897111@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: tomeras91 <57313761+tomeras91@users.noreply.github.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-07-24 09:39:49 -07:00
Taneem Ibrahimandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 7b40fb9645 [UX] Reject incompatible nested runtime overrides (#49247)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-24 07:16:22 -07:00
833483f357 Encoder cache extension hooks (#48218)
Signed-off-by: hotTea <958436561@qq.com>
Signed-off-by: hanxi-java <634498162@qq.com>
Co-authored-by: hanxi-java <634498162@qq.com>
Co-authored-by: 韩熙 <63780107+hanxi-java@users.noreply.github.com>
2026-07-24 02:52:00 -07:00
Nicolò Lucchesiandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 589a5b884b [PD][NixlPush][Bugfix] Fix blocking handshake call on writer thread (#49221)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-24 01:41:37 -07:00
Michael GoinandOpenAI Codex 275556c35c [Bugfix] Detect mixed precision in packed KV cache specs (#49623)
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-07-23 20:07:06 -07:00
Nicolò Lucchesiandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 2659467497 [CI][PD] Add hybrid SSM P_TP>D_TP accuracy sweep entry (#49593)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-23 22:49:53 +01:00
Michael GoinandOpenAI Codex e18f0037a5 [Bugfix][KV cache] Support sparse-MLA targets with SWA drafts (#48776)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-07-23 11:30:50 -07:00
Wentao Ye b354734d17 [Bug] Fix batch invariance rms norm comparison (#49603)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-07-23 12:27:38 -06:00
75ccdf3145 [Core] Update PyTorch to 2.13.0, torchvision to 0.28.0, triton to 3.7.1 (#48155)
Signed-off-by: Andrey Talman <atalman@users.noreply.github.com>
Co-authored-by: Andrey Talman <atalman@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-07-23 11:09:36 -07:00
Andreas Karatzas f00efc5265 [CI] Isolate cudagraph tests in child processes (#49510)
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
2026-07-23 17:16:19 +00:00
yue.yu 0416dab275 [Bugfix][Structured Output][Spec Decode] Advance grammar across reasoning boundary (#44993)
Signed-off-by: Allen.Yu <yuyue0225sc@163.com>
2026-07-23 09:14:15 -07:00