Commit Graph
2218 Commits
Author SHA1 Message Date
czhu-cohere 3918f3c5a3 [Profiler] Unify platform-aware torch profiling (#57460) 2026-09-20 19:17:23 -07:00
Yifan QiaoandClaude Fable 5.1 d2983f2f16 [Bugfix][Spec Decode] Stop dummy draft decode steps from writing KV through stale block-table rows (#56734)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 01:06:25 +00:00
YukioZzz 9679173788 [KVConnector][MoRIIO] Transfer hybrid mamba/KDA recurrent state in READ mode (#51052)
Signed-off-by: Yichao Zhu <Yichao.Zhu@amd.com>
2026-09-20 17:30:34 -04:00
roikoren755andmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 49ee12d742 [Misc] Rename --enable-mamba-fine-grained-prefix-cache (#53945 follow-up) (#57382)
Signed-off-by: Roi Koren <roik@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-20 09:50:34 -07:00
f648eed23d [Bugfix][KV Offload] Skip non-prefix-cacheable groups in SimpleCPUOffload (GLM-5.3-Flash kpool tail and QSA) (#56810)
Signed-off-by: Jared Wen <jaredwen@inferact.ai>
Signed-off-by: Jared Wen <w13431838023@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
2026-09-20 10:58:55 +00:00
Jared WenClaude Fable 5.1mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
1b9fa3eaa8 [Perf][GLM] Fuse the kpool tail slot mapping into one Triton kernel (#57534)
Signed-off-by: Jared Wen <jaredwen@inferact.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-20 18:20:55 +08:00
Misha GoinandErinYin e5fce7b56b [Core][Kernel] Share persistent workspaces for Marlin and Humming (#57421)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: ErinYin <liang.yin@daocloud.io>
2026-09-20 17:28:33 +08:00
70fc7bd695 [Feature][EPD] Support dynamic EPD (EC-connector) proxy (#54176)
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-09-20 12:24:41 +08:00
Fanli Lin 708c2495c0 [Test][Core] Compute the expected hybrid prefix-cache hit instead of hard-coding it (#57467)
Signed-off-by: Lin, Fanli <fanli.lin@intel.com>
2026-09-20 10:16:12 +08:00
5302d1fe40 [Model Runner V2] Support custom logits processors (#56497)
Signed-off-by: David Cheung <d7cheung@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-19 17:46:49 +00:00
Aditya Ganesh Kumar a5a30471ff [Tests] Cover get_unhashed_block_ids_all_groups (#55928)
Signed-off-by: Aditya Ganesh Kumar <adty910@gmail.com>
2026-09-19 09:13:54 +00:00
038cb14bd7 [CI][Bugfix] Correct the Laguna DFlash acceptance-length reference (#57647)
Signed-off-by: Oxana Korzh <okorzh@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-19 01:44:48 +00:00
90531daf6d [Core][Frontend] Bind KV-event publishers at port 0 and expose the bound endpoints (#55844)
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: mukeshbhandarkar <shell00035@gmail.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-09-18 18:11:05 -07:00
waizuichougou a1bf8ac12d [Bugfix][MRV2] Match fast-prefill padding to active LoRA batches (#56456)
Signed-off-by: waizuichougou <2082431897@qq.com>
Signed-off-by: waizuichougou <105572299+waizuichougou@users.noreply.github.com>
2026-09-18 22:13:23 +00:00
1dc2d854c1 [Feat][Model] Support encoder-side SWA-bounded replay for DeepSeek-V4.1-Flash (#56227)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
2026-09-18 15:01:24 -07:00
ys2025-AI 6aab78f477 [Bugfix][KVConnector] Make ExampleHiddenStatesConnector abort-safe (#56841)
Signed-off-by: ys2025-AI <yuanshuaien@163.com>
2026-09-18 21:53:02 +00:00
62af3df734 [Bugfix][MLA] Reserve sparse prefill buffers before KV cache sizing (#57575)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
2026-09-18 21:22:18 +00:00
Nicolò LucchesiandOpenAI Codex 49e0427ca5 [Bugfix][NIXL] Avoid receive reports for notification-only requests (#57570)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-18 13:33:02 -07:00
bnellnmandItay Etelis 23e26e0588 [KV Offloading] Back-pressure detection and remediation (#50045)
Signed-off-by: Bill Nell <bnell@redhat.com>
Signed-off-by: bnellnm <49004751+bnellnm@users.noreply.github.com>
Co-authored-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
2026-09-18 14:47:48 -04:00
Nicolò Lucchesi 164dca8d1e [Nixl] Separate transport-failure metrics from KV expiry (#55854)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-09-18 18:08:59 +02:00
Nicolò Lucchesi a7156060c1 [Core] Make parallel sampling (n>1) reqs admission atomic (#53936)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-09-18 18:08:44 +02:00
Simon Danielsson 70df48dc3d [Bugfix][KV Cache][GLM-5.3-Flash] Disable slot mapping kernel for the kpool tail buffer (#57317)
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
2026-09-17 22:42:54 -04:00
Ren Yuzhou 7b94293627 [Bugfix][ROCm][KV Offload] Use private pinned tensors for CPU KV offload (#57160)
Signed-off-by: Yuzhou Ren <ryzcqu@outlook.com>
2026-09-18 02:35:23 +00:00
092bdd6d57 [Frontend][Core] Add release_kv_cache_memory() API (#44890)
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: AlanFokCo <alanfok2868@gmail.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com>
Co-authored-by: AlanFokCo <alanfok2868@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-09-18 07:12:12 +08:00
Wei Zhao ccde9c61e9 [Bugfix] Restore KV cache metadata GET method for external event consumer (#56925)
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
2026-09-17 23:01:35 +00:00
Woosuk KwonandCodex 80447d2765 [Bugfix][DSv4.1] Fix FlashInfer DSpark non-causal attention (#57432)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
2026-09-17 15:14:37 -07:00
Misha Goin db7a24c230 [Perf] Add fused DFlash2 grouped convolution (#55960)
Signed-off-by: mgoin <mgoin64@gmail.com>
2026-09-17 21:38:59 +00:00
Doug Brown 2b02c6c29b [Model] Add Cohere2MoE Eagle3 auxiliary hidden states (#49819)
Signed-off-by: Doug Brown <doug@github.douggo.com>
2026-09-17 20:09:36 +00:00
Elvir CrnčevićOpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Tyler Michael Smith
ac2f0ea82c [MoE][Bugfix] Skip SP padded rows in grouped MoE routing (#56079)
Signed-off-by: Elvir Crncevic <elvircrn@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
2026-09-17 19:36:19 +00:00
67e5b0acc9 [Bugfix][HiSparse][NIXL] Import full blocks without tail prefill on D (#57049)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
2026-09-17 15:26:10 -04:00
Michele Campi 6a344d8c85 [Bugfix][Metrics] Do not log a 0.0% prefix cache hit rate before any query (#54990)
Signed-off-by: Michele Campi <215741962+MicheleCampi@users.noreply.github.com>
2026-09-17 18:19:13 +00:00
Tyler Michael SmithOpenAI CodexOpenAI CodexMisha Goinmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
4f9a41ac94 [CI/Build][Hardware][NVIDIA] Add Rubin CUDA 13.4 nightly images (#55953)
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-17 08:05:31 -07:00
Alexander Leeaoshen02Claude Opus 5aoshen02mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
9e1cd262cb [Bugfix] Use DP index for dense DP weight updates and EC CPU region (#56950)
Signed-off-by: lxy-alexander <lxy_alexander@outlook.com>
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-17 22:44:36 +08:00
Robert ShawandClaude Opus 5 861a299adc [Bugfix][Model Runner V2] Route dummy tokens to MoE experts during profiling (#57270)
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:11:14 -04:00
CaiJohnandjohn 75c71390d5 [Bugfix] Escape control characters in xgrammar choice grammar (#48115)
Signed-off-by: john <6712432+CaiJohn@users.noreply.github.com>
Co-authored-by: john <6712432+CaiJohn@users.noreply.github.com>
2026-09-17 15:57:47 +03:00
Martín el CheikhandClaude 2e50824766 [Bugfix] Fall back to native sampling when FlashInfer cannot target the GPU (#48956)
Signed-off-by: martin el cheikh <martinelcheikh@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-17 12:27:54 +00:00
aoshen02andaoshen02 9ff08f9493 [Bugfix] Honor skip_reading_prefix_cache for KV connector hits (#57269)
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
2026-09-17 03:46:28 -07:00
0d72627565 [Perf][EPD] Batch image requests per encoder (#57095)
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Co-authored-by: Zhou ziheng <jiaranran2@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-17 03:45:52 -07:00
Itay EtelisandOr Ozeri b6e7c1f1f0 [kv_offload] Skip scratch groups (#57145)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-09-17 13:22:49 +03:00
40b40d1d39 [Bugfix][KV Offload] Register the offload region in chunks (#51081)
Signed-off-by: Mikhail Kostryukov <mike@triptrack.net>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Colton Ottley <coltonottley@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-09-17 13:19:56 +03:00
jiangkuaixue123andCodex de24e51908 [MM][V2] Enable encoder-only ViT CUDA graph capture (#56922)
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-17 16:12:17 +08:00
2743dc4f92 [Feature][Multimodal] Add cross-encoder caching to Mooncake P2P (#56242)
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Co-authored-by: Tianyu Guo <guoty@inferact.ai>
Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-17 13:16:41 +08:00
MattandClaude Code 68f2c7e74f [ROCm][CI] Fix Nixl+Offloading PD edge cases on TheRock image (#57249)
Signed-off-by: Matthew Wong <Matthew.Wong2@amd.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-09-16 22:56:13 -05:00
Itay EtelisandOr Ozeri 9854b580df [Bugfix][KV Offload] Compact canonical MLA rows (#56799)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-09-17 06:24:06 +03:00
omerpaz95 fa3622a4a1 [EC Connector] Add Metrics Collection (#54960)
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
2026-09-17 10:36:23 +08:00
Oxana KorzhandAndreas Karatzas 2307a1d16c [Core] Don't issue a blocking collective RPC during engine handshake (#57226)
Signed-off-by: okorzh <okorzh@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-16 21:30:59 -05:00
Chuan (Richard) LiandCursor f730a93d2b [Scheduler] Add --max-num-active-seqs to cap RUNNING admission (#56758)
Signed-off-by: Li <chuali@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-17 10:22:12 +08:00
Jiangyun ZhuandCodex b1f9ebc0c3 [Test][Core] Add hybrid model prefix cache hit-rate coverage (#57127)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-16 19:17:45 -07:00
dongluwandAndreas Karatzas 19b6ff62f2 [ROCm][Bugfix] Ignore descales for unquantized AITER caches (#56726)
Signed-off-by: donglu <donglu@cohere.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-17 01:29:50 +00:00
Thillai Chithambaram a615f53364 [Bugfix][Metrics][MFU] Size activation traffic from the model dtype (#57070)
Signed-off-by: Thillai Chithambaram <thillaichithambaram.a@gmail.com>
2026-09-17 00:23:28 +00:00