df6c7ad031
[Core][Refactor] Move KV cache planning logic to kv_cache_planning.py ( #61183 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-13 03:05:39 +00:00
Tianyu Guo
258b97ff6b
[Bugfix][EC Connector] Size transfers by measured encoder output widths ( #57904 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
2026-10-12 12:32:39 +00:00
Shuolei Wang and aoshen02
b89852aad0
[RL][Frontend] Return token-level weight-version spans on final outputs ( #59582 )
...
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com >
Signed-off-by: Shuolei Wang <948904026@qq.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-11 11:33:29 +08:00
f8e6e365ae
[Bugfix][Mooncake] Save the EAGLE attention proof with hybrid Mamba partial tails ( #60695 )
...
Signed-off-by: Dao Le <Dao007forever@gmail.com >
Signed-off-by: Dao007forever <daole@inferact.ai >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
2026-10-10 00:08:08 +00:00
02e55d2af3
[KV Cache] GLM-5.3-Flash: use the generic packed KV layout; CircularBufferSpec tail with speculative ring slots ( #57169 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-10-09 15:48:37 -04:00
Ning Xie and Simon Mo
c4a5ff5c18
[Frontend] shutdown engine core and its sub processes asap when initializing ( #52299 )
...
Signed-off-by: Andy Xie <andy.xning@gmail.com >
Co-authored-by: Simon Mo <simon.mo@hey.com >
2026-10-08 11:46:01 -07:00
d8fa463e2b
[Bugfix][Prefix caching] Fix inconsistent hybrid Mamba prefix cache boundaries ( #60533 )
...
Signed-off-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-08 18:33:48 +00:00
czhu-cohere and aoshen02
6fddc9b21e
[Profiler] Add per-session profiling controls to Python and Rust frontends ( #57875 )
...
Signed-off-by: czhu <conway.zhu@cohere.com >
Signed-off-by: Conway Zhu <conway.zhu@cohere.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-06 05:21:19 +00:00
42d9f8ccdb
support rl feature : weight checker ( #51350 )
...
Signed-off-by: shiyuan680 <yangcheng104@huawei.com >
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: shiyuan680 <yangcheng104@huawei.com >
Co-authored-by: Ronald1995 <ronaldautomobile@163.com >
Co-authored-by: DeepSeek Harness <noreply@deepseek.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-30 03:13:43 +00:00
Tianyu Guo
53d2e16a97
[Bugfix][EPD] Skip sampling for encoder-only async steps ( #58490 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
2026-09-27 20:35:09 -07:00
Mark McLoughlin and AI Assistant
45d56c0baf
[Core] Model console logging as CLI configuration ( #57205 )
...
Add `--logging-config` CLI argument which can be supplied as
JSON or using dotted arguments. The `--log-level` argument
is provided for convenience, and `--log-config-file` is deprecated
in favor of `--logging-config.pylogging_config_file`.
Signed-off-by: Mark McLoughlin <markmc@redhat.com >
Co-authored-by: AI Assistant <noreply@openai.com >
2026-09-25 11:36:59 +01:00
Cyrus Leung
c8824a1e42
[MM] Move supports_multimodal_inputs and cache out of registry ( #57913 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-09-21 07:43:07 -07:00
1185254851
[Bugfix][KV Connector] Propagate cache reset failure during sleep ( #54581 )
...
Signed-off-by: CrazyCoder <18235787078@163.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
2026-09-21 09:25:02 +08:00
092bdd6d57
[Frontend][Core] Add release_kv_cache_memory() API ( #44890 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: AlanFokCo <alanfok2868@gmail.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com >
Co-authored-by: AlanFokCo <alanfok2868@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-09-18 07:12:12 +08:00
Wei Zhao
ccde9c61e9
[Bugfix] Restore KV cache metadata GET method for external event consumer ( #56925 )
...
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-09-17 23:01:35 +00:00
Oxana Korzh and Andreas Karatzas
2307a1d16c
[Core] Don't issue a blocking collective RPC during engine handshake ( #57226 )
...
Signed-off-by: okorzh <okorzh@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-16 21:30:59 -05:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
f37c550bf6
[Feature][Frontend] Expose effective attention block size for DCP ( #56538 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Signed-off-by: Julien Darve <jdarve@NVIDIA.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-09-16 08:29:43 +00:00
Chaojun Zhang and Kunshang Ji
622934e61f
[XPU] Honor device_ids for worker device placement ( #56015 )
...
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-09-16 11:53:23 +08:00
Wentao Ye
6b15bea080
[Refactor] Remove utils dead code ( #53941 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-08 11:22:09 -04:00
aoshen02
039ea82660
[Core] Sync DP state on the first step of a wave ( #52957 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
2026-09-06 02:16:58 +00:00
Jiangyun Zhu
98ed0856f3
[Model] add GLM-5.3-Flash support ( #53906 )
2026-09-03 09:40:35 -07:00
aoshen02
17da48596c
[Bugfix][DP] Synchronize the device on pause completion ( #52914 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
2026-08-26 09:50:33 -07:00
Ning Xie
23310bc3de
[sleep functionality] code refactor about sleep/wake_up ( #50431 )
...
Signed-off-by: Andy Xie <andy.xning@gmail.com >
2026-08-25 07:45:20 +00:00
Lucas Wilkinson
8bdc70ec7b
[6/N][KV-Cache Layout Refactor] Standardize KV cache layout ( #51718 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
2026-08-21 17:47:49 -07:00
lxy and “新颐”
0c8c3f41cf
[Bugfix][V1] Sync mamba_block_size via EngineCoreReadyResponse ( #50809 )
...
Signed-off-by: “新颐” <lxy537258@antgroup.com >
Co-authored-by: “新颐” <lxy537258@antgroup.com >
2026-08-19 16:44:36 +00:00
3a386cfaf5
[ROCm] Give EngineCore cleanup grace after request abort ( #52281 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-19 11:39:59 -05:00
aoshen02 and Claude Opus 5
7ddb50788d
[Bugfix][DP] Don't assume the engines started when forwarding a wake ( #51481 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-18 13:49:35 -07:00
Connor Carpenter and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f08a95f8d8
[Rust Frontend][gRPC] Advertise LoRA capabilities ( #52031 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-17 13:27:56 -07:00
fa9d67f782
[EC Connector] Added Build Connector Worker Meta for EC Connector ( #49585 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Signed-off-by: omerpaz95 <omerpaz95@gmail.com >
Signed-off-by: omerpaz95 <73347585+omerpaz95@users.noreply.github.com >
Co-authored-by: yewentao256 <zhyanwentao@126.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-08-15 15:40:38 -07:00
Connor Carpenter and Nick Hill
f473870ecf
[Rust Frontend][gRPC] Add RL lifecycle control ( #51316 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-14 11:32:39 -07:00
3962042304
[Bugfix][V1][Multimodal] Recover from P0/P1 processor cache drift ( #46747 ) ( #46747 )
...
Signed-off-by: William Zhao <williamzh@meta.com >
Signed-off-by: WillZZZy <93454470+WillZZZy@users.noreply.github.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
Co-authored-by: Claude <noreply@anthropic.com >
2026-08-12 12:06:57 +08:00
aoshen02 and Andy Lo
8cfa01cdd2
[BugFix] Dense multinode DP rescope with regression test ( #49212 )
...
Signed-off-by: Andy Lo <andy@mistral.ai >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Andy Lo <andy@mistral.ai >
2026-08-05 18:12:03 -07:00
5f2ee2fa8c
[Bugfix][Core] Log KV cache capacity after block-size resolution ( #50462 )
...
Signed-off-by: Tanmay Dixit <22117787+tandixit95@users.noreply.github.com >
Signed-off-by: Michael Goin <mgoin64@gmail.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Tanmay Dixit <22117787+tandixit95@users.noreply.github.com >
Co-authored-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-04 12:53:32 +00:00
Connor Carpenter and Nick Hill
48fc2e2797
feat(grpc): add KV event source discovery ( #50033 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-29 13:46:40 -07:00
Nick Hill and Andreas Karatzas
6453fc0b8c
[Bugfix] Don't reuse engine core payload buffer while zmq is sending it ( #50053 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-07-28 16:09:28 +00:00
Itay Alroy
35efdf6b34
[Elastic EP] Async preparation ( #47288 )
...
Signed-off-by: Itay Alroy <ialroy@nvidia.com >
2026-07-28 05:19:13 -07:00
Shuolei Wang
9069a57139
[Core][Frontend] Add weight version tagging for RL rollouts ( #49040 )
...
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com >
Signed-off-by: Shuolei Wang <948904026@qq.com >
2026-07-28 14:35:29 +08:00
Connor Carpenter and Nick Hill
e3c2fc3b3c
[Rust Frontend][gRPC] Add server and model discovery ( #49491 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-27 09:53:27 -07:00
Nick Hill and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
59a6b0411d
[Core] Fix internal LB load-balancing ( #49204 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-27 14:00:46 +00:00
0b0bd2b5f6
[Feature] Add fault tolerance framework (simplified) for DP+EP external LB deployments ( #44428 )
...
Signed-off-by: fangyuchu <fangyuchu@qq.com >
Signed-off-by: a798347923 <2645302020@qq.com >
Signed-off-by: TianZhuo <2770730562@qq.com >
Signed-off-by: a798347923 <39047817+a798347923@users.noreply.github.com >
Signed-off-by: 205150940 <112750056+205150940@users.noreply.github.com >
Signed-off-by: w00689259 <wangzhuo66@huawei.com >
Signed-off-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com >
Signed-off-by: zWaNg3 <389750525@qq.com >
Signed-off-by: yzchang-plus <1078477584@qq.com >
Signed-off-by: Jade Zheng <zheng.shoujian@outlook.com >
Co-authored-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com >
Co-authored-by: a798347923 <2645302020@qq.com >
Co-authored-by: TianZhuo <2770730562@qq.com >
Co-authored-by: 205150940 <112750056+205150940@users.noreply.github.com >
Co-authored-by: a798347923 <39047817+a798347923@users.noreply.github.com >
Co-authored-by: w00689259 <wangzhuo66@huawei.com >
Co-authored-by: zWaNg3 <389750525@qq.com >
Co-authored-by: yzchang-plus <1078477584@qq.com >
Co-authored-by: Jade Zheng <zheng.shoujian@outlook.com >
2026-07-25 11:49:09 -04:00
zhrrr and Nick Hill
12213c6795
[Bugfix] handle grammar compilation failures to avoid engine crash ( #47312 )
...
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-23 18:13:49 +08:00
5a65ba5f17
[Refactor] Move iteration logging to the frontend ( #46647 )
...
Signed-off-by: maxyanghu <hyoung2991@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Shang Wang <shangw@nvidia.com >
2026-07-15 17:59:05 -07:00
Taneem Ibrahim and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
9182e86971
Log fully resolved pooling config at startup ( #48030 )
...
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-14 22:00:46 +00:00
omerpaz95 and Or Ozeri
8df14cfc8c
[EC Connector] Add EC Transfer Params ( #42433 )
...
Signed-off-by: omerpaz95 <omerpaz95@gmail.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-07-12 14:35:33 +03:00
Robert Shaw
c5e3c40877
Fix P/D with DP Supervisor ( #46628 )
...
Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-06-25 13:13:08 -07:00
Wentao Ye
1cd3e0e945
[Bug] Fix IndentationError: expected an indented block after 'with' statement ( #46627 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-06-24 23:14:17 +08:00
93ec645878
[Bugfix] Fix illegal memory access from a forward during a partial wake_up ( #44483 )
...
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-06-24 22:12:23 +08:00
ebfbcfe46a
Stop setting CUDA_VISIBLE_DEVICES internally in vLLM, add device_ids arg ( #45026 )
...
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: kourosh hakhamaneshi <kouroshHakha@users.noreply.github.com >
2026-06-20 13:38:10 -07:00
