e319f86f15
[Feature] Per-row candidate IDs for prefill token scoring (M2 of #56860 ) ( #56984 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-03 14:17:09 +00:00
Benedikt Falk
c4973d94fe
[Bugfix][Frontend] Avoid generation for empty streaming input ( #59015 )
...
Signed-off-by: Benedikt Falk <bfalk@nvidia.com >
2026-10-02 18:48:43 +00:00
Taneem Ibrahim
f0c44cc7d5
[Bugfix] Fix stalled local-only Elastic EP scale-up ( #57820 )
...
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
2026-10-02 10:47:55 -04:00
592c6f3fbc
[Bugfix][Pooling] Handle multimodal cache misses without crashing ( #58975 )
...
Signed-off-by: linchaoying6 <linchaoying6@jd.com >
Co-authored-by: linchaoying6 <linchaoying6@jd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-10-02 10:14:33 -04:00
Martin Hickey
fcfacc3499
[Bugfix][Frontend] Seed derender detokenization from the prompt ( #59046 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-10-02 03:17:56 +00:00
Shijie Lyu
9aaef4296f
[Bugfix][Frontend] Check reused prompt token ids against the vocab before streaming ( #59555 )
...
Signed-off-by: Shijie Lyu <lshjhf@gmail.com >
2026-10-01 09:26:55 -07:00
4a9d6f9572
[Bugfix] Preserve sampling masks in DELTA and TITO streaming outputs ( #55935 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: fh3 <a@b>
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: fh3 <a@b>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-01 14:43:38 +08:00
70be6aae19
[Feature][Spec Decode] Support sampling mask replay for MRV2 MTP ( #59359 )
...
Signed-off-by: Cuiping Cheng <chengcuipingswu@163.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Cuiping Cheng <chengcuipingswu@163.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 13:51:39 +08:00
e5f3d08d72
[Frontend][RL] Align sleep-mode API responses and operation metrics ( #52864 )
...
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-29 23:16:38 -07:00
Andreas Karatzas
866fa130fa
[Core] Bound UniProc EngineCore startup threads to available CPUs ( #58946 )
...
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com >
2026-09-29 23:02:55 -07:00
Mia Ojeda
e006d761a5
[Bugfix][CPU][DiffusionGemma] Support narrower canvas w/ sync scheduling ( #59107 )
...
Signed-off-by: ojeda-e <estefania.b.ojeda@gmail.com >
2026-09-29 23:28:50 -04:00
42d9f8ccdb
support rl feature : weight checker ( #51350 )
...
Signed-off-by: shiyuan680 <yangcheng104@huawei.com >
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: shiyuan680 <yangcheng104@huawei.com >
Co-authored-by: Ronald1995 <ronaldautomobile@163.com >
Co-authored-by: DeepSeek Harness <noreply@deepseek.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-30 03:13:43 +00:00
a7a9ac2745
[Core] Combine per-engine utility results like the Rust client ( #59240 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-29 22:20:51 +00:00
Chanbin Lim
91de237333
[Bugfix][DP] add_dp_placement_groups does not require ray[default] ( #57648 )
...
Signed-off-by: beenpow <57864041+beenpow@users.noreply.github.com >
2026-09-29 18:15:13 +00:00
Almog Tavor
171d1b9ef6
[Elastic EP] Support Model Runner V2 ( #53934 )
...
Signed-off-by: almogtavor <almogtavor@gmail.com >
Signed-off-by: Almog Tavor <70065337+almogtavor@users.noreply.github.com >
2026-09-29 00:20:40 +00:00
Benedikt Falk
c68eb98b69
[Bugfix][Frontend] Keep in-flight requests on the same DP engine ( #59017 )
...
Signed-off-by: Benedikt Falk <bfalk@nvidia.com >
2026-09-28 23:52:29 +00:00
b721a4c709
[Feature] Add fixed-token prefill scoring ( #54335 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-09-28 07:50:09 -07:00
Tianyu Guo
53d2e16a97
[Bugfix][EPD] Skip sampling for encoder-only async steps ( #58490 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
2026-09-27 20:35:09 -07:00
Mark McLoughlin and AI Assistant
45d56c0baf
[Core] Model console logging as CLI configuration ( #57205 )
...
Add `--logging-config` CLI argument which can be supplied as
JSON or using dotted arguments. The `--log-level` argument
is provided for convenience, and `--log-config-file` is deprecated
in favor of `--logging-config.pylogging_config_file`.
Signed-off-by: Mark McLoughlin <markmc@redhat.com >
Co-authored-by: AI Assistant <noreply@openai.com >
2026-09-25 11:36:59 +01:00
Taneem Ibrahim
bbd7c24d6e
[MRV2] Validate MRV2 entrypoint logits processors ( #57728 )
...
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
2026-09-24 09:07:26 +00:00
c0c35ddd8e
[Bugfix][CI] Fix the flaky sharded-sampling tests, and the engine teardown need ( #58342 )
...
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com >
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-24 02:05:11 -07:00
1b3b88ec2b
[Core] structured generation mode for DiffusionGemma model (Jev-like) ( #57250 )
...
Signed-off-by: Matt Mastracci <matthew@mastracci.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Razorback16 <razorback16@protonmail.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Razorback16 <razorback16@protonmail.com >
2026-09-22 20:51:17 +00:00
shaohuaxi and Codex
5abf85d27a
[Bugfix][Frontend] Validate mixed prompt embedding mask lengths ( #57006 )
...
Signed-off-by: 子华 <huaxi.shx@alibaba-inc.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-22 05:50:31 +00:00
Juan Pérez de Algaba
4800024b98
[Security] Reject min_tokens that exceeds the filled max_tokens default ( #57731 )
...
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com >
2026-09-22 05:50:23 +00:00
Karen Chung
986e217158
[Feature] Add first-class KV hints request envelope for programmatic KV management ( #53423 )
...
Signed-off-by: Karen Chung <karenc@nvidia.com >
2026-09-21 09:56:11 -07:00
Cyrus Leung
c8824a1e42
[MM] Move supports_multimodal_inputs and cache out of registry ( #57913 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-09-21 07:43:07 -07:00
czhu-cohere
3918f3c5a3
[Profiler] Unify platform-aware torch profiling ( #57460 )
2026-09-20 19:17:23 -07:00
1185254851
[Bugfix][KV Connector] Propagate cache reset failure during sleep ( #54581 )
...
Signed-off-by: CrazyCoder <18235787078@163.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
2026-09-21 09:25:02 +08:00
5302d1fe40
[Model Runner V2] Support custom logits processors ( #56497 )
...
Signed-off-by: David Cheung <d7cheung@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-19 17:46:49 +00:00
90531daf6d
[Core][Frontend] Bind KV-event publishers at port 0 and expose the bound endpoints ( #55844 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: mukeshbhandarkar <shell00035@gmail.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-18 18:11:05 -07:00
Nicolò Lucchesi
a7156060c1
[Core] Make parallel sampling (n>1) reqs admission atomic ( #53936 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-09-18 18:08:44 +02:00
092bdd6d57
[Frontend][Core] Add release_kv_cache_memory() API ( #44890 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: AlanFokCo <alanfok2868@gmail.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com >
Co-authored-by: AlanFokCo <alanfok2868@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-09-18 07:12:12 +08:00
Wei Zhao
ccde9c61e9
[Bugfix] Restore KV cache metadata GET method for external event consumer ( #56925 )
...
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-09-17 23:01:35 +00:00
Oxana Korzh and Andreas Karatzas
2307a1d16c
[Core] Don't issue a blocking collective RPC during engine handshake ( #57226 )
...
Signed-off-by: okorzh <okorzh@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-16 21:30:59 -05:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
Robert Shaw and Claude
42919b49c5
[P/D] Report prefill worker cache hits in prompt_tokens_details ( #54222 )
...
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-16 15:23:31 +00:00
f37c550bf6
[Feature][Frontend] Expose effective attention block size for DCP ( #56538 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Signed-off-by: Julien Darve <jdarve@NVIDIA.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-09-16 08:29:43 +00:00
Chaojun Zhang and Kunshang Ji
622934e61f
[XPU] Honor device_ids for worker device placement ( #56015 )
...
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-09-16 11:53:23 +08:00
Chauncey
f0fbc98c48
[Frontend] Share max_num_queued_reqs across API server processes ( #54746 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
2026-09-15 14:50:26 +00:00
84e5f3edac
[Bugfix] Validate routed-expert prompt offsets before engine submission ( #56844 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Bhushan Asati <bhushanasati25@gmail.com >
Co-authored-by: Megha Agarwal <19240983+meghaagr13@users.noreply.github.com >
2026-09-15 09:17:41 +08:00
Itay Alroy
658c8131c7
[Elastic EP] Reuse CUDA graphs across reconfiguration ( #54985 )
2026-09-12 10:26:39 -04:00
djramic and Andreas Karatzas
c6ba5e89d6
[Bugfix][Elastic EP] Route only to surviving engines during scale-down ( #55772 )
...
Signed-off-by: Djordje Ramic <djoramic@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-08 21:28:28 -05:00
Wentao Ye
6b15bea080
[Refactor] Remove utils dead code ( #53941 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-08 11:22:09 -04:00
Wei Zhao
c3ec0d29f5
[Bugfix] Fix cuda profiler missing bug ( #55237 )
2026-09-07 03:15:57 +00:00
aoshen02
039ea82660
[Core] Sync DP state on the first step of a wave ( #52957 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
2026-09-06 02:16:58 +00:00
Shaurya Pratap Srivastava
e862c2f45b
[Docs] Add example for Renderer.render_cmpl() usage ( #54265 )
...
Signed-off-by: null-Exception1 <rising1legend@gmail.com >
2026-09-03 23:35:37 -07:00
560ef78bfe
[Perf][Model Runner V2] Compact sampling masks on GPU instead of unpacking the full-vocab bitmask on CPU ( #54901 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: vx120 <893600387@qq.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: vx120 <893600387@qq.com >
2026-09-03 19:03:36 -07:00
Ning Xie and Cyrus Leung
2a336d8239
[warmup] overlap renderer warmup and engine core initialization ( #54557 )
...
Signed-off-by: Andy Xie <andy.xning@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-09-03 19:44:21 +00:00
Jiangyun Zhu
98ed0856f3
[Model] add GLM-5.3-Flash support ( #53906 )
2026-09-03 09:40:35 -07:00