Commit Graph
745 Commits
Author SHA1 Message Date
e319f86f15 [Feature] Per-row candidate IDs for prefill token scoring (M2 of #56860) (#56984)
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 14:17:09 +00:00
Benedikt Falk c4973d94fe [Bugfix][Frontend] Avoid generation for empty streaming input (#59015)
Signed-off-by: Benedikt Falk <bfalk@nvidia.com>
2026-10-02 18:48:43 +00:00
Taneem Ibrahim f0c44cc7d5 [Bugfix] Fix stalled local-only Elastic EP scale-up (#57820)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
2026-10-02 10:47:55 -04:00
592c6f3fbc [Bugfix][Pooling] Handle multimodal cache misses without crashing (#58975)
Signed-off-by: linchaoying6 <linchaoying6@jd.com>
Co-authored-by: linchaoying6 <linchaoying6@jd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-10-02 10:14:33 -04:00
Martin Hickey fcfacc3499 [Bugfix][Frontend] Seed derender detokenization from the prompt (#59046)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
2026-10-02 03:17:56 +00:00
Shijie Lyu 9aaef4296f [Bugfix][Frontend] Check reused prompt token ids against the vocab before streaming (#59555)
Signed-off-by: Shijie Lyu <lshjhf@gmail.com>
2026-10-01 09:26:55 -07:00
4a9d6f9572 [Bugfix] Preserve sampling masks in DELTA and TITO streaming outputs (#55935)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: fh3 <a@b>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: fh3 <a@b>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-01 14:43:38 +08:00
70be6aae19 [Feature][Spec Decode] Support sampling mask replay for MRV2 MTP (#59359)
Signed-off-by: Cuiping Cheng <chengcuipingswu@163.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Cuiping Cheng <chengcuipingswu@163.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 13:51:39 +08:00
e5f3d08d72 [Frontend][RL] Align sleep-mode API responses and operation metrics (#52864)
Signed-off-by: Ronald1995 <ronaldautomobile@163.com>
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:16:38 -07:00
Andreas Karatzas 866fa130fa [Core] Bound UniProc EngineCore startup threads to available CPUs (#58946)
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
2026-09-29 23:02:55 -07:00
Mia Ojeda e006d761a5 [Bugfix][CPU][DiffusionGemma] Support narrower canvas w/ sync scheduling (#59107)
Signed-off-by: ojeda-e <estefania.b.ojeda@gmail.com>
2026-09-29 23:28:50 -04:00
42d9f8ccdb support rl feature : weight checker (#51350)
Signed-off-by: shiyuan680 <yangcheng104@huawei.com>
Signed-off-by: Ronald1995 <ronaldautomobile@163.com>
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: shiyuan680 <yangcheng104@huawei.com>
Co-authored-by: Ronald1995 <ronaldautomobile@163.com>
Co-authored-by: DeepSeek Harness <noreply@deepseek.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-09-30 03:13:43 +00:00
a7a9ac2745 [Core] Combine per-engine utility results like the Rust client (#59240)
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:20:51 +00:00
Chanbin Lim 91de237333 [Bugfix][DP] add_dp_placement_groups does not require ray[default] (#57648)
Signed-off-by: beenpow <57864041+beenpow@users.noreply.github.com>
2026-09-29 18:15:13 +00:00
Almog Tavor 171d1b9ef6 [Elastic EP] Support Model Runner V2 (#53934)
Signed-off-by: almogtavor <almogtavor@gmail.com>
Signed-off-by: Almog Tavor <70065337+almogtavor@users.noreply.github.com>
2026-09-29 00:20:40 +00:00
Benedikt Falk c68eb98b69 [Bugfix][Frontend] Keep in-flight requests on the same DP engine (#59017)
Signed-off-by: Benedikt Falk <bfalk@nvidia.com>
2026-09-28 23:52:29 +00:00
b721a4c709 [Feature] Add fixed-token prefill scoring (#54335)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-28 07:50:09 -07:00
Tianyu Guo 53d2e16a97 [Bugfix][EPD] Skip sampling for encoder-only async steps (#58490)
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
2026-09-27 20:35:09 -07:00
Mark McLoughlinandAI Assistant 45d56c0baf [Core] Model console logging as CLI configuration (#57205)
Add `--logging-config` CLI argument which can be supplied as
JSON or using dotted arguments. The `--log-level` argument
is provided for convenience, and `--log-config-file` is deprecated
in favor of `--logging-config.pylogging_config_file`.

Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: AI Assistant <noreply@openai.com>
2026-09-25 11:36:59 +01:00
Taneem Ibrahim bbd7c24d6e [MRV2] Validate MRV2 entrypoint logits processors (#57728)
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
2026-09-24 09:07:26 +00:00
c0c35ddd8e [Bugfix][CI] Fix the flaky sharded-sampling tests, and the engine teardown need (#58342)
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-24 02:05:11 -07:00
1b3b88ec2b [Core] structured generation mode for DiffusionGemma model (Jev-like) (#57250)
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Razorback16 <razorback16@protonmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Razorback16 <razorback16@protonmail.com>
2026-09-22 20:51:17 +00:00
shaohuaxiandCodex 5abf85d27a [Bugfix][Frontend] Validate mixed prompt embedding mask lengths (#57006)
Signed-off-by: 子华 <huaxi.shx@alibaba-inc.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-22 05:50:31 +00:00
Juan Pérez de Algaba 4800024b98 [Security] Reject min_tokens that exceeds the filled max_tokens default (#57731)
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com>
2026-09-22 05:50:23 +00:00
Karen Chung 986e217158 [Feature] Add first-class KV hints request envelope for programmatic KV management (#53423)
Signed-off-by: Karen Chung <karenc@nvidia.com>
2026-09-21 09:56:11 -07:00
Cyrus Leung c8824a1e42 [MM] Move supports_multimodal_inputs and cache out of registry (#57913)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-09-21 07:43:07 -07:00
czhu-cohere 3918f3c5a3 [Profiler] Unify platform-aware torch profiling (#57460) 2026-09-20 19:17:23 -07:00
1185254851 [Bugfix][KV Connector] Propagate cache reset failure during sleep (#54581)
Signed-off-by: CrazyCoder <18235787078@163.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
2026-09-21 09:25:02 +08:00
5302d1fe40 [Model Runner V2] Support custom logits processors (#56497)
Signed-off-by: David Cheung <d7cheung@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-19 17:46:49 +00:00
90531daf6d [Core][Frontend] Bind KV-event publishers at port 0 and expose the bound endpoints (#55844)
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: mukeshbhandarkar <shell00035@gmail.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-09-18 18:11:05 -07:00
Nicolò Lucchesi a7156060c1 [Core] Make parallel sampling (n>1) reqs admission atomic (#53936)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-09-18 18:08:44 +02:00
092bdd6d57 [Frontend][Core] Add release_kv_cache_memory() API (#44890)
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: AlanFokCo <alanfok2868@gmail.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com>
Co-authored-by: AlanFokCo <alanfok2868@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-09-18 07:12:12 +08:00
Wei Zhao ccde9c61e9 [Bugfix] Restore KV cache metadata GET method for external event consumer (#56925)
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
2026-09-17 23:01:35 +00:00
Oxana KorzhandAndreas Karatzas 2307a1d16c [Core] Don't issue a blocking collective RPC during engine handshake (#57226)
Signed-off-by: okorzh <okorzh@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-16 21:30:59 -05:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
Robert ShawandClaude 42919b49c5 [P/D] Report prefill worker cache hits in prompt_tokens_details (#54222)
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-16 15:23:31 +00:00
f37c550bf6 [Feature][Frontend] Expose effective attention block size for DCP (#56538)
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
2026-09-16 08:29:43 +00:00
Chaojun ZhangandKunshang Ji 622934e61f [XPU] Honor device_ids for worker device placement (#56015)
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-09-16 11:53:23 +08:00
Chauncey f0fbc98c48 [Frontend] Share max_num_queued_reqs across API server processes (#54746)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-09-15 14:50:26 +00:00
84e5f3edac [Bugfix] Validate routed-expert prompt offsets before engine submission (#56844)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Bhushan Asati <bhushanasati25@gmail.com>
Co-authored-by: Megha Agarwal <19240983+meghaagr13@users.noreply.github.com>
2026-09-15 09:17:41 +08:00
Itay Alroy 658c8131c7 [Elastic EP] Reuse CUDA graphs across reconfiguration (#54985) 2026-09-12 10:26:39 -04:00
djramicandAndreas Karatzas c6ba5e89d6 [Bugfix][Elastic EP] Route only to surviving engines during scale-down (#55772)
Signed-off-by: Djordje Ramic <djoramic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-08 21:28:28 -05:00
Wentao Ye 6b15bea080 [Refactor] Remove utils dead code (#53941)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-09-08 11:22:09 -04:00
Wei Zhao c3ec0d29f5 [Bugfix] Fix cuda profiler missing bug (#55237) 2026-09-07 03:15:57 +00:00
aoshen02 039ea82660 [Core] Sync DP state on the first step of a wave (#52957)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
2026-09-06 02:16:58 +00:00
Shaurya Pratap Srivastava e862c2f45b [Docs] Add example for Renderer.render_cmpl() usage (#54265)
Signed-off-by: null-Exception1 <rising1legend@gmail.com>
2026-09-03 23:35:37 -07:00
560ef78bfe [Perf][Model Runner V2] Compact sampling masks on GPU instead of unpacking the full-vocab bitmask on CPU (#54901)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: vx120 <893600387@qq.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: vx120 <893600387@qq.com>
2026-09-03 19:03:36 -07:00
Ning XieandCyrus Leung 2a336d8239 [warmup] overlap renderer warmup and engine core initialization (#54557)
Signed-off-by: Andy Xie <andy.xning@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-09-03 19:44:21 +00:00
Jiangyun Zhu 98ed0856f3 [Model] add GLM-5.3-Flash support (#53906) 2026-09-03 09:40:35 -07:00