4a9d6f9572
[Bugfix] Preserve sampling masks in DELTA and TITO streaming outputs ( #55935 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: fh3 <a@b>
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: fh3 <a@b>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-01 14:43:38 +08:00
70be6aae19
[Feature][Spec Decode] Support sampling mask replay for MRV2 MTP ( #59359 )
...
Signed-off-by: Cuiping Cheng <chengcuipingswu@163.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Cuiping Cheng <chengcuipingswu@163.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 13:51:39 +08:00
Guy Stone and Claude Fable 5
37d6174096
[Metrics] Add --custom-histogram-buckets to override histogram bucket families ( #48867 )
...
The core engine histograms ship with default bucket boundaries tuned for
typical serving workloads. The `--custom-histogram-buckets` option replaces
the boundaries of one or more bucket families. Use it, for example, to track
sub-300ms latency SLOs with the request-phase histograms, whose smallest
default boundary is 0.3s.
Signed-off-by: Guy Stone <guys@spotify.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-10-01 01:03:39 +00:00
Harry Mellor and Claude Opus 5.5
fc2c801ad9
Disable mypy arg-type and assignment checks in tests ( #59428 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 15:18:50 +00:00
e5f3d08d72
[Frontend][RL] Align sleep-mode API responses and operation metrics ( #52864 )
...
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-29 23:16:38 -07:00
Andreas Karatzas
866fa130fa
[Core] Bound UniProc EngineCore startup threads to available CPUs ( #58946 )
...
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com >
2026-09-29 23:02:55 -07:00
Wentao Ye and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
834ad456d3
[Refactor] Remove dead tests code ( #58916 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-30 04:39:03 +00:00
42d9f8ccdb
support rl feature : weight checker ( #51350 )
...
Signed-off-by: shiyuan680 <yangcheng104@huawei.com >
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: shiyuan680 <yangcheng104@huawei.com >
Co-authored-by: Ronald1995 <ronaldautomobile@163.com >
Co-authored-by: DeepSeek Harness <noreply@deepseek.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-30 03:13:43 +00:00
Chanbin Lim
91de237333
[Bugfix][DP] add_dp_placement_groups does not require ray[default] ( #57648 )
...
Signed-off-by: beenpow <57864041+beenpow@users.noreply.github.com >
2026-09-29 18:15:13 +00:00
Almog Tavor
171d1b9ef6
[Elastic EP] Support Model Runner V2 ( #53934 )
...
Signed-off-by: almogtavor <almogtavor@gmail.com >
Signed-off-by: Almog Tavor <70065337+almogtavor@users.noreply.github.com >
2026-09-29 00:20:40 +00:00
Benedikt Falk
c68eb98b69
[Bugfix][Frontend] Keep in-flight requests on the same DP engine ( #59017 )
...
Signed-off-by: Benedikt Falk <bfalk@nvidia.com >
2026-09-28 23:52:29 +00:00
b721a4c709
[Feature] Add fixed-token prefill scoring ( #54335 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-09-28 07:50:09 -07:00
Tianyu Guo
53d2e16a97
[Bugfix][EPD] Skip sampling for encoder-only async steps ( #58490 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
2026-09-27 20:35:09 -07:00
Wentao Ye
bf93e1f92c
[Refactor] Remove dead tests utils ( #58803 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-26 09:16:53 -04:00
Mark McLoughlin and AI Assistant
45d56c0baf
[Core] Model console logging as CLI configuration ( #57205 )
...
Add `--logging-config` CLI argument which can be supplied as
JSON or using dotted arguments. The `--log-level` argument
is provided for convenience, and `--log-config-file` is deprecated
in favor of `--logging-config.pylogging_config_file`.
Signed-off-by: Mark McLoughlin <markmc@redhat.com >
Co-authored-by: AI Assistant <noreply@openai.com >
2026-09-25 11:36:59 +01:00
1b3b88ec2b
[Core] structured generation mode for DiffusionGemma model (Jev-like) ( #57250 )
...
Signed-off-by: Matt Mastracci <matthew@mastracci.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Razorback16 <razorback16@protonmail.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Razorback16 <razorback16@protonmail.com >
2026-09-22 20:51:17 +00:00
Tony Lin
dc479a2d83
[Bugfix] Fix external LB DP rank handling when replicas share nodes ( #53743 )
...
Signed-off-by: Tony Lin <tony.lin@intel.com >
2026-09-21 17:19:14 +02:00
0b7f11a1ee
[AuxOutput] Add block-keyed storage for routed-expert outputs ( #45635 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-21 10:25:05 +00:00
czhu-cohere
3918f3c5a3
[Profiler] Unify platform-aware torch profiling ( #57460 )
2026-09-20 19:17:23 -07:00
5302d1fe40
[Model Runner V2] Support custom logits processors ( #56497 )
...
Signed-off-by: David Cheung <d7cheung@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-19 17:46:49 +00:00
90531daf6d
[Core][Frontend] Bind KV-event publishers at port 0 and expose the bound endpoints ( #55844 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: mukeshbhandarkar <shell00035@gmail.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-18 18:11:05 -07:00
Nicolò Lucchesi
a7156060c1
[Core] Make parallel sampling (n>1) reqs admission atomic ( #53936 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-09-18 18:08:44 +02:00
092bdd6d57
[Frontend][Core] Add release_kv_cache_memory() API ( #44890 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: AlanFokCo <alanfok2868@gmail.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com >
Co-authored-by: AlanFokCo <alanfok2868@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-09-18 07:12:12 +08:00
Wei Zhao
ccde9c61e9
[Bugfix] Restore KV cache metadata GET method for external event consumer ( #56925 )
...
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-09-17 23:01:35 +00:00
4f9a41ac94
[CI/Build][Hardware][NVIDIA] Add Rubin CUDA 13.4 nightly images ( #55953 )
...
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-17 08:05:31 -07:00
Oxana Korzh and Andreas Karatzas
2307a1d16c
[Core] Don't issue a blocking collective RPC during engine handshake ( #57226 )
...
Signed-off-by: okorzh <okorzh@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-16 21:30:59 -05:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
Robert Shaw and Claude
42919b49c5
[P/D] Report prefill worker cache hits in prompt_tokens_details ( #54222 )
...
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-16 15:23:31 +00:00
f37c550bf6
[Feature][Frontend] Expose effective attention block size for DCP ( #56538 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Signed-off-by: Julien Darve <jdarve@NVIDIA.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-09-16 08:29:43 +00:00
Chauncey
f0fbc98c48
[Frontend] Share max_num_queued_reqs across API server processes ( #54746 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
2026-09-15 14:50:26 +00:00
djramic and Andreas Karatzas
c6ba5e89d6
[Bugfix][Elastic EP] Route only to surviving engines during scale-down ( #55772 )
...
Signed-off-by: Djordje Ramic <djoramic@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-08 21:28:28 -05:00
Wei Zhao
c3ec0d29f5
[Bugfix] Fix cuda profiler missing bug ( #55237 )
2026-09-07 03:15:57 +00:00
aoshen02
039ea82660
[Core] Sync DP state on the first step of a wave ( #52957 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
2026-09-06 02:16:58 +00:00
Qiming Zhang
f5e441de10
[Bugfix][Test] Fix off-by-one error in sampled token rank causing flaky logprobs test ( #53976 )
...
Signed-off-by: mayuyuace <qiming1.zhang@intel.com >
2026-08-31 18:49:43 +00:00
Nicolò Lucchesi
dafbef15a1
[Core] Add max_num_queued_reqs and max_num_queued_tokens for queue size management ( #49445 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-08-31 14:26:49 +00:00
Oxana Korzh and Claude Opus 5
fb68025138
[Bugfix][MRV2] Release model and KV cache on in-process engine shutdown ( #54162 )
...
Signed-off-by: Oxana Korzh <okorzh@amd.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-28 22:09:08 -07:00
aoshen02
17da48596c
[Bugfix][DP] Synchronize the device on pause completion ( #52914 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
2026-08-26 09:50:33 -07:00
Tony
903a02192f
[Bugfix] Handle empty FlatLogprobs slices and delta output ( #53704 )
...
Signed-off-by: tony <864832769@qq.com >
2026-08-26 21:00:54 +08:00
Naveenraj Kamalakannan and Benjamin Chislett
d5cadcee86
[Feature][DSpark]: Logprobs adaptive verification ( #52242 )
...
Signed-off-by: Naveenraj Kamalakannan <therealnaveenkamal@gmail.com >
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
2026-08-25 10:22:33 -04:00
Lucas Wilkinson
8bdc70ec7b
[6/N][KV-Cache Layout Refactor] Standardize KV cache layout ( #51718 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
2026-08-21 17:47:49 -07:00
a1c5b1fd9f
[Core][V1] Support trace_decode_token_ids for deterministic decode replay ( #46701 )
...
Signed-off-by: ianzxliu <ianzxliu@tencent.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Zhixia Liu <liuzhixia.nju@gmail.com >
Co-authored-by: ianzxliu <ianzxliu@tencent.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-20 15:12:43 +08:00
76fb6d210a
[Bugfix][Core] Reserve the KV null block when validating max_model_len ( #47272 )
...
Signed-off-by: hyungjun oh <92hyungjun@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-20 02:49:23 +00:00
lxy and “新颐”
0c8c3f41cf
[Bugfix][V1] Sync mamba_block_size via EngineCoreReadyResponse ( #50809 )
...
Signed-off-by: “新颐” <lxy537258@antgroup.com >
Co-authored-by: “新颐” <lxy537258@antgroup.com >
2026-08-19 16:44:36 +00:00
3a386cfaf5
[ROCm] Give EngineCore cleanup grace after request abort ( #52281 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-19 11:39:59 -05:00
Connor Carpenter and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f08a95f8d8
[Rust Frontend][gRPC] Advertise LoRA capabilities ( #52031 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-17 13:27:56 -07:00
Tyler Michael Smith and OpenAI Codex
017e9f4448
Promote prefix_cache_retention_interval to an argument and change the default to 0 ( #52216 )
...
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-17 13:20:50 +00:00
Connor Carpenter and Nick Hill
f473870ecf
[Rust Frontend][gRPC] Add RL lifecycle control ( #51316 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-14 11:32:39 -07:00
Ali Jaseem and Claude
64ca614fe4
[Bugfix] Fix --data-parallel-start-rank 0 being treated as unset in create_engine_config ( #47692 )
...
Signed-off-by: Syed Ali Jaseem <syedali0057@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-08-13 15:12:26 +00:00
Wentao Ye
4988df2eb8
[Config] Update default _max_num_batched_tokens from 8192 to 16384 ( #51726 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-08-11 11:09:34 -04:00
