Guy Stone and Claude Fable 5
37d6174096
[Metrics] Add --custom-histogram-buckets to override histogram bucket families ( #48867 )
...
The core engine histograms ship with default bucket boundaries tuned for
typical serving workloads. The `--custom-histogram-buckets` option replaces
the boundaries of one or more bucket families. Use it, for example, to track
sub-300ms latency SLOs with the request-phase histograms, whose smallest
default boundary is 0.3s.
Signed-off-by: Guy Stone <guys@spotify.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-10-01 01:03:39 +00:00
e5f3d08d72
[Frontend][RL] Align sleep-mode API responses and operation metrics ( #52864 )
...
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-29 23:16:38 -07:00
42d9f8ccdb
support rl feature : weight checker ( #51350 )
...
Signed-off-by: shiyuan680 <yangcheng104@huawei.com >
Signed-off-by: Ronald1995 <ronaldautomobile@163.com >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: shiyuan680 <yangcheng104@huawei.com >
Co-authored-by: Ronald1995 <ronaldautomobile@163.com >
Co-authored-by: DeepSeek Harness <noreply@deepseek.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-30 03:13:43 +00:00
linitra24 and Jee Jee Li
953f90d25f
[LoRA] Support variable num_labels for sequence classification ( #57766 )
...
Signed-off-by: linitra24 <renshuang.zhou@daocloud.io >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
2026-09-28 14:58:35 +00:00
Mark McLoughlin and AI Assistant
45d56c0baf
[Core] Model console logging as CLI configuration ( #57205 )
...
Add `--logging-config` CLI argument which can be supplied as
JSON or using dotted arguments. The `--log-level` argument
is provided for convenience, and `--log-config-file` is deprecated
in favor of `--logging-config.pylogging_config_file`.
Signed-off-by: Mark McLoughlin <markmc@redhat.com >
Co-authored-by: AI Assistant <noreply@openai.com >
2026-09-25 11:36:59 +01:00
Robert Shaw
6697a7dd30
[Scheduler] Tune --long-prefill-token-threshold adaptiveness ( #58459 )
...
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com >
2026-09-23 22:23:18 -04:00
Louie Tsai and Harry Mellor
6dc34b6334
[CPU] Add device-memory-utilization CLI alias ( #56547 )
...
Signed-off-by: louie-tsai <louie.tsai@intel.com >
Signed-off-by: Louie Tsai <louie.tsai@intel.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-09-22 17:37:15 +08:00
639461eda4
[RL][Sleep] Retain frozen weights across level-2 sleep ( #57891 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-22 12:36:55 +08:00
Karen Chung
986e217158
[Feature] Add first-class KV hints request envelope for programmatic KV management ( #53423 )
...
Signed-off-by: Karen Chung <karenc@nvidia.com >
2026-09-21 09:56:11 -07:00
Tony Lin
dc479a2d83
[Bugfix] Fix external LB DP rank handling when replicas share nodes ( #53743 )
...
Signed-off-by: Tony Lin <tony.lin@intel.com >
2026-09-21 17:19:14 +02:00
0b7f11a1ee
[AuxOutput] Add block-keyed storage for routed-expert outputs ( #45635 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-21 10:25:05 +00:00
roikoren755 and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
49ee12d742
[Misc] Rename --enable-mamba-fine-grained-prefix-cache ( #53945 follow-up) ( #57382 )
...
Signed-off-by: Roi Koren <roik@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-20 09:50:34 -07:00
90531daf6d
[Core][Frontend] Bind KV-event publishers at port 0 and expose the bound endpoints ( #55844 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: mukeshbhandarkar <shell00035@gmail.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-09-18 18:11:05 -07:00
1dc2d854c1
[Feat][Model] Support encoder-side SWA-bounded replay for DeepSeek-V4.1-Flash ( #56227 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
2026-09-18 15:01:24 -07:00
Harry Mellor and Claude Opus 5
8a5cf54387
[Multimodal] Type dummy options per modality ( #57576 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-18 17:29:11 +01:00
092bdd6d57
[Frontend][Core] Add release_kv_cache_memory() API ( #44890 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: AlanFokCo <alanfok2868@gmail.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com >
Co-authored-by: AlanFokCo <alanfok2868@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-09-18 07:12:12 +08:00
Chuan (Richard) Li and Cursor
f730a93d2b
[Scheduler] Add --max-num-active-seqs to cap RUNNING admission ( #56758 )
...
Signed-off-by: Li <chuali@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-17 10:22:12 +08:00
jxj and Harry Mellor
6b6bf6b3c3
[Bugfix] Remove unsupported comma-separated detailed trace values ( #55702 )
...
Signed-off-by: git-jxj <65210887+git-jxj@users.noreply.github.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-09-16 18:44:48 +00:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
903285fbcc
[Bugfix] Let an optional Literal flag accept the None it advertises ( #52370 )
...
Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-09-16 12:14:54 +00:00
wangxiyuan and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
118e17f5ff
[Platform] Move env check function to platform interface ( #48599 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-16 10:08:36 +00:00
fa008bdccf
[Perf][Kernel] Integrate DeepSelect TopK for the DSA sparse indexer ( #56464 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Kimi Code <noreply@moonshot.ai >
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-09-13 00:44:37 -07:00
Itay Alroy
658c8131c7
[Elastic EP] Reuse CUDA graphs across reconfiguration ( #54985 )
2026-09-12 10:26:39 -04:00
7cdd9304ae
[KV Connector] Support symmetric DCP disagg for hybrid mamba models ( #55531 )
...
Signed-off-by: Wei Zhao <weizha@oci-aga-slurm-1-vscode-02.cm.cluster >
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Wei Zhao <weizha@oci-aga-slurm-1-vscode-02.cm.cluster >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-10 11:12:15 -07:00
9521c60bdc
[ROCm][Perf] Kimi-K3 Fused kernels for KDA prefill reland ( #54038 )
...
Signed-off-by: kliuae <kuanfu.liu@embeddedllm.com >
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com >
Signed-off-by: kliuae <17350011+kliuae@users.noreply.github.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: TJian <tunjian.tan@embeddedllm.com >
2026-09-10 15:37:54 +08:00
ea40bb9e90
[feature] Watermarked generation and detection (Gumbel-max algorithm) ( #54053 )
...
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-09 21:00:15 -07:00
cc4210f671
[Perf] Improve BF16x3 router GEMM accuracy and make it default on sm100 ( #55899 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
2026-09-09 07:33:12 -07:00
Harry Huang and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
3116c5d06b
[Qwen4Exp] Support UVA PLE-offload and Engram tensor parallelism ( #54371 )
...
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-09 07:32:34 -07:00
263c4ff95f
[Bugfix][Spec Decode] Cache the Mamba state at the block-grid position of EAGLE resume ( #53945 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Signed-off-by: Adam Shaver <ashaver@nvidia.com >
Signed-off-by: akshaver <168006157+akshaver@users.noreply.github.com >
Co-authored-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: roikoren755 <26850796+roikoren755@users.noreply.github.com >
2026-09-08 09:59:58 -04:00
Wei Zhao
a69402aaa8
[Perf] Integrate FlashInfer KDA kernels ( #55364 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
2026-09-07 20:29:30 -07:00
aoshen02
039ea82660
[Core] Sync DP state on the first step of a wave ( #52957 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
2026-09-06 02:16:58 +00:00
ee0a4c46ae
[Bugfix] Account for PCP in multi-node world size validation ( #55111 )
...
Signed-off-by: DebugSy <532151398@qq.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-09-03 10:52:42 +00:00
ffe3bb3c72
[Hardware][XPU] Register matmul and linear batch-invariant kernels for XPU ( #49209 )
...
Signed-off-by: tzielinski-habana <tomasz.zielinski@intel.com >
Signed-off-by: Tomasz Zielinski <85164140+tzielinski-habana@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-09-02 13:31:38 +00:00
4ac452ad98
[Core] Release NCCL communicator memory in sleep mode ( #51485 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-09-01 14:31:16 +08:00
Nicolò Lucchesi
dafbef15a1
[Core] Add max_num_queued_reqs and max_num_queued_tokens for queue size management ( #49445 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-08-31 14:26:49 +00:00
Giancarlo Delfin
d154d90d6c
[Model Runner V2] batch-sharded sample ( #50465 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
2026-08-24 15:49:57 -07:00
Kevin H. Luu and Andreas Karatzas
592e06f2ae
Revert "[ROCm][Perf] Kimi-K3 Fused kernels for KDA prefill" ( #53294 )
...
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-08-21 13:34:13 -07:00
463aa5e30f
[ROCm][Perf] Kimi-K3 Fused kernels for KDA prefill ( #52606 )
...
Signed-off-by: kliuae <kuanfu.liu@embeddedllm.com >
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: TJian <tunjian.tan@embeddedllm.com >
2026-08-21 09:07:13 -07:00
72aedcc426
[DCP] Default query replication for GLM sparse attention ( #50382 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-21 09:07:05 -07:00
7cfb97e337
[Frontend][Core][Spec Decode] Per-request acceptance stats in OpenAI API responses ( #48915 )
...
Signed-off-by: Matthew Kotila <7692737+matthewkotila@users.noreply.github.com >
Co-authored-by: Reed Meyerson <31574681+reed-meyerson@users.noreply.github.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-08-20 15:42:49 -07:00
a1c5b1fd9f
[Core][V1] Support trace_decode_token_ids for deterministic decode replay ( #46701 )
...
Signed-off-by: ianzxliu <ianzxliu@tencent.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Zhixia Liu <liuzhixia.nju@gmail.com >
Co-authored-by: ianzxliu <ianzxliu@tencent.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-20 15:12:43 +08:00
Tyler Michael Smith and OpenAI Codex
017e9f4448
Promote prefix_cache_retention_interval to an argument and change the default to 0 ( #52216 )
...
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-17 13:20:50 +00:00
wangxiyuan and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
cdc4824a21
[Misc] Remove override_attention_dtype ( #48684 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-14 11:15:43 -04:00
Ali Jaseem and Claude
64ca614fe4
[Bugfix] Fix --data-parallel-start-rank 0 being treated as unset in create_engine_config ( #47692 )
...
Signed-off-by: Syed Ali Jaseem <syedali0057@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-08-13 15:12:26 +00:00
50ba4bc6b2
[Feature] Mask Replay ( #49577 )
...
Signed-off-by: vx120 <893600387@qq.com >
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-13 08:44:31 +00:00
hotTea and HMCCMH
7bbbf7c8e5
[Core] Configure custom encoder cache managers from VllmConfig ( #51251 )
...
Signed-off-by: hotTea <958436561@qq.com >
Signed-off-by: HMCCMH <chenminghaoscu@163.com >
Co-authored-by: HMCCMH <chenminghaoscu@163.com >
2026-08-13 16:08:01 +08:00
Wentao Ye
4988df2eb8
[Config] Update default _max_num_batched_tokens from 8192 to 16384 ( #51726 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-08-11 11:09:34 -04:00
Isotr0py
073c510c91
[Migration] Migrate bitsandbytes support to OOT plugin ( #43529 )
...
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
2026-08-09 10:27:31 +08:00
aoshen02 and OpenAI Codex
22a175921c
[BugFix] Reject invalid data-parallel RPC ports ( #51469 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-08 12:23:26 +08:00
wang.yuqi and Cyrus Leung
2fa490470d
[Model] Fused mm preprocess normalisation on the Device ( #50411 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com >
2026-08-06 03:11:14 -07:00
