d58da79de7
[Metrics][P/D] Add KV-fetch stage gauges for async KV loads ( #58874 )
...
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-02 01:24:33 +00:00
28c57456db
[Bugfix][Frontend] Accept Anthropic tool_addition and tool_removal content blocks ( #57693 )
...
Signed-off-by: Anurag Singh <dr.anuragkr@gmail.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: Antigravity <antigravity@google.com >
Co-authored-by: sfeng33 <4florafeng@gmail.com >
2026-10-01 20:03:36 -04:00
Divakar Verma and Claude Sonnet 5
e6674262f1
[Bugfix][CI] Widen DBO+DP+EP GSM8K accuracy margin on ROCm ( #59700 )
...
Signed-off-by: Divakar Verma <divakar.verma@amd.com >
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com >
2026-10-01 23:37:36 +00:00
hcl and Harry Mellor
e35082e7b6
[Bugfix][CLI] Include inherited field docstrings in get_attr_docs ( #49821 )
...
Signed-off-by: Chenglun Hu <chenglunhu@gmail.com >
Signed-off-by: hclsys <chenglunhu@gmail.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-10-01 23:35:58 +00:00
Aarushi Jain
d2125ea681
[ROCm][CI] Raise the MI355 DeepSeek-R1 GSM8K startup wait to 1800s ( #59666 )
...
Signed-off-by: aarushjain29 <aarushi.jain2@amd.com >
2026-10-01 22:30:13 +00:00
Flora Feng
a5105dba05
[Frontend] Port Step-3.5 parsers to the streaming parser engine ( #59321 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-10-01 22:18:37 +00:00
ca65eb67d9
[Perf][HiSparse] Avoid repeated prefix scans and residency updates ( #57930 )
...
Signed-off-by: S1ro1 <matej.sirovatka@gmail.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-10-01 21:48:18 +00:00
Yongye Zhu and Claude Opus 5.5
7566d83bd3
[Attention][MiniMax-M3] NVFP4 KV cache on the MSA sparse attention path ( #59300 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 14:39:55 -07:00
Zijing Liu
ea84051819
[KVConnector][NIXL] Count completion notifications that arrive after KV expiry ( #58875 )
...
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com >
2026-10-01 21:30:58 +00:00
2a52715ef7
[Bugfix][Responses] Use standard reasoning content-part events ( #59652 )
...
Signed-off-by: ai-jz <ai-jz@users.noreply.github.com >
Co-authored-by: ai-jz <ai-jz@users.noreply.github.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-01 19:20:57 +00:00
Chanbin Lim and Chendi.Xue
47de9d4d04
[KV Connector][NIXL] Coalesce host-buffer KV copies across cache groups ( #54483 )
...
Signed-off-by: beenpow <57864041+beenpow@users.noreply.github.com >
Co-authored-by: Chendi.Xue <chendi.xue@intel.com >
2026-10-01 13:10:54 -05:00
e9be5323f0
[Perf] Integrate flash-maxsim Triton kernels for late-interaction scoring ( #40337 )
...
Signed-off-by: roi.pony <roi.pony@ibm.com >
Co-authored-by: roi.pony <roi.pony@ibm.com >
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
2026-10-01 14:06:43 -04:00
Matthew Bonanni and Claude Opus 5.5
3a69636645
[Bugfix][HiSparse] Fix a chunked-prefill preemption livelock ( #59494 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 13:02:31 -04:00
26bfdfb6bc
[ROCm][Perf] Parallelise AITER MLA page-index expansion over token chunks ( #57978 )
...
Signed-off-by: lifulu <fululi12@amd.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-01 12:00:15 -05:00
Shijie Lyu
9aaef4296f
[Bugfix][Frontend] Check reused prompt token ids against the vocab before streaming ( #59555 )
...
Signed-off-by: Shijie Lyu <lshjhf@gmail.com >
2026-10-01 09:26:55 -07:00
aniskumar-nv and Cyrus Leung
83cadd65d9
[Bugfix] Tie lm_head.weight for Nemotron Parse when checkpoint omits it ( #53020 )
...
Signed-off-by: Anish Kumar <aniskumar@nvidia.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-01 09:26:33 -07:00
b538d807ff
[Bugfix][MRV2] Keep GDN prefill checkpoint metadata local to each cache group ( #59536 )
...
Signed-off-by: Jingqiao Zhang <ai-jz@users.noreply.github.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Jingqiao Zhang <ai-jz@users.noreply.github.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
2026-10-01 22:59:15 +08:00
5d9214d6b2
[Bugfix][Frontend] Strip x-anthropic-billing-header billing header from /v1/chatcompletions ( #59419 )
...
Signed-off-by: Maroon Ayoub <mayoub@redhat.com >
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Robert Shaw <robertgshaw2@gmail.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-10-01 13:44:40 +00:00
cb6458c5ce
[Core] Remove deprecated mamba_cache_mode "all" ( #58997 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Thomas Parnell <tpa@zurich.ibm.com >
2026-10-01 02:59:37 -07:00
Mark McLoughlin and OpenAI Codex
91fcec1235
[Core][Logging] Add built-in JSON formatter ( #58739 )
...
Add `--logging-config.formatter=json` to enable the python-json-logger
formatter without having to use a complete Python logging dictConfig.
Signed-off-by: Mark McLoughlin <markmc@redhat.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-10-01 09:11:46 +01:00
5addde2a0a
[CPU][Zen] Pass f32 weight scales to the zentorch INT8 MoE ( #59434 )
...
Signed-off-by: Ganesh R <Ganesh.R@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
2026-10-01 15:51:53 +08:00
jiangyunfan1 and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
e12b735109
[Test] Make Anthropic messages test compatible with SDK 1.x via extra_body ( #57780 )
...
Signed-off-by: jiangyunfan1 <jiangyunfan1@h-partners.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-01 07:38:29 +00:00
e12291d733
[CPU][Zen] Add DA8W4 (W4A8) int4 support for dense and MoE layers ( #54024 )
...
Signed-off-by: R <Ganesh.R@amd.com >
Signed-off-by: Ganesh R <Ganesh.R@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-01 15:18:25 +08:00
Wentao Ye
4d6874d137
[GLM 5.3 Perf] Skip qlnorm calculation for MHA, 4.4~7.7% E2E TTFT Improvement ( #58845 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-10-01 15:13:57 +08:00
87a4bf664f
[Perf][GDN] Slice pure spec-decode rows instead of a host-mask gather ( #58763 )
...
Signed-off-by: Luca Motz <luca.motz@icloud.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com >
2026-10-01 06:53:06 +00:00
4a9d6f9572
[Bugfix] Preserve sampling masks in DELTA and TITO streaming outputs ( #55935 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: fh3 <a@b>
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: fh3 <a@b>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-01 14:43:38 +08:00
Shuolei Wang and Thien
c9578f10ab
[Qwen3.8-Flash-Next] Fuse main QK-norm/RoPE/gate and KV-cache write into the QSA pre-indexer launch ( #57097 )
...
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com >
Signed-off-by: Thien <thien@inferact.ai >
Co-authored-by: Thien <thien@inferact.ai >
2026-10-01 14:21:06 +08:00
70be6aae19
[Feature][Spec Decode] Support sampling mask replay for MRV2 MTP ( #59359 )
...
Signed-off-by: Cuiping Cheng <chengcuipingswu@163.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Cuiping Cheng <chengcuipingswu@163.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 13:51:39 +08:00
Shijie Lyu
37cecc8141
[Bugfix][Tokenizer] Fix off-by-one max_token_id from vocab_size ( #59491 )
...
Signed-off-by: Shijie Lyu <lshjhf@gmail.com >
2026-10-01 05:43:30 +00:00
Juntian Liu and Claude Opus 5.5
4c2d277643
[Perf][DSv4.1] Faster Engram host lookups: sorted rows, inline big lookups ( #59327 )
...
Signed-off-by: Juntian Liu <Juntianl777@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-01 11:54:44 +08:00
namgyu-youn and Thien Tran
4ed438b79c
[Perf][Qwen4Exp] Add SM100 low-latency decode GEMM plans ( #59214 )
...
Signed-off-by: namgyu-youn <namgyu.dev@gmail.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-10-01 03:40:55 +00:00
Yoni Gozlan
2eaa3bc5ac
[Frontend] Enrich chat_parsing streaming events ( #58603 )
...
Give serving adapters the delimiter spans and opener captures they need,
and keep streaming after a region that fails to parse.
Signed-off-by: Yoni Gozlan <yonigozlan@users.noreply.github.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Fix mypy index types in the event-stream test.
2026-10-01 11:13:13 +08:00
Nils Matteson
9e6550bd05
[Bugfix] Register IR providers before hashing configuration ( #58756 )
...
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com >
2026-10-01 02:34:28 +00:00
766b4ae54c
[Kimi-K3] Add FlashInfer speculative KDA backend ( #54255 )
...
Signed-off-by: Duncan Moss <dmoss@nvidia.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-10-01 02:28:50 +00:00
edaf3e7b61
[Attention][SM120] Occupancy-adaptive Triton split-K segment count ( #58482 )
...
Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: vadiklyutiy <156319763+vadiklyutiy@users.noreply.github.com >
2026-10-01 02:20:33 +00:00
7314360c9e
[Perf][PP] Skip sampled-token broadcasts whose requests leave the engine ( #58542 )
...
Signed-off-by: LostFox11 <wangziyue17@huawei.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: LostFox11 <wangziyue17@huawei.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-30 19:18:55 -07:00
34c458d27f
[Bugfix] Don't let a structured-output request sample from an unmasked row ( #54442 )
...
Signed-off-by: ArcheyChen <45146080+ArcheyChen@users.noreply.github.com >
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
Co-authored-by: GPT-5.6 <noreply@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-10-01 02:11:44 +00:00
df8fd42116
[CI][Spec Decode] Add async scheduling accuracy tests ( #55840 )
...
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-10-01 01:06:24 +00:00
Guy Stone and Claude Fable 5
37d6174096
[Metrics] Add --custom-histogram-buckets to override histogram bucket families ( #48867 )
...
The core engine histograms ship with default bucket boundaries tuned for
typical serving workloads. The `--custom-histogram-buckets` option replaces
the boundaries of one or more bucket families. Use it, for example, to track
sub-300ms latency SLOs with the request-phase histograms, whose smallest
default boundary is 0.3s.
Signed-off-by: Guy Stone <guys@spotify.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-10-01 01:03:39 +00:00
Nick Hill and Claude Opus 5.5
c055c1e075
[Core] Include the LoRA path in prefix-cache block hashes ( #59335 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 08:37:09 +08:00
Venky and Claude Opus 5.5
d7c55116ac
[CI] Fix MiniMax-M3 PP aux-state test mock after #58648 ( #59508 )
...
Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-09-30 17:34:52 -07:00
Kevin H. Luu and Claude Opus 5.5
8bc31bd210
[CI] Deflake the multi-API-server metrics test and Mooncake PD ports ( #59507 )
...
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 17:22:58 -07:00
jackLei
21c08ac458
[CI] Deflake shutdown wait-timeout test by synchronizing on request admission ( #58125 )
...
Signed-off-by: jackLei0901 <42642542+jackLei0901@users.noreply.github.com >
2026-10-01 00:10:03 +00:00
Andrii Skliar and Andrii Skliar
deb148e730
[Mamba] Add FlashInfer ReplaySSM support for MTP ( #52928 )
...
Signed-off-by: Andrii Skliar <askliar@nvidia.com >
Co-authored-by: Andrii Skliar <askliar@nvidia.com >
2026-09-30 21:41:17 +00:00
kliuae and Nick Hill
7e583e615c
[Bugfix][Core] Fix mamba prefill checkpoint block reservation and prompt-end eviction in align mode ( #59175 )
...
Signed-off-by: kliuae <kuanfu.liu@embeddedllm.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-30 21:26:54 +00:00
Venky and Claude Opus 5.5
42f0c17ea7
[Bugfix][PP][Spec Decode] MiniMax-M3 EAGLE3 aux-state relay at PP > 1 and per-stage FlashInfer autotune ( #57197 )
...
Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-01 01:14:24 +04:00
3eb6cec22a
[Bugfix][HiSparse] Adopt GPU prefix copies after the hit's allocation ( #59282 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-30 15:43:09 -04:00
84f738f187
[Perf][MiniMax-M3] Triton indexer: decode grid retune + SM12.0 split-K ( #56151 )
...
Signed-off-by: venkywonka <23023424+venkywonka@users.noreply.github.com >
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com >
Co-authored-by: Kyle Liang <199154931+kyleliang-nv@users.noreply.github.com >
2026-09-30 23:33:23 +04:00
djramic and Andreas Karatzas
b56b54bc54
[ROCm][CI] Relax simple-nemotron-h-8b GSM8K threshold on ROCm ( #59477 )
...
Signed-off-by: Djordje Ramic <djoramic@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-30 19:27:21 +00:00
Matthew Bonanni and Claude Opus 5.5
a21a3f2ec7
[CI] Add supports_multimodal_inputs to test_executor_replace's mock config ( #59476 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 19:04:47 +00:00
