952694e384
[Bugfix] Validate NIXL speculative config compatibility ( #49230 )
...
Signed-off-by: Tzu-Ling <tzulingk@nvidia.com >
Co-authored-by: GPT-5.6 Sol <noreply@openai.com >
Co-authored-by: Cursor Grok 4.5 <noreply@cursor.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-08-04 02:51:04 +08:00
wangxiyuan
dd11df04f3
[Misc] Remove deprecated calculate_kv_scales runtime KV scale calculation ( #49389 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2026-08-03 11:15:13 -04:00
Nicolò Lucchesi
8ba87e0181
[CI] Mooncake PD integration tests ( #46844 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-08-03 06:55:51 -07:00
Fede Kamelhar and Harry Mellor
005fa01756
docs: document reasoning_content output removal as a breaking client change ( #50624 )
...
Signed-off-by: fede-kamel <fkamelhar@gmail.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-08-03 13:45:23 +00:00
zofia and Kunshang Ji
e481da9508
[XPU] [Linear] add torch as xpu linear backend ( #49664 )
...
Signed-off-by: Zhu, Zufang <zufang.zhu@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-08-03 14:30:21 +08:00
Nicolò Lucchesi
17beffd55f
[Misc] Clarify mono audio requirement ( #50141 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-07-31 10:46:53 +00:00
yinfengLiu
f1e8fd27ae
[ROCm] Add AITER FP8 ViT encoder attention ( #49937 )
...
Signed-off-by: Liuyinfeng01 <yinfeliu@amd.com >
2026-07-30 06:22:10 +00:00
Brandon Pelfrey and Cyrus Leung
5c7a7f9462
[docs] Add documentation for pynvvideocodec video decoding backend ( #49066 )
...
Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-07-29 17:23:04 +00:00
Seiji Eicher
6370e53f24
[Frontend] Reuse prefill token ids on the decode chat path for disaggregated serving ( #48145 )
...
Signed-off-by: Seiji Eicher <seiji@anyscale.com >
2026-07-29 10:02:57 +02:00
Philip Pesic
dc1be79031
Add CachePolicyFactory for pluggable/external eviction policies ( #49114 )
...
Signed-off-by: Philip Pesic <philippesic06@gmail.com >
2026-07-29 07:34:55 +03:00
nvbfalk
60915c972c
[Feature] Add VidCom2 video token pruning ( #47750 )
...
Signed-off-by: Benedikt Falk <bfalk@nvidia.com >
2026-07-28 03:20:17 +00:00
Harjoth Khara and Claude Opus 5
8de50e46d4
[Docs] Document NVFP4 GEMM kernel selection and Marlin weight-only fallback ( #49376 )
...
Signed-off-by: harjoth <harjoth.khara@gmail.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-07-26 20:53:28 -07:00
da3a252fd1
[KVOffload][P2P] Generic P2P secondary tier: peer lookup and serving via ParentManager ( #48021 )
...
Signed-off-by: Liran Schour <lirans@il.ibm.com >
Signed-off-by: liranschour <liranschour@users.noreply.github.com >
Co-authored-by: Or Ozeri <or@ozery.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-07-26 11:45:47 +03:00
Brandon Pelfrey and OpenAI Codex
94682b79f4
[multimodal] Make PyNvVideoCodec decoder concurrency configurable ( #49753 )
...
Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-07-24 23:31:41 -07:00
Andreas Karatzas
16aca639b7
[ROCm] Upgrade NIXL and UCX ( #49251 )
...
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
2026-07-21 22:30:02 -05:00
Chang Guo and OpenAI Codex
f007cceb42
[KV Offload] Support self-describing KV events with TieringOffloadingSpec ( #48679 )
...
Signed-off-by: Change72 <changg@nvidia.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-07-20 16:41:58 +03:00
Itay Etelis and Itay Etelis
5245c80564
[Doc] Document blocks_per_chunk in the KV offloading guide ( #49100 )
...
Signed-off-by: Itay Etelis <itay.etelis@ibm.com >
Co-authored-by: Itay Etelis <itay.etelis@ibm.com >
2026-07-20 09:48:43 +03:00
426d48bfa1
[KV Offload] Add optional tier locality to FS/OBJ KV events ( #48281 )
...
Signed-off-by: Change72 <changg@nvidia.com >
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-07-17 10:31:52 +03:00
ab0a20d151
[Docs] Add Phi-3.5-mini-instruct to batch invariance tested models ( #46396 )
...
Signed-off-by: Yuval Luria <yuvalluria@users.noreply.github.com >
Co-authored-by: Yuval Luria <yuvalluria@users.noreply.github.com >
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com >
2026-07-16 18:05:58 -04:00
liranschour and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
615834ee58
[KVOffload][P2P] Well-known default host/port env vars and per-DP-rank control port ( #47636 )
...
Signed-off-by: Liran Schour <lirans@il.ibm.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-15 15:22:56 +03:00
e23b19309b
Deepstream video backend ( #42424 )
...
Signed-off-by: Viranjan Pagar <vpagar@nvidia.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-07-10 02:23:30 -07:00
alberto and Chauncey
5715fde12c
[Feature][Parser] Support include_reasoning param for non-Harmony models ( #44301 )
...
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-07-10 15:34:02 +08:00
2d814a0082
[kv_offload] Emit tier-owned BlockStored events from FS/OBJ secondary tiers ( #47923 )
...
Signed-off-by: Change72 <changg@nvidia.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-07-10 06:17:23 +03:00
93e2ab7111
Disable dynamic speculative decoding when DP is enabled ( #45963 )
...
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
2026-07-07 12:52:41 +00:00
Harry Mellor
dd94484577
Bump Transformers version to 5.10.4 ( #41359 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-07 05:13:28 -07:00
700e882eab
Add TorchCodec as a video decoding backend ( #46609 )
...
Signed-off-by: Nicolas Hug <contact@nicolas-hug.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn >
2026-07-06 19:58:51 -07:00
86db6c3070
[Frontend] add per-request timing metrics field to response body of Chat/Completions APIs ( #46768 )
...
Signed-off-by: Nicholas Edelman <nedelman@nvidia.com >
Signed-off-by: Simon Mo <simon@inferact.ai >
Co-authored-by: Simon Mo <simon@inferact.ai >
Co-authored-by: GPT-5.5 <noreply@cursor.com >
2026-07-06 17:48:29 -07:00
Zhenzhong Xu and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
d9c1767cd4
[INC][ARK] Direct Register Custom Op for ARK ( #46361 )
...
Signed-off-by: Zhenzhong1 <zhenzhong.xu@intel.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-06 13:45:50 +08:00
Chaojun Zhang and Copilot
1a308c449c
[XPU] Add W8A8 FP8 linear kernel with multi-granularity quant support ( #43645 )
...
Signed-off-by: Chaojun,Zhang <chaojun.zhang@intel.com >
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com >
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
2026-07-04 18:10:01 +08:00
07516fda67
[MRV2][SD] Make Dynamic SD comatible with Full Cuda Graphs ( #45953 )
...
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-03 23:58:27 -04:00
sychen52 and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
67ff0ae30f
Support nvfp4 kv with kv-cache-dtype-skip-layers sliding_window ( #42890 )
...
Signed-off-by: Shiyang Chen <shiychen@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-04 02:29:13 +00:00
3af8789559
[Feature] Universal speculative decoding for heterogeneous vocabularies (TLI) ( #38174 )
...
Signed-off-by: wan-danfeng <wandanfeng0802@gmail.com >
Signed-off-by: Wonderful <wandanfeng0802@gmail.com >
Co-authored-by: Wan_DF <wonderful199082@126.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
2026-07-02 01:34:20 -07:00
bec232a914
Secondary tier implementation for PD disaggregation ( #42285 )
...
Signed-off-by: Liran Schour <lirans@il.ibm.com >
Signed-off-by: liranschour <liranschour@users.noreply.github.com >
Co-authored-by: Or Ozeri <or@ozery.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-06-30 07:51:44 +03:00
MichaelCaoo and Claude
33f50773cb
[Doc] Fix typos, grammar, and broken commands across docs ( #46398 )
...
Signed-off-by: MichaelCaoo <a992033227@163.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-06-23 02:01:22 +00:00
Change72 and Claude
a9f7b2d41c
[feature][kv_offload] Self-describing KV events for OffloadingConnector ( #43468 )
...
Signed-off-by: Change72 <changg@nvidia.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-06-22 07:27:46 +00:00
d14e551a53
[Model] Remove MiniMaxText01, MiniMaxVL01, MiniMaxForCausalLM ( #45993 )
...
Signed-off-by: Xianbao QIAN <xianbao.qian@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-22 15:20:46 +08:00
Samuel Shen
c9135db27c
[Docs] Update stale LMCache examples ( #45762 )
...
Signed-off-by: Samuel Shen <slshen@tensormesh.ai >
2026-06-19 03:21:36 +00:00
Yifan Qiao and Nick Hill
35e4dd4a69
[KV Connector][Mooncake] Async lookup to reduce scheduler overhead ( #45659 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-06-18 21:44:02 +00:00
8d4f54966c
fix(quantization): Fix AWQ dequantize on Intel XPU and refactor AutoAWQ config ( #42727 )
...
Signed-off-by: Alex <alex.tech.lab@outlook.com >
Signed-off-by: AlexHuang <jihuihuang@tencent.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-18 20:12:28 +08:00
Angelo Ruocco
ee0fd6984a
docs, kv_offloading: add docs for selective offload ( #45279 )
...
Signed-off-by: Angelo Ruocco <ang@zurich.ibm.com >
2026-06-17 14:58:00 +03:00
Dao007forever and Claude Opus 4.8
44b2512767
[KV Connector][Mooncake] Add cache_prefix to namespace store keys ( #45767 )
...
Signed-off-by: Dao Le <Dao007forever@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-06-16 10:24:20 -07:00
Flora Feng
cd9078fe59
[Frontend] Skip structural tags for auto tool_choice without strict mode ( #45600 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-06-15 19:55:31 +00:00
4ef4492e9b
[V1][Spec Decode] Add Dynamic SD ( #32374 )
...
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com >
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
2026-06-14 00:14:27 -07:00
Isotr0py
6635279d8a
[Migration] Migrate GGUF quantization support to plugin ( #39612 )
...
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
2026-06-12 12:02:21 -07:00
Sai Sridhar Tarra
a30addc754
[Docs][KV Connector][NIXL] document KV Transfer stat logging and Prometheus metrics ( #44055 )
...
Signed-off-by: Sai Sridhar <tarrasridhar1154@gmail.com >
2026-06-12 15:39:11 +00:00
2043258dec
[Frontend] Support strict mode for tool calling ( #45003 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-12 07:51:48 +00:00
Fynn Schmitt-Ulms and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
ebc6ef971a
Hidden states extraction improvements ( #43805 )
...
Signed-off-by: Fynn Schmitt-Ulms <fschmitt@redhat.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-11 09:44:45 -04:00
89c6a41001
[Bench] Add BFCL dataset for vllm bench serve tool-calling workloads ( #42457 )
...
Signed-off-by: Li Zhang <lzhanga@amazon.com >
Co-authored-by: Li Zhang <lzhanga@amazon.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-06-09 23:59:18 -07:00
Change72
320c52b134
[Bench] benchmark_serving_multi_turn: make non-standard conversation_id payload opt-in ( #43756 )
...
Signed-off-by: Change72 <cguo51@asu.edu >
2026-06-09 19:41:56 -07:00
Ronen Schaffer and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
3d119f78f7
[Docs] Add KV offloading usage guide (single- and multi-tier) ( #44415 )
...
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-09 19:20:23 +03:00
