Qiming Zhang
f5e441de10
[Bugfix][Test] Fix off-by-one error in sampled token rank causing flaky logprobs test ( #53976 )
...
Signed-off-by: mayuyuace <qiming1.zhang@intel.com >
2026-08-31 18:49:43 +00:00
Nicolò Lucchesi
dafbef15a1
[Core] Add max_num_queued_reqs and max_num_queued_tokens for queue size management ( #49445 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-08-31 14:26:49 +00:00
Oxana Korzh and Claude Opus 5
fb68025138
[Bugfix][MRV2] Release model and KV cache on in-process engine shutdown ( #54162 )
...
Signed-off-by: Oxana Korzh <okorzh@amd.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-28 22:09:08 -07:00
aoshen02
17da48596c
[Bugfix][DP] Synchronize the device on pause completion ( #52914 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
2026-08-26 09:50:33 -07:00
Tony
903a02192f
[Bugfix] Handle empty FlatLogprobs slices and delta output ( #53704 )
...
Signed-off-by: tony <864832769@qq.com >
2026-08-26 21:00:54 +08:00
Naveenraj Kamalakannan and Benjamin Chislett
d5cadcee86
[Feature][DSpark]: Logprobs adaptive verification ( #52242 )
...
Signed-off-by: Naveenraj Kamalakannan <therealnaveenkamal@gmail.com >
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
2026-08-25 10:22:33 -04:00
Lucas Wilkinson
8bdc70ec7b
[6/N][KV-Cache Layout Refactor] Standardize KV cache layout ( #51718 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
2026-08-21 17:47:49 -07:00
a1c5b1fd9f
[Core][V1] Support trace_decode_token_ids for deterministic decode replay ( #46701 )
...
Signed-off-by: ianzxliu <ianzxliu@tencent.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Zhixia Liu <liuzhixia.nju@gmail.com >
Co-authored-by: ianzxliu <ianzxliu@tencent.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-20 15:12:43 +08:00
76fb6d210a
[Bugfix][Core] Reserve the KV null block when validating max_model_len ( #47272 )
...
Signed-off-by: hyungjun oh <92hyungjun@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-20 02:49:23 +00:00
lxy and “新颐”
0c8c3f41cf
[Bugfix][V1] Sync mamba_block_size via EngineCoreReadyResponse ( #50809 )
...
Signed-off-by: “新颐” <lxy537258@antgroup.com >
Co-authored-by: “新颐” <lxy537258@antgroup.com >
2026-08-19 16:44:36 +00:00
3a386cfaf5
[ROCm] Give EngineCore cleanup grace after request abort ( #52281 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-19 11:39:59 -05:00
Connor Carpenter and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f08a95f8d8
[Rust Frontend][gRPC] Advertise LoRA capabilities ( #52031 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-17 13:27:56 -07:00
Tyler Michael Smith and OpenAI Codex
017e9f4448
Promote prefix_cache_retention_interval to an argument and change the default to 0 ( #52216 )
...
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-17 13:20:50 +00:00
Connor Carpenter and Nick Hill
f473870ecf
[Rust Frontend][gRPC] Add RL lifecycle control ( #51316 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-14 11:32:39 -07:00
Ali Jaseem and Claude
64ca614fe4
[Bugfix] Fix --data-parallel-start-rank 0 being treated as unset in create_engine_config ( #47692 )
...
Signed-off-by: Syed Ali Jaseem <syedali0057@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-08-13 15:12:26 +00:00
Wentao Ye
4988df2eb8
[Config] Update default _max_num_batched_tokens from 8192 to 16384 ( #51726 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-08-11 11:09:34 -04:00
7b9f2dad89
[Frontend] Watch frontend processes during engine startup ( #43417 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-06 11:17:24 -07:00
Karen Chung
f57123aa2d
feat(frontend): session id plumbing into requests ( #48048 )
...
Signed-off-by: Karen Chung <karenc@nvidia.com >
2026-08-03 22:18:58 +00:00
Ning Xie
726ef437a1
[chore] delete useless code ( #49424 )
...
Signed-off-by: Andy Xie <andy.xning@gmail.com >
2026-07-31 11:54:08 -07:00
Zach Zhu
58f9659397
[Frontend][Core] Standardize request error handling with VLLMError hierarchy ( #49665 )
...
Signed-off-by: Zach Zhu <zzqshu@126.com >
2026-07-29 04:20:40 +00:00
Connor Carpenter and Nick Hill
e3c2fc3b3c
[Rust Frontend][gRPC] Add server and model discovery ( #49491 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-27 09:53:27 -07:00
Nick Hill and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
59a6b0411d
[Core] Fix internal LB load-balancing ( #49204 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-27 14:00:46 +00:00
Walter Beller-Morales and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
8040ef2426
[Frontend] expose stream_interval as req sampling param ( #49754 )
...
Signed-off-by: walterbm <walter.beller.morales@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-27 11:30:37 +08:00
5a65ba5f17
[Refactor] Move iteration logging to the frontend ( #46647 )
...
Signed-off-by: maxyanghu <hyoung2991@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Shang Wang <shangw@nvidia.com >
2026-07-15 17:59:05 -07:00
Harry Mellor
dd94484577
Bump Transformers version to 5.10.4 ( #41359 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-07 05:13:28 -07:00
wang.yuqi
31ca9504b1
[Frontend] Split ServingRender into renderer and entrypoint. ( #44285 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-23 11:19:09 +00:00
Ashar
4cb5e746b6
[Rust Frontend]: Add /get_world_size route with static parallel size ( #44801 )
2026-06-18 13:10:20 +00:00
Ryan Rock and gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
aab639c705
[Core][AMD] Propagate shutdown timeout to MultiprocExecutor ( #43154 )
...
Signed-off-by: Ryan Rock <ryan.rock@amd.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-06-12 15:13:31 -05:00
Walter Beller-Morales
c73b0d0db9
[Core][Engine] allow DP ray placement groups to be set on specific nodes ( #44669 )
...
Signed-off-by: walterbm <walter.beller.morales@gmail.com >
2026-06-05 20:07:47 +00:00
Nick Hill and yewentao256
da107a59e5
[MRV2] Also enable MRV2 for Llama and Mistral dense models ( #43458 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Co-authored-by: yewentao256 <zhyanwentao@126.com >
2026-06-02 11:18:46 -07:00
Nick Hill
cab5c9a2a9
[Core] Move max_concurrent_batches to VllmConfig ( #44274 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-06-02 08:57:25 -07:00
654bd2bca4
[Bugfix] Sync block_size from EngineCore to frontend for hybrid Mamba… ( #42967 )
...
Signed-off-by: Amit Gruner <agruner@crusoe.ai >
Co-authored-by: Amit Gruner <agruner@crusoe.ai >
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com >
2026-06-02 13:41:00 +00:00
Vadim Gimpelson
5d126dd155
[Bugfix] Exclude Ray DP from #42585 's deferred port allocation ( #43864 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
2026-05-28 15:55:14 +00:00
Ashwin Giridharan and abinggo
84e351555a
[Bugfix] Auto-raise max_num_batched_tokens for prefix-LM multimodal models ( #43051 )
...
Signed-off-by: Ashwin Giridharan <girida@amazon.com >
Co-authored-by: abinggo <107740309+abinggo@users.noreply.github.com >
2026-05-22 21:23:50 -07:00
liuzhenwei
07a40ede19
[UT][XPU] fix test_parallel_sampling due to global random state ( #42388 )
...
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com >
2026-05-12 18:03:23 +08:00
d7af6b34d8
[Model Runner V2] Bug fix: logprob dtype int64/int32 issue ( #41761 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-11 21:55:43 +00:00
shaharmor98 and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f9f770ca0b
fix nixl side-channel host selection ( #41806 )
...
Signed-off-by: Shahar Mor <smor@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-11 07:40:37 +00:00
Wentao Ye
ea0e501bb1
[KV Connector] Remove compat support for pre-v0.12.0 constructor signatures without KVCacheConfig ( #39832 )
...
The v0.12.0 release contained initial support for HMA in KV Connectors. As part
of these changes, a KVCacheConfig argument was added to KV connector
constructors. Backwards compatibility support for out-of-tree connectors was
included in this change, with a very prominent warning. See #25712 and #27887 .
Since the warning has been around for over 5 months, we can safely remove
the support of it.
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-05-09 23:39:46 +00:00
Luis 🚀 and Andrew Sansom
14043dfecd
feat: Enable prompt_embeds Content Part Support in vLLM Chat Completions API ( #40720 )
...
Signed-off-by: Luis Robaina <luis@protopia.ai >
Signed-off-by: Luis Robaina 🚀 <luisfabian1545@gmail.com >
Signed-off-by: LuisRobaina <luis@protopia.ai >
Co-authored-by: Andrew Sansom <qthequartermasterman@gmail.com >
2026-05-01 10:05:55 +08:00
Mark McLoughlin and Or Ozeri
d3af8c1831
[Core][Metrics][BugFix] Replace num_cached_tokens/num_external_computed_tokens with PrefillStats ( #37460 )
...
Related to `Counters can only be incremented by non-negative amounts`
error with the `vllm:prompt_tokens_by_source_total` metric.
Signed-off-by: Mark McLoughlin <markmc@redhat.com >
Co-authored-by: Or Ozeri <or@ozery.com >
2026-04-14 09:00:45 +01:00
Martin Hickey and Or Ozeri
cc07dad789
[HMA] [KVEvent] Enable GPU-side KV events for HMA ( #37688 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
Co-authored-by: Or Ozeri <or@ozery.com >
2026-04-12 10:01:02 +03:00
wang.yuqi
66c079ae83
[Frontend][4/n] Improve pooling entrypoints | pooling. ( #39153 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-04-09 10:09:45 +00:00
triangleXIV and mgoin
7c94ae16c6
[BugFix] --max-model-len=-1 causes over-limit requests to hang and starve the entire service ( #39102 )
...
Signed-off-by: triangle14 <y1019026570@gmail.com >
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mgoin <mgoin64@gmail.com >
2026-04-08 14:03:17 -07:00
yzong-rh
6dad4c5722
[Test] Fix flaky race condition in test_abort_final_step ( #38414 )
...
Signed-off-by: Yifan <yzong@redhat.com >
2026-03-28 09:06:56 +00:00
d5af196c18
[2/N] Elastic EP Milestone 2: Integrating NIXL-EP ( #35627 )
...
Signed-off-by: Itay Alroy <ialroy@nvidia.com >
Co-authored-by: Yongji Wu <wuyongji317@gmail.com >
Co-authored-by: Ron Tourgeman <rtourgeman@nvidia.com >
2026-03-13 09:25:33 -04:00
Sage
a2268617cf
[Frontend] Delegate preprocessing to OpenAIServingRender ( #36483 )
...
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com >
2026-03-13 00:39:43 -07:00
Wentao Ye
7279374f91
[Perf] Compute maxsim in worker side, reducing redundant copies, 2.7% E2E throughput improvement ( #36159 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-03-09 20:55:58 -07:00
lif and mcelrath
00b814ba5a
[V0 Deprecation] Remove unused swap_space parameter ( #36216 )
...
Signed-off-by: majiayu000 <1835304752@qq.com >
Co-authored-by: mcelrath
2026-03-07 22:09:55 +08:00
Ning Xie
176c799f4c
[openai api] log exception in exception handler (1/N) ( #31164 )
...
Signed-off-by: Andy Xie <andy.xning@gmail.com >
2026-03-05 16:00:12 +00:00
Nick Hill
dbf0da817a
[Core] Cleanup engine pause/sleep logic ( #34528 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-02-24 19:33:34 -08:00
Qiming Zhang