iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
21,963 Commits 3 Branches 0 Tags
38cc054fc2269941cd35ecdfd198aacbb2eabfb8
Commit Graph
230 Commits
Author SHA1 Message Date
Mark McLoughlinandAI Assistant 45d56c0baf [Core] Model console logging as CLI configuration (#57205) 
Add `--logging-config` CLI argument which can be supplied as
JSON or using dotted arguments. The `--log-level` argument
is provided for convenience, and `--log-config-file` is deprecated
in favor of `--logging-config.pylogging_config_file`.

Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: AI Assistant <noreply@openai.com>
 2026-09-25 11:36:59 +01:00
Matt MastracciClaude Fable 5.1Lucas WilkinsonCodexRazorback16
 		1b3b88ec2b [Core] structured generation mode for DiffusionGemma model (Jev-like) (#57250) 
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Razorback16 <razorback16@protonmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Razorback16 <razorback16@protonmail.com>
 2026-09-22 20:51:17 +00:00
Tony Lin dc479a2d83 [Bugfix] Fix external LB DP rank handling when replicas share nodes (#53743) 
Signed-off-by: Tony Lin <tony.lin@intel.com>
 2026-09-21 17:19:14 +02:00
Hongxin Xuaoshen02OpenAI CodexRoger WangNick HillCodex
 		0b7f11a1ee [AuxOutput] Add block-keyed storage for routed-expert outputs (#45635) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-21 10:25:05 +00:00
czhu-cohere 3918f3c5a3 [Profiler] Unify platform-aware torch profiling (#57460) 2026-09-20 19:17:23 -07:00
David CheungClaudeNick Hill
 		5302d1fe40 [Model Runner V2] Support custom logits processors (#56497) 
Signed-off-by: David Cheung <d7cheung@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-09-19 17:46:49 +00:00
hangangqiangmukeshbhandarkaraoshen02Claude Fable 5.1aoshen02
 		90531daf6d [Core][Frontend] Bind KV-event publishers at port 0 and expose the bound endpoints (#55844) 
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: mukeshbhandarkar <shell00035@gmail.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
 2026-09-18 18:11:05 -07:00
Nicolò Lucchesi a7156060c1 [Core] Make parallel sampling (n>1) reqs admission atomic (#53936) 
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
 2026-09-18 18:08:44 +02:00
Dakai AnAlanFokCoaoshen02OpenAI Codex
 		092bdd6d57 [Frontend][Core] Add release_kv_cache_memory() API (#44890) 
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: AlanFokCo <alanfok2868@gmail.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Dakai An <77474977+andakai@users.noreply.github.com>
Co-authored-by: AlanFokCo <alanfok2868@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-09-18 07:12:12 +08:00
Wei Zhao ccde9c61e9 [Bugfix] Restore KV cache metadata GET method for external event consumer (#56925) 
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
 2026-09-17 23:01:35 +00:00
Tyler Michael SmithOpenAI CodexOpenAI CodexMisha Goinmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		4f9a41ac94 [CI/Build][Hardware][NVIDIA] Add Rubin CUDA 13.4 nightly images (#55953) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-17 08:05:31 -07:00
Oxana KorzhandAndreas Karatzas 2307a1d16c [Core] Don't issue a blocking collective RPC during engine handshake (#57226) 
Signed-off-by: okorzh <okorzh@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-09-16 21:30:59 -05:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 16:44:29 +00:00
Robert ShawandClaude 42919b49c5 [P/D] Report prefill worker cache hits in prompt_tokens_details (#54222) 
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-09-16 15:23:31 +00:00
JulienDarveBugen ZhaoOpenAI Codex
 		f37c550bf6 [Feature][Frontend] Expose effective attention block size for DCP (#56538) 
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: Julien Darve <jdarve@NVIDIA.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-09-16 08:29:43 +00:00
Chauncey f0fbc98c48 [Frontend] Share max_num_queued_reqs across API server processes (#54746) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-09-15 14:50:26 +00:00
djramicandAndreas Karatzas c6ba5e89d6 [Bugfix][Elastic EP] Route only to surviving engines during scale-down (#55772) 
Signed-off-by: Djordje Ramic <djoramic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-09-08 21:28:28 -05:00
Wei Zhao c3ec0d29f5 [Bugfix] Fix cuda profiler missing bug (#55237) 2026-09-07 03:15:57 +00:00
aoshen02 039ea82660 [Core] Sync DP state on the first step of a wave (#52957) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
 2026-09-06 02:16:58 +00:00
Qiming Zhang f5e441de10 [Bugfix][Test] Fix off-by-one error in sampled token rank causing flaky logprobs test (#53976) 
Signed-off-by: mayuyuace <qiming1.zhang@intel.com>
 2026-08-31 18:49:43 +00:00
Nicolò Lucchesi dafbef15a1 [Core] Add max_num_queued_reqs and max_num_queued_tokens for queue size management (#49445) 
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
 2026-08-31 14:26:49 +00:00
Oxana KorzhandClaude Opus 5 fb68025138 [Bugfix][MRV2] Release model and KV cache on in-process engine shutdown (#54162) 
Signed-off-by: Oxana Korzh <okorzh@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-28 22:09:08 -07:00
aoshen02 17da48596c [Bugfix][DP] Synchronize the device on pause completion (#52914) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
 2026-08-26 09:50:33 -07:00
Tony 903a02192f [Bugfix] Handle empty FlatLogprobs slices and delta output (#53704) 
Signed-off-by: tony <864832769@qq.com>
 2026-08-26 21:00:54 +08:00
Naveenraj KamalakannanandBenjamin Chislett d5cadcee86 [Feature][DSpark]: Logprobs adaptive verification (#52242) 
Signed-off-by: Naveenraj Kamalakannan <therealnaveenkamal@gmail.com>
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com>
 2026-08-25 10:22:33 -04:00
Lucas Wilkinson 8bdc70ec7b [6/N][KV-Cache Layout Refactor] Standardize KV cache layout (#51718) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-08-21 17:47:49 -07:00
Zhixia LiuianzxliuClaudeaoshen02CodexNick Hill
 		a1c5b1fd9f [Core][V1] Support trace_decode_token_ids for deterministic decode replay (#46701) 
Signed-off-by: ianzxliu <ianzxliu@tencent.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Zhixia Liu <liuzhixia.nju@gmail.com>
Co-authored-by: ianzxliu <ianzxliu@tencent.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-08-20 15:12:43 +08:00
92hyungjunClaudeNick Hill
 		76fb6d210a [Bugfix][Core] Reserve the KV null block when validating max_model_len (#47272) 
Signed-off-by: hyungjun oh <92hyungjun@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-08-20 02:49:23 +00:00
lxyand“新颐” 0c8c3f41cf [Bugfix][V1] Sync mamba_block_size via EngineCoreReadyResponse (#50809) 
Signed-off-by: “新颐” <lxy537258@antgroup.com>
Co-authored-by: “新颐” <lxy537258@antgroup.com>
 2026-08-19 16:44:36 +00:00
Andreas KaratzasOpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		3a386cfaf5 [ROCm] Give EngineCore cleanup grace after request abort (#52281) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-08-19 11:39:59 -05:00
Connor Carpenterandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> f08a95f8d8 [Rust Frontend][gRPC] Advertise LoRA capabilities (#52031) 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-08-17 13:27:56 -07:00
Tyler Michael SmithandOpenAI Codex 017e9f4448 Promote prefix_cache_retention_interval to an argument and change the default to 0 (#52216) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-17 13:20:50 +00:00
Connor CarpenterandNick Hill f473870ecf [Rust Frontend][gRPC] Add RL lifecycle control (#51316) 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-08-14 11:32:39 -07:00
Ali JaseemandClaude 64ca614fe4 [Bugfix] Fix --data-parallel-start-rank 0 being treated as unset in create_engine_config (#47692) 
Signed-off-by: Syed Ali Jaseem <syedali0057@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-08-13 15:12:26 +00:00
Wentao Ye 4988df2eb8 [Config] Update default _max_num_batched_tokens from 8192 to 16384 (#51726) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-08-11 11:09:34 -04:00
Bugen ZhaoOpenAI CodexNick Hill
 		7b9f2dad89 [Frontend] Watch frontend processes during engine startup (#43417) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-08-06 11:17:24 -07:00
Karen Chung f57123aa2d feat(frontend): session id plumbing into requests (#48048) 
Signed-off-by: Karen Chung <karenc@nvidia.com>
 2026-08-03 22:18:58 +00:00
Ning Xie 726ef437a1 [chore] delete useless code (#49424) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2026-07-31 11:54:08 -07:00
Zach Zhu 58f9659397 [Frontend][Core] Standardize request error handling with VLLMError hierarchy (#49665) 
Signed-off-by: Zach Zhu <zzqshu@126.com>
 2026-07-29 04:20:40 +00:00
Connor CarpenterandNick Hill e3c2fc3b3c [Rust Frontend][gRPC] Add server and model discovery (#49491) 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-07-27 09:53:27 -07:00
Nick Hillandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 59a6b0411d [Core] Fix internal LB load-balancing (#49204) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-27 14:00:46 +00:00
Walter Beller-Moralesandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 8040ef2426 [Frontend] expose stream_interval as req sampling param (#49754) 
Signed-off-by: walterbm <walter.beller.morales@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-27 11:30:37 +08:00
Max HuRoger WangShang Wang
 		5a65ba5f17 [Refactor] Move iteration logging to the frontend (#46647) 
Signed-off-by: maxyanghu <hyoung2991@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Shang Wang <shangw@nvidia.com>
 2026-07-15 17:59:05 -07:00
Harry Mellor dd94484577 Bump Transformers version to 5.10.4 (#41359) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-07-07 05:13:28 -07:00
wang.yuqi 31ca9504b1 [Frontend] Split ServingRender into renderer and entrypoint. (#44285) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-06-23 11:19:09 +00:00
Ashar 4cb5e746b6 [Rust Frontend]: Add /get_world_size route with static parallel size (#44801) 2026-06-18 13:10:20 +00:00
Ryan Rockandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> aab639c705 [Core][AMD] Propagate shutdown timeout to MultiprocExecutor (#43154) 
Signed-off-by: Ryan Rock <ryan.rock@amd.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-06-12 15:13:31 -05:00
Walter Beller-Morales c73b0d0db9 [Core][Engine] allow DP ray placement groups to be set on specific nodes (#44669) 
Signed-off-by: walterbm <walter.beller.morales@gmail.com>
 2026-06-05 20:07:47 +00:00
Nick Hillandyewentao256 da107a59e5 [MRV2] Also enable MRV2 for Llama and Mistral dense models (#43458) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: yewentao256 <zhyanwentao@126.com>
 2026-06-02 11:18:46 -07:00