iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
19,362 Commits 3 Branches 0 Tags
f1899b2ffdf87598350c416a074c8c291c845902
Commit Graph
661 Commits
Author SHA1 Message Date
Connor CarpenterandNick Hill 48fc2e2797 feat(grpc): add KV event source discovery (#50033) 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-07-29 13:46:40 -07:00
Zach Zhu 58f9659397 [Frontend][Core] Standardize request error handling with VLLMError hierarchy (#49665) 
Signed-off-by: Zach Zhu <zzqshu@126.com>
 2026-07-29 04:20:40 +00:00
Nick HillandAndreas Karatzas 6453fc0b8c [Bugfix] Don't reuse engine core payload buffer while zmq is sending it (#50053) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-07-28 16:09:28 +00:00
Itay Alroy 35efdf6b34 [Elastic EP] Async preparation (#47288) 
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
 2026-07-28 05:19:13 -07:00
Shuolei Wang 9069a57139 [Core][Frontend] Add weight version tagging for RL rollouts (#49040) 
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com>
Signed-off-by: Shuolei Wang <948904026@qq.com>
 2026-07-28 14:35:29 +08:00
Connor CarpenterandNick Hill e3c2fc3b3c [Rust Frontend][gRPC] Add server and model discovery (#49491) 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-07-27 09:53:27 -07:00
Nick Hillandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 59a6b0411d [Core] Fix internal LB load-balancing (#49204) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-27 14:00:46 +00:00
Walter Beller-Moralesandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 8040ef2426 [Frontend] expose stream_interval as req sampling param (#49754) 
Signed-off-by: walterbm <walter.beller.morales@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-27 11:30:37 +08:00
Schwinn Saereesitthipitak b5b61c622c [Core][Distributed] Add process-checkpoint lifecycle hooks for communicators (starting with Flashinfer) (#46877) 
Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
 2026-07-26 14:47:50 -04:00
fangyuchuzWaNg3a798347923TianZhuo205150940a798347923w00689259zWaNg3yzchang-plusJade Zheng
 		0b0bd2b5f6 [Feature] Add fault tolerance framework (simplified) for DP+EP external LB deployments (#44428) 
Signed-off-by: fangyuchu <fangyuchu@qq.com>
Signed-off-by: a798347923 <2645302020@qq.com>
Signed-off-by: TianZhuo <2770730562@qq.com>
Signed-off-by: a798347923 <39047817+a798347923@users.noreply.github.com>
Signed-off-by: 205150940 <112750056+205150940@users.noreply.github.com>
Signed-off-by: w00689259 <wangzhuo66@huawei.com>
Signed-off-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com>
Signed-off-by: zWaNg3 <389750525@qq.com>
Signed-off-by: yzchang-plus <1078477584@qq.com>
Signed-off-by: Jade Zheng <zheng.shoujian@outlook.com>
Co-authored-by: zWaNg3 <37772915+zWaNg3@users.noreply.github.com>
Co-authored-by: a798347923 <2645302020@qq.com>
Co-authored-by: TianZhuo <2770730562@qq.com>
Co-authored-by: 205150940 <112750056+205150940@users.noreply.github.com>
Co-authored-by: a798347923 <39047817+a798347923@users.noreply.github.com>
Co-authored-by: w00689259 <wangzhuo66@huawei.com>
Co-authored-by: zWaNg3 <389750525@qq.com>
Co-authored-by: yzchang-plus <1078477584@qq.com>
Co-authored-by: Jade Zheng <zheng.shoujian@outlook.com>
 2026-07-25 11:49:09 -04:00
Junpu Yu 1ad84fea86 [Bugfix][Spec Decode] Select earliest-completing stop string in check_stop_strings (#49391) 
Signed-off-by: Junpu Yu <davidyu@nvidia.com>
 2026-07-23 18:14:06 +08:00
zhrrrandNick Hill 12213c6795 [Bugfix] handle grammar compilation failures to avoid engine crash (#47312) 
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-07-23 18:13:49 +08:00
yzong-rh a287eb163f [Front-end] [Messages] Populate num_cache_creation_tokens (#48535) 
Signed-off-by: Yifan Zong <yzong@redhat.com>
 2026-07-18 13:04:35 -04:00
Aaron HaoandSumanth R Hegde fb1d8ccaf5 [rl] Stateful Trainer Send: New Abstractions [1/N] (#48042) 
Signed-off-by: haoaaron <ahao@anyscale.com>
Signed-off-by: Aaron Hao <ahao@anyscale.com>
Co-authored-by: Sumanth R Hegde <39546518+SumanthRH@users.noreply.github.com>
 2026-07-17 15:11:06 +08:00
Max HuRoger WangShang Wang
 		5a65ba5f17 [Refactor] Move iteration logging to the frontend (#46647) 
Signed-off-by: maxyanghu <hyoung2991@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Shang Wang <shangw@nvidia.com>
 2026-07-15 17:59:05 -07:00
Taneem Ibrahimandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 9182e86971 Log fully resolved pooling config at startup (#48030) 
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-14 22:00:46 +00:00
omerpaz95andOr Ozeri 8df14cfc8c [EC Connector] Add EC Transfer Params (#42433) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-07-12 14:35:33 +03:00
vx120crp0128aoshen02Claude Opus 4.8
 		fc1c548093 Runtime Draft Weight Update for Speculative Decoding (#46725) 
Signed-off-by: vx120 <893600387@qq.com>
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: crp0128 <191679376@qq.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-11 22:51:53 -07:00
Harry Mellor dd94484577 Bump Transformers version to 5.10.4 (#41359) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-07-07 05:13:28 -07:00
Qiming Zhang 2fa10566e3 [Core][DP] Rotate load-balancer tie-break to avoid systematic engine bias (#47420) 
Signed-off-by: mayuyuace <qiming1.zhang@intel.com>
 2026-07-06 07:09:16 +00:00
Aaron HaoandClaude Sonnet 4.6 77a9c5ae28 Weight sync refactor + move sparse nccl engine (#44353) 
Signed-off-by: hao-aaron <ahao@anyscale.com>
Signed-off-by: haoaaron <ahao@anyscale.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-07-01 01:25:19 -07:00
Robert Shaw c5e3c40877 Fix P/D with DP Supervisor (#46628) 
Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
 2026-06-25 13:13:08 -07:00
Ashwin PhadkeandHarry Mellor d511b5bae9 Chore: Fix minor doc sentence, grammar, quote errors (#40469) 
Signed-off-by: Ashwin Phadke <23502062+ashwin-phadke@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-06-24 18:58:24 +00:00
Wentao Ye 1cd3e0e945 [Bug] Fix IndentationError: expected an indented block after 'with' statement (#46627) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-06-24 23:14:17 +08:00
meihancaoshen02Nick Hill
 		93ec645878 [Bugfix] Fix illegal memory access from a forward during a partial wake_up (#44483) 
Signed-off-by: Meihan-chen <zr010426ztt@outlook.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-06-24 22:12:23 +08:00
Tyler Michael SmithClaudeCodexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>kourosh hakhamaneshi
 		ebfbcfe46a Stop setting CUDA_VISIBLE_DEVICES internally in vLLM, add device_ids arg (#45026) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: kourosh hakhamaneshi <kouroshHakha@users.noreply.github.com>
 2026-06-20 13:38:10 -07:00
Sumanth R Hegde 7ff7f5c8eb Revert "Fix Stale Encoder Cache After Weight Update" (#46125) 2026-06-20 07:09:09 -07:00
Ashar 4cb5e746b6 [Rust Frontend]: Add /get_world_size route with static parallel size (#44801) 2026-06-18 13:10:20 +00:00
littlecircle0730 08985351f3 Fix Stale Encoder Cache After Weight Update (#45093) 
Signed-off-by: littlecircle0730 <littlecircle0730@gmail.com>
 2026-06-18 09:32:10 +00:00
Ting SUNandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 7b5d60cc37 [Bugfix][V1] Clean up compiled-model bytecode hooks on VllmRunner exit (#45195) 
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-16 20:31:17 -07:00
Nick Hill d8d95998dc [Core] Add prefill step cadence for better non-PD DP balancing (#44558) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-16 13:17:18 -07:00
yifei wuandHarry Mellor a52205bccf [Model] Add HrmTextForCausalLM (Hierarchical Reasoning Model — Text) (#43098) 
Signed-off-by: Wuyifei <wuyifei@me.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-06-16 22:41:41 +08:00
Ryan Rockandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> aab639c705 [Core][AMD] Propagate shutdown timeout to MultiprocExecutor (#43154) 
Signed-off-by: Ryan Rock <ryan.rock@amd.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-06-12 15:13:31 -05:00
Ethan Feng b7f9b6ab27 [Metrics] Add group-aware KV cache capacity to vllm:cache_config_info (#42206) 
The startup log already reports the correct group-aware KV cache capacity for
hybrid models, but Prometheus did not expose matching info in 'vllm:cache_config_info`.

This PR adds kv_cache_size_tokens and kv_cache_max_concurrency.

Signed-off-by: Ethan Feng <ethan.fengch@gmail.com>
 2026-06-12 11:49:44 +00:00
Lucas WilkinsonMartin KuklaMatthew BonanniDipika SikkaNickLucchejiahancAlec KohlhoffPorras HuangClaudescoootscooob
 		eb28452b10 [Model] Add DiffusionGemma Support (#45163) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Martin Kukla <martin.kukla@cantab.net>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Dipika Sikka <dsikka@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
Co-authored-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
Co-authored-by: Alec Kohlhoff <134344302+aleckohlhoff@users.noreply.github.com>
Co-authored-by: Porras Huang <20535584+porrashuang@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: scoootscooob <167050519+scoootscooob@users.noreply.github.com>
 2026-06-11 22:17:35 -07:00
Walter Beller-Morales c73b0d0db9 [Core][Engine] allow DP ray placement groups to be set on specific nodes (#44669) 
Signed-off-by: walterbm <walter.beller.morales@gmail.com>
 2026-06-05 20:07:47 +00:00
qizixi 96229fa99e [KVConnector][1/N] PP-aware handshake aggregation and intermediate-PP output plumbing (#43720) 
Signed-off-by: zixi-qi <zixi@inferact.ai>
 2026-06-04 22:04:19 -07:00
Wentao Ye 41a4829f22 [Logs Refactor] Optimize shutdown logs, easier to follow and consistent (#43707) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-06-04 14:36:32 -04:00
Mengqing Caozjy0516mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		0c6631f02a [KVCache] Support Pluggable KVCacheSpec (#37505) 
Signed-off-by: MengqingCao <cmq0113@163.com>
Signed-off-by: Mengqing Cao <cmq0113@163.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-03 09:05:16 -07:00
Nick Hill e15f20258b [ModelRunnerV2] Avoid pipeline parallel bubbles (#42187) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-02 14:02:01 -07:00
Nick Hill cab5c9a2a9 [Core] Move max_concurrent_batches to VllmConfig (#44274) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-02 08:57:25 -07:00
grunerAmit GrunerJiangyun Zhu
 		654bd2bca4 [Bugfix] Sync block_size from EngineCore to frontend for hybrid Mamba… (#42967) 
Signed-off-by: Amit Gruner <agruner@crusoe.ai>
Co-authored-by: Amit Gruner <agruner@crusoe.ai>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
 2026-06-02 13:41:00 +00:00
czhu-cohere 8b9deeec4b [Bugfix] Fix Ray placement group allocation with grouped nodes (#43998) 
Signed-off-by: <conway.zhu@cohere.com>
Signed-off-by: root <conway.zhu@cohere.com>
 2026-05-29 12:51:05 -06:00
Bugen Zhao 1521173c17 [Rust Frontend] Add /version endpoint using engine-reported value (#43854) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-05-29 00:32:27 +00:00
Vadim Gimpelson c1c4db8b4b Log dummy DP step in iteration details (#41406) 
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Signed-off-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com>
 2026-05-28 12:18:39 +00:00
Wei Zhao f2caefe226 [UX] Increase DP Coordinator startup timeout from 30s to 120s (#42343) 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
 2026-05-28 03:31:45 -07:00
Nick Hill 626fa9bba5 [BugFix] Fix blocked reasoning parsing with MRV2 (#43808) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-05-28 04:59:34 +00:00
aoshen02aoshen524Claude Opus 4.7
 		1fc2cee50a [KVConnector][Mooncake] Wire reset_cache cascade end-to-end (#42694) 
Signed-off-by: aoshen524 <aoshen524@gmail.com>
Signed-off-by: Ao Shen <aoshen@inferact.ai>
Co-authored-by: aoshen524 <aoshen524@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 20:52:35 -07:00
Vadim Gimpelsonandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 812e7e7364 [Bugfix][V1] Fix TOCTOU race causing intermittent EADDRINUSE on multi-API-server DP startup (#42585) 
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Signed-off-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-26 14:06:00 -07:00
Gabriel Wu 82536acc54 Keep scheduler alive for delayed KV connector frees (#43433) 
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
 2026-05-23 06:23:32 +00:00