iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
20,626 Commits 3 Branches 0 Tags
65ce85fcdcd79770eeb8cd5eeb4e4ba06120bf6d
Commit Graph
205 Commits
Author SHA1 Message Date
wang.yuqiandCyrus Leung eeb549a74d [Frontend] Move engine/protocol.py out openai folder (#54492) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2026-08-31 10:13:50 +00:00
Connor Carpenter 7a67941c1d [Rust Frontend][gRPC] Add audio and video media inputs (#53760) 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
 2026-08-30 11:31:37 +08:00
Cheng JiangchengvjiangJee Jee LiBugen ZhaorussellfengJee Jee Li
 		b2f685834a [Hy4] support Hy4-preview model (#54160) 
Signed-off-by: chengvjiang <chengvjiang@tencent.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: chengvjiang <chengvjiang@tencent.com>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: russellfeng <russellfeng@tencent.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
 2026-08-29 12:07:28 +08:00
RoachZhaoClaude Opus 5Bugen ZhaoOpenAI Codex
 		4c6c9d569f [Rust Frontend] Take the raw buffer in mm tensor lowering when possible (#53528) 
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: roachsinai <roachsinai@qq.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-08-29 01:05:14 +00:00
Pranav ThakurandBugen Zhao 085e9bb07e [Rust Frontend] Add support for truncate_prompt_tokens and truncation_side (#48584) 
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: pranavthakur0-0 <pranavthakurdew@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-08-29 00:16:37 +00:00
ReidBugen ZhaoOpenAI Codex
 		74850f9f66 [Rust Frontend] Optimize SSE streaming hot path (#51321) 
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: reidliu41 <reid201711@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-08-28 20:35:38 +00:00
Bugen Zhao 11012d2a35 [Rust Frontend] Reduce copy in auxiliary frame resolution (#54148) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-08-28 09:01:17 -07:00
Connor CarpenterandBugen Zhao 3cc015acd2 [Rust Frontend][gRPC] Enforce LoRA path validation across transports (#53756) 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
 2026-08-27 16:00:12 -07:00
Bugen ZhaoandNick Hill 9db222c6be [Rust Frontend] Align OpenAI request and response edge cases (#53218) 
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-08-27 19:23:38 +00:00
Jung JiyuTyler Michael SmithOpenAI Codex
 		3fd72ffd7b [Bugfix][Rust Frontend] Fix LogprobsTensors wire schema mismatch (#53939) 
Signed-off-by: jungjiyu <libraryofjiyu@gmail.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-26 16:16:43 -07:00
Jiahao LiangandChauncey 59d7fc92ea [Bugfix][DeepSeek V4] Handle trailing system messages in prompt rendering (#51262) 
Signed-off-by: jiahaoliang <gzliangjiahao@gmail.com>
Signed-off-by: Jiahao Liang <gzliangjiahao@gmail.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
 2026-08-25 07:49:44 +00:00
Connor CarpenterBugen Zhaomergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		1f9444a34f [Rust Frontend][gRPC] Add LoRA lifecycle control (#52840) 
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-08-25 04:13:19 +00:00
Biswa Panda 9dba2c9b37 [Rust Frontend][RL]: report engine world size over gRPC (#53204) 
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
 2026-08-23 02:31:23 +00:00
Taneem Ibrahim a014e35f38 [Pooling] Fix Rust pooling endpoint for benchmark (#53352) 
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
 2026-08-22 15:45:09 -04:00
Taneem Ibrahimandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 704f12aa1e [Pooling] Report input throughput for batched requests (#53213) 
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-08-22 09:01:40 -04:00
Lucas Wilkinson 8bdc70ec7b [6/N][KV-Cache Layout Refactor] Standardize KV cache layout (#51718) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-08-21 17:47:49 -07:00
Bugen ZhaoandOpenAI Codex 08b73d07cc [Frontend] Prevent Kimi K3 reserved markers in response text (#52889) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-21 09:06:52 -07:00
Roy Wang df344bf36f [Rust][Benchmark] Load HF datasets from parquet shards via hf-hub, fixing truncated-cache sampling (#51570) 2026-08-21 05:07:43 +00:00
ReidandBugen Zhao 2adf4b9e2a [Rust Frontend] Fix Kimi K3 reasoning_effort="none" handling (#53043) 
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: reidliu41 <reid201711@gmail.com>
 2026-08-21 02:47:07 +00:00
Bugen Zhao df3b3422b4 [Rust Frontend] Add HY3 unified parser and local XGrammar structural-tag builder (#53054) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-08-20 19:13:54 -07:00
Bugen ZhaoOpenAI CodexNick Hill
 		83c5d59209 [Rust Frontend] Replace external protoc with pure Rust lib protox (#52892) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-08-20 18:43:51 -07:00
Bugen Zhao f32b17b6d6 [Rust Frontend] Support --generation-config vllm (#53044) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-08-20 17:31:25 -07:00
Matthew KotilaReed MeyersonClaude Opus 4.8
 		7cfb97e337 [Frontend][Core][Spec Decode] Per-request acceptance stats in OpenAI API responses (#48915) 
Signed-off-by: Matthew Kotila <7692737+matthewkotila@users.noreply.github.com>
Co-authored-by: Reed Meyerson <31574681+reed-meyerson@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-08-20 15:42:49 -07:00
Sagemergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Bugen Zhao
 		727274a75b [Rust Frontend] Fix Qwen parser auto-detection (#51169) 
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
 2026-08-20 09:39:05 +00:00
Roy WangClaude Fable 5Bugen Zhao
 		963fcfa48c [Rust][Benchmark] Align speed-bench CLI flags with Python and add flag parity test (#51592) 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: esmeetu <jasonailu87@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-08-20 08:50:51 +00:00
Quentin GallouédecandBugen Zhao 16cfe728d8 [Bugfix][Rust Frontend] Reject n > 1 in the /inference/v1/generate route (#52844) 
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Quentin Gallouédec <quentin.gallouedec@huggingface.co>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-08-20 07:41:06 +00:00
Tyler Michael SmithandOpenAI Codex aabc1a0a0b [Bugfix][Benchmark] Check readiness before tokenizer init in rust vllm-bench (#51863) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-19 02:52:25 +00:00
Biswa Pandaandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 6a391a931a [Rust Frontend][RL] add routed expert prompt offset (#52703) 
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-08-18 20:04:28 +00:00
Connor CarpenterandBugen Zhao d75136c030 [Rust Frontend] Wait for all utility calls to finish (#52671) 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
 2026-08-18 11:23:34 -07:00
Bugen ZhaoandNick Hill 41f179b57a [Rust Frontend] Simplify data-parallel size ownership (#52575) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-08-18 09:06:05 +00:00
Bugen ZhaoNick HillAndreas Karatzas
 		c89d692cb5 [Build] Propagate vLLM version to Rust binaries (#52593) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-08-17 23:37:37 -07:00
Woosuk KwonBugen ZhaoOpenAI Codex
 		e0e5a7fb28 [Rust Frontend] Fix GLM-5.2 chat template rendering parity (#51426) 
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-08-18 03:16:54 +00:00
Connor Carpenterandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> f08a95f8d8 [Rust Frontend][gRPC] Advertise LoRA capabilities (#52031) 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-08-17 13:27:56 -07:00
Biswa Panda 42156466db [Rust Frontend][gRPC] Preserve skip_special_tokens decoding option (#52384) 
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
 2026-08-15 08:54:58 +00:00
Connor CarpenterandNick Hill f473870ecf [Rust Frontend][gRPC] Add RL lifecycle control (#51316) 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-08-14 11:32:39 -07:00
vx120aoshen02CodexClaude Opus 4.6OpenAI Codex
 		50ba4bc6b2 [Feature] Mask Replay (#49577) 
Signed-off-by: vx120 <893600387@qq.com>
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-13 08:44:31 +00:00
WillZZZyCyrus LeungClaude
 		3962042304 [Bugfix][V1][Multimodal] Recover from P0/P1 processor cache drift (#46747) (#46747) 
Signed-off-by: William Zhao <williamzh@meta.com>
Signed-off-by: WillZZZy <93454470+WillZZZy@users.noreply.github.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-08-12 12:06:57 +08:00
Clinton ThomasLucas Bourtoulemergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		b64a2708b0 Bound generation inputs before expensive work (#51447) 
Signed-off-by: Clinton Thomas <1033162+KernelClint@users.noreply.github.com>
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-08-11 22:03:50 +08:00
aoshen02Claude Opus 4.6mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		33584901ba [Frontend] Add content_parts to /inference/v1/generate for raw multim… (#51478) 
Signed-off-by: Allen Shen <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-08-11 10:00:12 +08:00
Bugen Zhaoandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> f1e921b6d6 [Rust Frontend] Support dynamic tools from developer messages (#51144) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-08-10 18:30:38 -07:00
Bugen Zhaoandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> d8c70f2243 [Rust Frontend] Upgrade MiniJinja to 2.22 & remove method lookup workaround (#51235) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-08-11 00:23:24 +00:00
Connor Carpenterandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> fa722b9f01 [Rust Frontend][gRPC] Add explicit data-parallel rank routing (#51178) 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-08-10 15:05:24 -07:00
Connor Carpenterandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 635dd6aae6 [Build][gRPC] Publish protobuf schemas to Buf (#51276) 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-08-10 13:34:39 -07:00
Sage 865781e627 [Rust Frontend] Add standalone Rust renderer (#50289) 2026-08-06 11:23:26 +00:00
Chauncey 22170354d8 [Feature] Parse request priority from HTTP header (#51089) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-08-05 21:10:29 -07:00
SageandBugen Zhao 5bf32605a9 [Rust Frontend] Deduplicate request preprocessing for /tokenize (#50448) 
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-08-05 08:27:35 +00:00
Bugen Zhao 7743486190 [Frontend] DeepSeek V4 0731 reasoning effort prompts & mappings (#50580) 2026-08-04 08:51:32 -07:00
Connor CarpenterandBugen Zhao e98a8774cb [Rust Frontend][gRPC] Add multimodal image inference (#50368) 
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-08-04 09:34:31 +00:00
Bugen Zhaoandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 0b1c151cbc [Rust Frontend] Align tool rendering for Kimi K3 (#50540) 
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
 2026-08-04 08:03:46 +00:00
Reid 72cd5424da [Rust][Benchmark] Preserve UTF-8 across benchmark stream chunks (#50868) 
Signed-off-by: reidliu41 <reid201711@gmail.com>
 2026-08-04 06:15:02 +00:00