Pranav Thakur and Bugen Zhao
085e9bb07e
[Rust Frontend] Add support for truncate_prompt_tokens and truncation_side ( #48584 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: pranavthakur0-0 <pranavthakurdew@gmail.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-08-29 00:16:37 +00:00
74850f9f66
[Rust Frontend] Optimize SSE streaming hot path ( #51321 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Signed-off-by: reidliu41 <reid201711@gmail.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-08-28 20:35:38 +00:00
Connor Carpenter and Bugen Zhao
3cc015acd2
[Rust Frontend][gRPC] Enforce LoRA path validation across transports ( #53756 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
2026-08-27 16:00:12 -07:00
Bugen Zhao and Nick Hill
9db222c6be
[Rust Frontend] Align OpenAI request and response edge cases ( #53218 )
...
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-08-27 19:23:38 +00:00
1f9444a34f
[Rust Frontend][gRPC] Add LoRA lifecycle control ( #52840 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-08-25 04:13:19 +00:00
Biswa Panda
9dba2c9b37
[Rust Frontend][RL]: report engine world size over gRPC ( #53204 )
...
Signed-off-by: Biswa Panda <biswa.panda@gmail.com >
2026-08-23 02:31:23 +00:00
83c5d59209
[Rust Frontend] Replace external protoc with pure Rust lib protox ( #52892 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-20 18:43:51 -07:00
Bugen Zhao
f32b17b6d6
[Rust Frontend] Support --generation-config vllm ( #53044 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-08-20 17:31:25 -07:00
Quentin Gallouédec and Bugen Zhao
16cfe728d8
[Bugfix][Rust Frontend] Reject n > 1 in the /inference/v1/generate route ( #52844 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Quentin Gallouédec <quentin.gallouedec@huggingface.co >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-08-20 07:41:06 +00:00
Bugen Zhao and Nick Hill
41f179b57a
[Rust Frontend] Simplify data-parallel size ownership ( #52575 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-18 09:06:05 +00:00
c89d692cb5
[Build] Propagate vLLM version to Rust binaries ( #52593 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-08-17 23:37:37 -07:00
Connor Carpenter and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f08a95f8d8
[Rust Frontend][gRPC] Advertise LoRA capabilities ( #52031 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-17 13:27:56 -07:00
Biswa Panda
42156466db
[Rust Frontend][gRPC] Preserve skip_special_tokens decoding option ( #52384 )
...
Signed-off-by: Biswa Panda <biswa.panda@gmail.com >
2026-08-15 08:54:58 +00:00
Connor Carpenter and Nick Hill
f473870ecf
[Rust Frontend][gRPC] Add RL lifecycle control ( #51316 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-14 11:32:39 -07:00
3962042304
[Bugfix][V1][Multimodal] Recover from P0/P1 processor cache drift ( #46747 ) ( #46747 )
...
Signed-off-by: William Zhao <williamzh@meta.com >
Signed-off-by: WillZZZy <93454470+WillZZZy@users.noreply.github.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
Co-authored-by: Claude <noreply@anthropic.com >
2026-08-12 12:06:57 +08:00
b64a2708b0
Bound generation inputs before expensive work ( #51447 )
...
Signed-off-by: Clinton Thomas <1033162+KernelClint@users.noreply.github.com >
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-11 22:03:50 +08:00
33584901ba
[Frontend] Add content_parts to /inference/v1/generate for raw multim… ( #51478 )
...
Signed-off-by: Allen Shen <aoshen@inferact.ai >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-11 10:00:12 +08:00
Bugen Zhao and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f1e921b6d6
[Rust Frontend] Support dynamic tools from developer messages ( #51144 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-10 18:30:38 -07:00
Connor Carpenter and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
fa722b9f01
[Rust Frontend][gRPC] Add explicit data-parallel rank routing ( #51178 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-10 15:05:24 -07:00
Sage
865781e627
[Rust Frontend] Add standalone Rust renderer ( #50289 )
2026-08-06 11:23:26 +00:00
Chauncey
22170354d8
[Feature] Parse request priority from HTTP header ( #51089 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
2026-08-05 21:10:29 -07:00
Sage and Bugen Zhao
5bf32605a9
[Rust Frontend] Deduplicate request preprocessing for /tokenize ( #50448 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-08-05 08:27:35 +00:00
Connor Carpenter and Bugen Zhao
e98a8774cb
[Rust Frontend][gRPC] Add multimodal image inference ( #50368 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-08-04 09:34:31 +00:00
Karen Chung
f57123aa2d
feat(frontend): session id plumbing into requests ( #48048 )
...
Signed-off-by: Karen Chung <karenc@nvidia.com >
2026-08-03 22:18:58 +00:00
zcxGGmu and zq
c4e9f09de7
[Bugfix][Frontend] Reject empty gRPC stop strings ( #50746 )
...
Signed-off-by: zq <zhouquan1511@163.com >
Co-authored-by: zq <zhouquan1511@163.com >
2026-08-03 20:50:15 +00:00
Bugen Zhao and OpenAI Codex
e9096fcb12
[Rust Frontend] Improve startup failure and readiness logs ( #50406 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-07-30 14:18:17 -07:00
Connor Carpenter and Nick Hill
48fc2e2797
feat(grpc): add KV event source discovery ( #50033 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-29 13:46:40 -07:00
Maria Guevara
c44e191b01
[Rust Frontend] Add --limit-mm-per-prompt support ( #49604 )
...
Signed-off-by: Maria Guevara <kawaiiplush14@gmail.com >
2026-07-29 17:21:41 +08:00
Bugen Zhao
2899dca843
[Model] Add Kimi K3 support: Rust frontend [1/2] ( #50104 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-28 14:31:13 -07:00
Reid
912d6b619d
[Rust Frontend] Align sampling validation with Python ( #47494 )
...
Signed-off-by: reidliu41 <reid201711@gmail.com >
2026-07-28 12:02:00 +00:00
Reid
bf9f23003c
[Rust Frontend] Fix finish reason for named tool choices ( #49496 )
...
Signed-off-by: reidliu41 <reid201711@gmail.com >
2026-07-28 10:53:18 +00:00
Connor Carpenter and Nick Hill
e3c2fc3b3c
[Rust Frontend][gRPC] Add server and model discovery ( #49491 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-27 09:53:27 -07:00
Connor Carpenter and OpenAI Codex
c79ad3ae21
[Rust Frontend][gRPC] Add abort control RPC ( #49255 )
...
Co-authored-by: OpenAI Codex <codex@openai.com >
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
2026-07-22 11:31:01 +00:00
Connor Carpenter and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
1940c8441e
[Rust Frontend][gRPC] Add engine-aware health reporting ( #48992 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
2026-07-21 10:54:39 +08:00
Salt Sato
d835ad572c
[Bugfix][Rust Frontend] Map missing prompt logprobs for single-token prompts in chat and raw generate ( #49111 )
...
Signed-off-by: Feathbow <feathbow@gmail.com >
2026-07-20 10:00:06 +00:00
Reid
8c3393f373
[Bugfix][Rust Frontend] Limit chat top_logprobs in responses ( #48134 )
...
Signed-off-by: reidliu41 <reid201711@gmail.com >
2026-07-16 09:41:44 +00:00
Tahsin Tunan
1b30ae4ca4
[Rust Frontend] Fix flaky tls_handshake_timeout_drops_silent_client test ( #47873 )
...
Signed-off-by: Tahsin Tunan <tahsintunan@gmail.com >
2026-07-15 11:05:38 +00:00
e281ac663a
[Rust Frontend] Integrate MM audio support ( #48554 )
...
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
2026-07-15 15:00:17 +08:00
Bugen Zhao
487dfb3418
[CI] Add SPDX license header to Rust/Protobuf sources ( #48472 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-13 10:22:47 +01:00
omerpaz95 and Or Ozeri
8df14cfc8c
[EC Connector] Add EC Transfer Params ( #42433 )
...
Signed-off-by: omerpaz95 <omerpaz95@gmail.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-07-12 14:35:33 +03:00
83762b77b0
[Frontend] Add /abort_requests to the RLHF dev API router ( #47173 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-12 14:21:02 +08:00
Bugen Zhao
074bdd0d99
[Rust Frontend] Integrate MM video support ( #47959 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-10 08:15:33 +00:00
dcdd756d75
[CI] GSM8K eval integration test for KV offloading ( #46893 )
...
Signed-off-by: Tyler Michael Smith <tyler@tylermsmith.com >
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Signed-off-by: Itay Etelis <itay.etelis@ibm.com >
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Signed-off-by: Or Ozeri <oro@il.ibm.com >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Co-authored-by: Itay Etelis <itay.etelis@ibm.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-08 17:59:48 -04:00
Bugen Zhao and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
e55cc59e52
[Rust Frontend][CI] Unblock more end-to-end test cases ( #47735 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-07 08:27:20 +00:00
Tahsin Tunan
34e6dfced8
[Rust Frontend] Stamp arrival_time at the frontend entry ( #47787 )
...
Signed-off-by: Tahsin Tunan <tahsintunan@gmail.com >
2026-07-07 03:27:10 +00:00
Isotr0py
400a9c386d
[Rust Frontend] Bump llm-multimodal version ( #47530 )
...
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
2026-07-03 09:48:36 +00:00
yangyang-cs95 and Bugen Zhao
6429d5f527
[Rust Frontend] add repetition_detection support to sampling params ( #46684 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-03 14:06:23 +08:00
Bugen Zhao
25fcb65d51
[Rust Frontend] Use enum-backed domain types for engine outputs and structured outputs ( #47283 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-02 10:41:46 +01:00
Pranav Thakur and Bugen Zhao
08a8a4af3f
feat(rust): expose profiler control routes in Rust frontend ( #46306 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-02 08:46:07 +00:00
Bugen Zhao
c638f9216a
[Rust Frontend] Split engine core DTOs into separate modules ( #47265 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-01 15:28:21 +01:00
