1bc6e072b0
[Rust Frontend] Add per-request timing metrics to chat and completions ( #59314 )
...
Signed-off-by: wxin <wxin@nvidia.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 15:39:10 -07:00
Martin Hickey
7039c7f17b
[Bugfix][Rust Frontend] Omit unset sampling params in render output ( #61220 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-10-12 19:31:03 +00:00
Bugen Zhao and Claude Opus 5.5
35e920d671
[Rust Frontend] Support --stream-interval and per-request stream_interval ( #60589 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 14:10:42 -07:00
Bugen Zhao and Claude Opus 5.5
9a585c62aa
[Rust Frontend] Box rarely-set fields of per-token output types ( #60588 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 11:01:16 -07:00
Alec
14e3902d51
[Frontend][Rust] Add gRPC forbidden token sequences and cache usage ( #59837 )
...
Signed-off-by: Alec Flowers <aflowers@nvidia.com >
2026-10-07 07:46:44 -07:00
e319f86f15
[Feature] Per-row candidate IDs for prefill token scoring (M2 of #56860 ) ( #56984 )
...
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-03 14:17:09 +00:00
Bugen Zhao and Claude Opus 5.5
24fb5fc08c
[Rust Frontend] Add hf parser for response templates ( #59005 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 08:16:38 +00:00
b721a4c709
[Feature] Add fixed-token prefill scoring ( #54335 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-09-28 07:50:09 -07:00
Biswa Panda and Codex
cf78797cb7
[Rust Frontend] Pass vision preprocessing context for Nemotron-H ( #57634 )
...
Pass the remaining engine context-length budget to model-owned vision processors through VisionPreprocessingContext. Preserve Nemotron batched engine fields and recognize llm_config as a text_config alias.
Use the merged upstream llm-multimodal revision f0985ef65967615db2c79279aa07818499301bfd.
Co-authored-by: Codex <noreply@openai.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-09-24 14:08:26 +08:00
Bugen Zhao and Claude Fable 5.1
d50723df04
[Rust Frontend] Introduce parser-owned output grammar interfaces ( #55269 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-22 05:35:07 +00:00
Bugen Zhao and Codex
ff3c9cba3f
[Rust Frontend] Add MiMo V2.5 parser support ( #57933 )
...
Co-authored-by: Codex <noreply@openai.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-09-22 04:27:09 +00:00
Karen Chung
986e217158
[Feature] Add first-class KV hints request envelope for programmatic KV management ( #53423 )
...
Signed-off-by: Karen Chung <karenc@nvidia.com >
2026-09-21 09:56:11 -07:00
Biswa Panda and Bugen Zhao
95d723700b
[Rust Frontend] Return sampling masks over gRPC ( #56777 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Biswa Panda <biswa.panda@gmail.com >
2026-09-18 08:26:17 +00:00
Bugen Zhao
bfd713bf8b
[Rust Frontend] Support HF config overrides --hf-overrides ( #56931 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-09-16 02:52:55 +00:00
Nicolò Lucchesi and Codex
1713a09491
[Rust Frontend] Forward per-request watermarking controls ( #56338 )
...
Co-authored-by: Codex <noreply@openai.com >
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-09-14 23:17:02 +08:00
Bugen Zhao and Codex
485421b1c3
[Rust Frontend] Honor HF revisions, offline mode, and cache directory ( #56386 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-12 03:39:17 +00:00
41cffa9af0
[Rust Frontend] Strongly type wire dtypes and multimodal modalities ( #56174 )
...
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-09-10 02:11:22 +00:00
Bugen Zhao
ad127d9a0f
[Perf][Rust Frontend] Coalesce decoded chunks per engine update ( #55012 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-09-02 15:44:23 -07:00
Bugen Zhao
b911fe85c4
[Rust Frontend] Attribute decoded text to tokens ( #52910 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-09-01 22:25:31 +00:00
Pranav Thakur and Bugen Zhao
085e9bb07e
[Rust Frontend] Add support for truncate_prompt_tokens and truncation_side ( #48584 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: pranavthakur0-0 <pranavthakurdew@gmail.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-08-29 00:16:37 +00:00
Bugen Zhao and Nick Hill
9db222c6be
[Rust Frontend] Align OpenAI request and response edge cases ( #53218 )
...
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-08-27 19:23:38 +00:00
Bugen Zhao
df3b3422b4
[Rust Frontend] Add HY3 unified parser and local XGrammar structural-tag builder ( #53054 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-08-20 19:13:54 -07:00
Bugen Zhao
f32b17b6d6
[Rust Frontend] Support --generation-config vllm ( #53044 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-08-20 17:31:25 -07:00
Biswa Panda and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
6a391a931a
[Rust Frontend][RL] add routed expert prompt offset ( #52703 )
...
Signed-off-by: Biswa Panda <biswa.panda@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-18 20:04:28 +00:00
Connor Carpenter and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
fa722b9f01
[Rust Frontend][gRPC] Add explicit data-parallel rank routing ( #51178 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-10 15:05:24 -07:00
Sage and Bugen Zhao
5bf32605a9
[Rust Frontend] Deduplicate request preprocessing for /tokenize ( #50448 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-08-05 08:27:35 +00:00
Karen Chung
f57123aa2d
feat(frontend): session id plumbing into requests ( #48048 )
...
Signed-off-by: Karen Chung <karenc@nvidia.com >
2026-08-03 22:18:58 +00:00
zcxGGmu and zq
c4e9f09de7
[Bugfix][Frontend] Reject empty gRPC stop strings ( #50746 )
...
Signed-off-by: zq <zhouquan1511@163.com >
Co-authored-by: zq <zhouquan1511@163.com >
2026-08-03 20:50:15 +00:00
Samuel Laferriere and Claude
d6938b7408
[Bugfix][Rust Frontend] Select earliest-completing stop string ( #50200 )
...
Signed-off-by: Samuel Laferriere <9342524+samlaf@users.noreply.github.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-07-31 00:33:19 +00:00
Reid
912d6b619d
[Rust Frontend] Align sampling validation with Python ( #47494 )
...
Signed-off-by: reidliu41 <reid201711@gmail.com >
2026-07-28 12:02:00 +00:00
Bugen Zhao and OpenAI Codex
5ed3faa43d
[Rust Frontend] Add ordinary-text tokenizer encoding ( #49992 )
...
Co-authored-by: OpenAI Codex <noreply@openai.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-28 15:46:51 +08:00
Sage
0f6cf7f628
[Rust Frontend] Extract request preparation from the inference path ( #49045 )
...
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com >
2026-07-22 11:31:36 +00:00
6570c9800c
[Model] Add Inkling model support [1/N] ( #48799 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-07-15 23:40:07 -07:00
Bugen Zhao
487dfb3418
[CI] Add SPDX license header to Rust/Protobuf sources ( #48472 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-13 10:22:47 +01:00
omerpaz95 and Or Ozeri
8df14cfc8c
[EC Connector] Add EC Transfer Params ( #42433 )
...
Signed-off-by: omerpaz95 <omerpaz95@gmail.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-07-12 14:35:33 +03:00
Bugen Zhao
074bdd0d99
[Rust Frontend] Integrate MM video support ( #47959 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-10 08:15:33 +00:00
Harry Mellor
dd94484577
Bump Transformers version to 5.10.4 ( #41359 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-07-07 05:13:28 -07:00
Bugen Zhao and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
e55cc59e52
[Rust Frontend][CI] Unblock more end-to-end test cases ( #47735 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-07 08:27:20 +00:00
Tahsin Tunan
34e6dfced8
[Rust Frontend] Stamp arrival_time at the frontend entry ( #47787 )
...
Signed-off-by: Tahsin Tunan <tahsintunan@gmail.com >
2026-07-07 03:27:10 +00:00
Reid
39a1d32b59
[Rust Frontend] Avoid extra copies for multimodal tensors ( #47581 )
...
Signed-off-by: reidliu41 <reid201711@gmail.com >
2026-07-07 03:09:55 +00:00
yangyang-cs95 and Bugen Zhao
6429d5f527
[Rust Frontend] add repetition_detection support to sampling params ( #46684 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-03 14:06:23 +08:00
Bugen Zhao
c638f9216a
[Rust Frontend] Split engine core DTOs into separate modules ( #47265 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-01 15:28:21 +01:00
Bugen Zhao
7a327f0b4f
[Rust Frontend] Simplify unit tests with shared TestTokenizer ( #47125 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-06-30 15:34:43 +01:00
Reid and Bugen Zhao
d350fa8ddd
[Bugfix][Rust Frontend] Reject min_tokens above max_tokens ( #46733 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: reidliu41 <reid201711@gmail.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-06-26 03:41:33 +00:00
Bugen Zhao
ae7c8ec223
[Rust Frontend] Switch rustls to native-tls/OpenSSL ( #46696 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-06-25 17:19:44 -07:00
Bugen Zhao and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
ce9f64020b
[Rust Frontend] Pass effective reasoning_parser_kwargs for structured output ( #46360 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-06-24 03:13:44 +00:00
Reid and Bugen Zhao
2d721ab5d8
[Rust Frontend] Align Rust allowed_token_ids validation with Python ( #46348 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: reidliu41 <reid201711@gmail.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-06-23 08:32:33 +00:00
Bugen Zhao
25bc3be49c
[Rust Frontend] Correct --reasoning-parser semantics ( #46359 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-06-23 15:38:39 +08:00
80abe0de7d
[Rust Frontend] Support thinking_token_budget for chat and completions ( #46137 )
...
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: RickyChen / 陳昭儒 <ricky.chen@infinirc.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-06-22 16:00:02 +08:00
Reid and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
31124749d1
[Bugfix] [Rust Frontend] Fix stop string truncation with repeated matches ( #46113 )
...
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Signed-off-by: reidliu41 <reid201711@gmail.com >
2026-06-22 14:11:29 +08:00
