167 Commits
Author SHA1 Message Date
Ting SUN eb8de8d1a1 [Bugfix][OpenAI] Respect allowed_token_ids in beam search (#47630)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
2026-10-17 08:50:40 +00:00
JovanandOpenAI Codex 5be4266f5c [Bugfix][Frontend] Preserve Cohere and Anthropic streaming errors (#61947)
Signed-off-by: Kevin <jia__kuan@163.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-10-17 06:10:16 +00:00
Shuolei Wangandaoshen02 b3a7e2ff6e [RL][Frontend] Return weight-version spans on OpenAI completions and chat (#61347)
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-10-17 02:08:07 +00:00
3dcfaeb4e8 [Frontend] Apply each endpoint's top_logprobs cut in derender (#59518)
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com>
Co-authored-by: Yin Li <kevinli.ai.work@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-10-15 15:04:04 +00:00
Martin Hickey 252be28c18 [Frontend] Replace return_mm_kwargs with an mm_payload enum on render requests (#61427)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
2026-10-15 12:41:46 +00:00
Flora FengandChauncey 80dd705c1d [Frontend] Make finish_reason and Responses item status truthful (#59324)
Signed-off-by: Flora Feng <4florafeng@gmail.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-10-15 10:12:02 +01:00
Bugen ZhaoandClaude Opus 5.5 8fa5ae844a [Frontend] Let an unset --tool-strict-level fall back to the tool parser's default (#61328)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 15:48:38 +08:00
GuanLuoandsfeng33 6d1daa9fe5 [Bugfix][Frontend] Expose completion response schemas in OpenAPI (#60926)
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
2026-10-13 18:54:06 +00:00
wang.yuqi c1692c85f1 [Misc] mv chat_utils.py to renderers. (#60522)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-10-10 06:50:34 +00:00
Maroon AyoubandClaude 8a03ccf7b3 [Frontend] Add return_mm_kwargs to render requests (#60728)
Signed-off-by: Maroon Ayoub <mayoub@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-09 08:14:13 +00:00
dingdangmaoandFlora Feng bab0ac61f8 [Bugfix][Frontend] Forward the reasoning wiring to the engine for batch chat completions (#59263)
Signed-off-by: apex-mochen <2756823972@qq.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-08 19:32:46 +00:00
SrirajandFlora Feng d77cf81156 [Bugfix][Frontend] Honor default penalties in batch chat completions (#60307)
Signed-off-by: sricursion <sricursion@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-07 00:21:00 +00:00
Ting SUNandFlora Feng 4a705c7b54 [Bugfix][Frontend] Preserve logprobs when top_logprobs is null (#47838)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Signed-off-by: Flora Feng <4florafeng@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-05 22:43:31 +00:00
Yida WengandFlora Feng 18c8a65edd [BugFix][Frontend] Pass reasoning_ended through render → generate (#60059) (#60062)
Signed-off-by: Yida Weng <84164537+YidaWeng@users.noreply.github.com>
Signed-off-by: Flora Feng <4florafeng@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-05 20:39:25 +00:00
Raphaël RiallandandSimon Veitner c32495d58f [watermark] compatibility validation (#56801)
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
2026-10-05 14:37:13 -04:00
Stefan WangandRobert Shaw b0eb87fe49 [Bugfix][Frontend] Return streaming errors before the first token (#40986)
Signed-off-by: 1fanwang <1fannnw@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
2026-10-04 16:42:49 +00:00
9bd993564f [Frontend] Parse tool calls and reasoning from checkpoint response templates (#58604)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Yoni Gozlan <yonigozlan@users.noreply.github.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
2026-10-02 02:33:00 +00:00
Shijie Lyu ac68c30872 [Bugfix][Frontend] Ignore reused prompt token ids for media in /v1/responses (#59173)
Signed-off-by: Shijie Lyu <lshjhf@gmail.com>
2026-09-30 05:21:09 +00:00
c6843d397c [Bugfix][Frontend] Validate reused prompt token ids; render messages for media and echo (#55771)
Signed-off-by: Shijie Lyu <sjlyu@amazon.com>
Co-authored-by: Shijie Lyu <sjlyu@amazon.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-09-29 10:36:22 +08:00
Flora Feng 6f65fde419 [Bugfix][Frontend] Apply Harmony adjust_request in batched chat completions (#58958)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-09-28 04:12:04 +00:00
Flora Feng 027b6f3a22 [Bugfix][Frontend] Use a fresh parser per choice in non-streaming chat completions (#58939)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-09-28 02:01:35 +00:00
Flora Feng 4d07e90653 [Bugfix][Frontend] Sample batched chat completions from the adjusted requests (#58929)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
2026-09-28 01:46:55 +00:00
8b47e8b22c [Perf][Frontend] Defer reasoning usage recounts for non-continuous chat streams (#56067)
Signed-off-by: Cheng Rui <286040359@qq.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-09-25 14:08:23 -07:00
f339e0e757 [Bugfix][Frontend] Keep logprobs of parser-suppressed streaming chunks (#58583)
Signed-off-by: errmakov <ide404@gmail.com>
Co-authored-by: Yanxiao Zhao <39199723+sdpkjc@users.noreply.github.com>
Co-authored-by: Prakhar Agarwal <270064960+agarwalprakhar2511@users.noreply.github.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-09-25 16:33:19 +00:00
a7d203efe5 fix(config): apply presence_penalty/frequency_penalty from override-generation-config (#50769)
Signed-off-by: Chenglun Hu <chenglunhu@gmail.com>
Signed-off-by: hclsys <chenglunhu@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-09-24 08:48:18 -07:00
Ting SUN b33c8a657d [Bugfix][Frontend] Keep length finish_reason for max_tokens-truncated streaming tool calls (#46303)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
2026-09-23 14:31:10 +08:00
Cyrus Leung 56cac80b14 [Deprecation] Remove assistant_token_mask support (#57520)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-09-19 00:17:51 +08:00
dee37d8911 [Frontend] Add --tool-strict-level for server-side control for structural tag activation (#56268)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: lazymio <mio@lazym.io>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
2026-09-18 04:32:39 +00:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
Alexander LeeandOpenAI Codex 10e8d5614c [Bugfix][Beam Search] Respect skip_special_tokens during decoding (#56211)
Signed-off-by: lxy-alexander <lxy_alexander@outlook.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-09-14 05:49:44 +00:00
wang.yuqi fe4d4109c2 [Frontend][last/N] Move all non-OpenAI content out of the OpenAI folder. (#56369)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-09-11 05:33:26 +00:00
Raphaël RiallandOpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
ea40bb9e90 [feature] Watermarked generation and detection (Gumbel-max algorithm) (#54053)
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-09 21:00:15 -07:00
e962733e08 [Security] Validate cache salts before they reach LMCache (#51444)
Signed-off-by: Clinton Thomas <1033162+KernelClint@users.noreply.github.com>
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-09-05 03:56:06 +00:00
Nicolò Lucchesi dafbef15a1 [Core] Add max_num_queued_reqs and max_num_queued_tokens for queue size management (#49445)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-08-31 14:26:49 +00:00
wang.yuqiandCyrus Leung eeb549a74d [Frontend] Move engine/protocol.py out openai folder (#54492)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
2026-08-31 10:13:50 +00:00
Juan Pérez de Algaba 1dc464d426 [Bugfix] Bound cache_salt length to prevent DoS via scheduler CPU exhaustion (#54353)
Signed-off-by: jperezde <jperezde@redhat.com>
2026-08-30 04:28:16 +00:00
Kaif KohariandCyrus Leung 7fbfca2670 [Bugfix][Frontend] Only echo the assistant turn in batched chat completions (#52529)
Signed-off-by: Kaif <kaifkohari10@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-08-29 09:44:17 -07:00
Dieter Schwarzmann 585bb07c70 feat: add SSE keep-alive comments for idle streaming responses (#51034)
Signed-off-by: fuzzifikation <40043768+fuzzifikation@users.noreply.github.com>
2026-08-23 18:13:02 -07:00
andrewbcohere f94dcde5c5 Fix Cohere ChatV2 citation and tool handling issues (#52175)
Signed-off-by: Andrew Berneshawi <andrewberneshawi@cohere.com>
2026-08-21 15:23:17 -07:00
7cfb97e337 [Frontend][Core][Spec Decode] Per-request acceptance stats in OpenAI API responses (#48915)
Signed-off-by: Matthew Kotila <7692737+matthewkotila@users.noreply.github.com>
Co-authored-by: Reed Meyerson <31574681+reed-meyerson@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-20 15:42:49 -07:00
Chaunceyandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> ac2ae8798c [Frontend] Support count_reasoning_tokens in the Streaming Parser Engine (#45802)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-15 06:51:14 -07:00
aoshen02OpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
7553aac77b [Frontend] Add routed-experts prompt offset (#51906)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-13 08:09:00 -07:00
Clinton ThomasLucas Bourtoulemergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
b64a2708b0 Bound generation inputs before expensive work (#51447)
Signed-off-by: Clinton Thomas <1033162+KernelClint@users.noreply.github.com>
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-11 22:03:50 +08:00
Tarun Kumar 48bada6ea4 Fix chat completion 500 on non-object JSON bodies (#51654)
Signed-off-by: Tarun Kumar <takumar@redhat.com>
2026-08-11 01:31:19 +00:00
Chauncey 22170354d8 [Feature] Parse request priority from HTTP header (#51089)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2026-08-05 21:10:29 -07:00
Karen Chung f57123aa2d feat(frontend): session id plumbing into requests (#48048)
Signed-off-by: Karen Chung <karenc@nvidia.com>
2026-08-03 22:18:58 +00:00
Cyrus Leung b3f97dae24 [Frontend] Require cache_salt to be non-empty via schema (#50816)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-08-03 06:51:27 +00:00
andrewbcohereandCyrus Leung 9c110fa522 [Frontend] Cohere chat v2 api support (#47189)
Signed-off-by: Andrew Berneshawi <andrewberneshawi@cohere.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-07-31 22:53:54 -07:00
yzong-rhandBen Browning e8b358b4c0 [Bugfix][Responses] Add tests for Chat Completions Responses API Render Parity (#50334)
Signed-off-by: Yifan Zong <yzong@redhat.com>
Co-authored-by: Ben Browning <56071+bbrowning@users.noreply.github.com>
2026-07-31 15:45:40 -04:00
wang.yuqi f5ffc59b6a [Renderer] Warm up the renderer properly. (#50408)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-07-31 11:46:53 +00:00