MOVZX and Hinata
2066d3143f
[Mini-VLLM] Cut non-OpenAI surfaces, dead kernels, Mistral stack
...
- Remove Anthropic, Cohere, SageMaker, decisions, structured-decisions,
rl, render-launcher and batch entrypoints; keep scale-out token routes
- Drop the Swagger static bundle and the /docs, /redoc routes
- Delete device-keyed tuning configs for fused_moe, quant utils, mamba
select-state and helion; no sm86 entry ever existed for any of them
- Remove ten unused rope variants plus the fbgemm_fp8 and fp_quant
quantizers and their config, metrics and loader plumbing
- Cut the CPU device build: csrc/cpu, cpu_extension.cmake, cpu
requirements and the setup.py cpu target path
- Delete 29 environment variables nothing reads and 14 chat-template
fallbacks for models this fork no longer registers
- Remove nine dead CUDA ops (deepseek_v4, kimi_k3, minimax, muse_spark,
dsv3 gemm) with their kernels, bindings, wrappers and CMake entries
- Delete the connector test suites that targeted removed connectors
- Cut the Mistral tokenizer and parser stack, its tests, and the
mistral_common and ijson dependency pins
- Restore the Qwen4Exp registry rows and nvidia/mtp.py that an earlier
cut deleted while keeping the model tree
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 17:10:56 +07:00
Ting SUN
eb8de8d1a1
[Bugfix][OpenAI] Respect allowed_token_ids in beam search ( #47630 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
2026-10-17 08:50:40 +00:00
Shuolei Wang and aoshen02
b3a7e2ff6e
[RL][Frontend] Return weight-version spans on OpenAI completions and chat ( #61347 )
...
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
2026-10-17 02:08:07 +00:00
Martin Hickey
252be28c18
[Frontend] Replace return_mm_kwargs with an mm_payload enum on render requests ( #61427 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-10-15 12:41:46 +00:00
wang.yuqi
c1692c85f1
[Misc] mv chat_utils.py to renderers. ( #60522 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-10-10 06:50:34 +00:00
Maroon Ayoub and Claude
8a03ccf7b3
[Frontend] Add return_mm_kwargs to render requests ( #60728 )
...
Signed-off-by: Maroon Ayoub <mayoub@redhat.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-09 08:14:13 +00:00
Sriraj and Flora Feng
d77cf81156
[Bugfix][Frontend] Honor default penalties in batch chat completions ( #60307 )
...
Signed-off-by: sricursion <sricursion@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-07 00:21:00 +00:00
Ting SUN and Flora Feng
4a705c7b54
[Bugfix][Frontend] Preserve logprobs when top_logprobs is null ( #47838 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-05 22:43:31 +00:00
Yida Weng and Flora Feng
18c8a65edd
[BugFix][Frontend] Pass reasoning_ended through render → generate ( #60059 ) ( #60062 )
...
Signed-off-by: Yida Weng <84164537+YidaWeng@users.noreply.github.com >
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-05 20:39:25 +00:00
Raphaël Rialland and Simon Veitner
c32495d58f
[watermark] compatibility validation ( #56801 )
...
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai >
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Simon Veitner <sveitner@redhat.com >
2026-10-05 14:37:13 -04:00
Shijie Lyu
ac68c30872
[Bugfix][Frontend] Ignore reused prompt token ids for media in /v1/responses ( #59173 )
...
Signed-off-by: Shijie Lyu <lshjhf@gmail.com >
2026-09-30 05:21:09 +00:00
c6843d397c
[Bugfix][Frontend] Validate reused prompt token ids; render messages for media and echo ( #55771 )
...
Signed-off-by: Shijie Lyu <sjlyu@amazon.com >
Co-authored-by: Shijie Lyu <sjlyu@amazon.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-29 10:36:22 +08:00
a7d203efe5
fix(config): apply presence_penalty/frequency_penalty from override-generation-config ( #50769 )
...
Signed-off-by: Chenglun Hu <chenglunhu@gmail.com >
Signed-off-by: hclsys <chenglunhu@gmail.com >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-09-24 08:48:18 -07:00
Cyrus Leung
56cac80b14
[Deprecation] Remove assistant_token_mask support ( #57520 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-09-19 00:17:51 +08:00
Alexander Lee and OpenAI Codex
10e8d5614c
[Bugfix][Beam Search] Respect skip_special_tokens during decoding ( #56211 )
...
Signed-off-by: lxy-alexander <lxy_alexander@outlook.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-09-14 05:49:44 +00:00
ea40bb9e90
[feature] Watermarked generation and detection (Gumbel-max algorithm) ( #54053 )
...
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-09 21:00:15 -07:00
e962733e08
[Security] Validate cache salts before they reach LMCache ( #51444 )
...
Signed-off-by: Clinton Thomas <1033162+KernelClint@users.noreply.github.com >
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-09-05 03:56:06 +00:00
wang.yuqi and Cyrus Leung
eeb549a74d
[Frontend] Move engine/protocol.py out openai folder ( #54492 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com >
2026-08-31 10:13:50 +00:00
Juan Pérez de Algaba
1dc464d426
[Bugfix] Bound cache_salt length to prevent DoS via scheduler CPU exhaustion ( #54353 )
...
Signed-off-by: jperezde <jperezde@redhat.com >
2026-08-30 04:28:16 +00:00
7cfb97e337
[Frontend][Core][Spec Decode] Per-request acceptance stats in OpenAI API responses ( #48915 )
...
Signed-off-by: Matthew Kotila <7692737+matthewkotila@users.noreply.github.com >
Co-authored-by: Reed Meyerson <31574681+reed-meyerson@users.noreply.github.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-08-20 15:42:49 -07:00
7553aac77b
[Frontend] Add routed-experts prompt offset ( #51906 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-13 08:09:00 -07:00
b64a2708b0
Bound generation inputs before expensive work ( #51447 )
...
Signed-off-by: Clinton Thomas <1033162+KernelClint@users.noreply.github.com >
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-11 22:03:50 +08:00
Tarun Kumar
48bada6ea4
Fix chat completion 500 on non-object JSON bodies ( #51654 )
...
Signed-off-by: Tarun Kumar <takumar@redhat.com >
2026-08-11 01:31:19 +00:00
Karen Chung
f57123aa2d
feat(frontend): session id plumbing into requests ( #48048 )
...
Signed-off-by: Karen Chung <karenc@nvidia.com >
2026-08-03 22:18:58 +00:00
Cyrus Leung
b3f97dae24
[Frontend] Require cache_salt to be non-empty via schema ( #50816 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-08-03 06:51:27 +00:00
andrewbcohere and Cyrus Leung
9c110fa522
[Frontend] Cohere chat v2 api support ( #47189 )
...
Signed-off-by: Andrew Berneshawi <andrewberneshawi@cohere.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-07-31 22:53:54 -07:00
yzong-rh and Ben Browning
e8b358b4c0
[Bugfix][Responses] Add tests for Chat Completions Responses API Render Parity ( #50334 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
Co-authored-by: Ben Browning <56071+bbrowning@users.noreply.github.com >
2026-07-31 15:45:40 -04:00
Julien Denize
1a20d23dab
[PARSER][Mistral] unified engine-based parser for reasoning and tool calls ( #48947 )
2026-07-30 15:38:00 +02:00
Bugen Zhao
f5a7cce9b6
[Model] Add Kimi K3 support: Python frontend [2/2] ( #50093 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-07-29 01:06:01 -07:00
Walter Beller-Morales and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
8040ef2426
[Frontend] expose stream_interval as req sampling param ( #49754 )
...
Signed-off-by: walterbm <walter.beller.morales@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-27 11:30:37 +08:00
Umut Polat
040cbf95cc
[Misc] Use VLLMValidationError in chat completion tool and batch validators ( #49214 )
...
Signed-off-by: Umut Polat <52835619+umut-polat@users.noreply.github.com >
2026-07-21 11:38:20 +00:00
hcl
530ee36a0d
fix(openai): reject non-numeric logprobs with 400 instead of 500 ( #49144 )
...
Signed-off-by: Chenglun Hu <chenglunhu@gmail.com >
2026-07-20 10:04:50 +00:00
yzong-rh
ac5f38a0f7
[Refactor] Extract StructuredOutputsParams creation logic from Request.to_sampling_params ( #49003 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
2026-07-19 05:18:00 -07:00
langzhao-netizen and Claude
8b8af2caf7
[Frontend] Expose logprob_token_ids on Python OpenAI endpoints ( #43463 )
...
Signed-off-by: Lang Zhao <lang.zhao@galileo.ai >
Co-authored-by: Claude <noreply@anthropic.com >
2026-07-13 14:40:21 -07:00
omerpaz95 and Or Ozeri
8df14cfc8c
[EC Connector] Add EC Transfer Params ( #42433 )
...
Signed-off-by: omerpaz95 <omerpaz95@gmail.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-07-12 14:35:33 +03:00
86db6c3070
[Frontend] add per-request timing metrics field to response body of Chat/Completions APIs ( #46768 )
...
Signed-off-by: Nicholas Edelman <nedelman@nvidia.com >
Signed-off-by: Simon Mo <simon@inferact.ai >
Co-authored-by: Simon Mo <simon@inferact.ai >
Co-authored-by: GPT-5.5 <noreply@cursor.com >
2026-07-06 17:48:29 -07:00
Ting SUN
d2afe39647
[Bugfix][Frontend] Preserve default sampling params in batch chat ( #47597 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
2026-07-04 19:06:39 +00:00
Taneem Ibrahim
2f21224527
[Misc] Update request-extras parity for batch chat completion ( #47333 )
...
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
2026-07-04 10:19:04 -04:00
David Feng and Claude
18f658bb31
[Bugfix][Frontend] Fix batch chat endpoint corrupting logprobs when return_token_ids is set ( #47384 )
...
Signed-off-by: David Feng <fenghourun@meta.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-07-03 03:01:34 -07:00
ganesh
fca432e60a
[Bugfix] Propagate default stop_token_ids to per-request SamplingParams ( #35076 )
...
Signed-off-by: sriganesh123 <arjulasriganesh@gmail.com >
2026-06-30 12:10:09 +08:00
Ranran and Benjamin Chislett
09841ae705
[Render][Speculator] Add return_loss_mask to render endpoint for training data generation ( #46846 )
...
Signed-off-by: Ranran Haoran Zhang <ranzhang@redhat.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
2026-06-28 00:07:33 -07:00
Hyunkyun Moon
950ee4c2e4
[API] Add token offsets to render endpoints (/v1/.../render) ( #44226 )
...
Signed-off-by: HyunKyun Moon <mhg5303@gmail.com >
2026-06-26 05:02:52 -07:00
6c427dd401
[BugFix] Omit empty tool_calls from OpenAI chat responses ( #44105 )
...
Signed-off-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com >
Signed-off-by: Chauncey <chaunceyjiang@gmail.com >
Co-authored-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-06-23 13:43:53 +08:00
Flora Feng
0d80979644
[Chore] Consolidate reasoning/tool parser attributes into unified Parser in chat serving ( #45548 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-06-15 11:16:45 -04:00
Andreas Karatzas
54d0c36fff
[CI] Stabilize OpenAI schema fuzzing for malformed structural tags ( #44131 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-06-01 19:56:15 -07:00
Umut Polat
f46e6be169
[Misc] Use VLLMValidationError consistently in chat completion and completion protocol validators ( #36254 )
...
Signed-off-by: umut-polat <52835619+umut-polat@users.noreply.github.com >
2026-06-01 04:04:11 +00:00
Ashwin Giridharan and Chauncey
52a31ccecc
[Bugfix] Map reasoning_effort to enable_thinking in chat template kwargs ( #43401 )
...
Signed-off-by: Ashwin Giridharan <girida@amazon.com >
Signed-off-by: Chauncey <chaunceyjiang@gmail.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-05-27 05:39:49 -07:00
linzm1007 and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
739af5c7e1
[Reasoning] [Bugfix] Reject invalid thinking_token_budget values ( #43402 )
...
Signed-off-by: linzm1007 <linzm1007@126.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-26 03:37:30 -07:00
Andreas Karatzas
5940590855
[ROCm][CI] Stabilize 400 error return code for invalid schema inputs ( #43016 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-05-24 10:06:49 +00:00
ruizhang and Rui Zhang
60af5c16ee
[Frontend] Add truncation side to OpenAI endpoints ( #43260 )
...
Signed-off-by: Rui Zhang <rza21.bc@gmail.com >
Signed-off-by: Rui Zhang <rui.zhang@globalrelay.net >
Co-authored-by: Rui Zhang <rui.zhang@globalrelay.net >
2026-05-21 20:32:31 -07:00