MOVZX and Hinata
5aa2ac64f8
[Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces
...
Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.
- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
render routes, and vllm/config/speech_to_text.py. GenerationTask is
generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
Multi, Offloading and Example connectors stay; production uses the
CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
arch gate, declarations and op registrations. The CUDA build never
defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
fail with "The model ... does not exist. Valid aliases: ...".
Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 21:06:19 +07:00
MOVZX and Hinata
2066d3143f
[Mini-VLLM] Cut non-OpenAI surfaces, dead kernels, Mistral stack
...
- Remove Anthropic, Cohere, SageMaker, decisions, structured-decisions,
rl, render-launcher and batch entrypoints; keep scale-out token routes
- Drop the Swagger static bundle and the /docs, /redoc routes
- Delete device-keyed tuning configs for fused_moe, quant utils, mamba
select-state and helion; no sm86 entry ever existed for any of them
- Remove ten unused rope variants plus the fbgemm_fp8 and fp_quant
quantizers and their config, metrics and loader plumbing
- Cut the CPU device build: csrc/cpu, cpu_extension.cmake, cpu
requirements and the setup.py cpu target path
- Delete 29 environment variables nothing reads and 14 chat-template
fallbacks for models this fork no longer registers
- Remove nine dead CUDA ops (deepseek_v4, kimi_k3, minimax, muse_spark,
dsv3 gemm) with their kernels, bindings, wrappers and CMake entries
- Delete the connector test suites that targeted removed connectors
- Cut the Mistral tokenizer and parser stack, its tests, and the
mistral_common and ijson dependency pins
- Restore the Qwen4Exp registry rows and nvidia/mtp.py that an earlier
cut deleted while keeping the model tree
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 17:10:56 +07:00
Ting SUN
eb8de8d1a1
[Bugfix][OpenAI] Respect allowed_token_ids in beam search ( #47630 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
2026-10-17 08:50:40 +00:00
3dcfaeb4e8
[Frontend] Apply each endpoint's top_logprobs cut in derender ( #59518 )
...
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com >
Co-authored-by: Yin Li <kevinli.ai.work@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-15 15:04:04 +00:00
Martin Hickey
252be28c18
[Frontend] Replace return_mm_kwargs with an mm_payload enum on render requests ( #61427 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-10-15 12:41:46 +00:00
Xunzhuo
89cab55086
[Model] Support Decision 3.0 decision models (vllm-sr/d3) ( #61565 )
...
Signed-off-by: Xunzhuo Liu <xunzhuo.liu@amd.com >
2026-10-14 17:10:55 +00:00
c923e7769b
[Feat] Add text-only OpenAI Decisions API MVP ( #60465 )
...
Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-10-14 07:51:08 +08:00
Aditya Ganesh Kumar
d9a543c500
[Bugfix][Frontend] Forward --enable-log-outputs to the Cohere Chat v2 ( #61089 )
...
Signed-off-by: Aditya Ganesh Kumar <adty910@gmail.com >
2026-10-13 12:01:59 +00:00
380638afbb
[Model] Add Clef-Flash support ( #61308 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-10-13 05:46:37 +00:00
fa39582756
[Frontend] Serve LiquidAI/d1-3B on /v1/systemone ( #60676 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-12 13:00:47 -04:00
092c35dd15
[Model] Support Cloudflare/clef-omni on /v1/systemone ( #61064 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
2026-10-12 21:27:34 +08:00
Yufeng He
187a0eb98a
[Bugfix][Frontend] Forward --enable-log-outputs to the Anthropic handler ( #60982 )
...
Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com >
2026-10-10 16:36:03 +00:00
wang.yuqi
c1692c85f1
[Misc] mv chat_utils.py to renderers. ( #60522 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-10-10 06:50:34 +00:00
Chauncey
fa5cfd48d5
[Refactor] Deduplicate trace header handling in BaseServing ( #60828 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
2026-10-10 05:27:58 +00:00
3c802af3cf
[Feat][Frontend] structured decisions: DiffusionGemma read strategy ( #60466 )
...
Signed-off-by: Matt Mastracci <matthew@mastracci.com >
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-10-10 09:33:22 +08:00
Taewoong Kim and Cyrus Leung
75e3b17c05
[Frontend] Add cache salt support to generative scoring ( #60841 )
...
Signed-off-by: Taewoong Kim <ktw2172@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-09 16:07:50 +00:00
Taewoong Kim and Chauncey
18c19eaf23
[Frontend] Add cache salt and tracing to structured decisions ( #60786 )
...
Signed-off-by: Taewoong Kim <ktw2172@gmail.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-10-09 08:53:50 +00:00
8cbd5d0300
[Frontend] structured decisions: noul and score question types ( #60696 )
...
Signed-off-by: Matt Mastracci <matthew@mastracci.com >
Co-authored-by: DeepSeek <noreply@deepseek.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-10-09 04:48:16 +00:00
ae9adec905
[Frontend] Expose /v1/systemone without an opt-in flag ( #60677 )
...
Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-08 20:21:05 +00:00
Francisco Javier Arceo and Misha Goin
705e124c05
[Bugfix] Followup to #59299 : preserve Unicode in structured decision state prompts ( #60651 )
...
Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com >
Co-authored-by: Misha Goin <mgoin64@gmail.com >
2026-10-08 11:00:03 -07:00
d33872dfeb
[Feat] Structured decisions endpoint (/v1/systemone) ( #59299 )
...
Signed-off-by: Matt Mastracci <matthew@mastracci.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
Co-authored-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-10-08 14:48:12 +00:00
wang.yuqi
459764104d
[Frontend] Move offline_utils.py to entrypoints/common. ( #58052 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-10-08 09:56:05 +00:00
7b665f7a58
[Frontend] Integer token IDs for generate output logprobs (GenerateLogProbs) ( #58181 )
...
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com >
Co-authored-by: Yin Li <kevinli.ai.work@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-06 17:22:47 +00:00
Mai Hoàng Đức and Flora Feng
5cd8a2626b
[Bugfix][Structured Output] Cap JSON schema nesting to prevent 500s and API-server stalls ( #60036 )
...
Signed-off-by: Mazukiri <maihoangduc04@gmail.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-06 00:06:59 +00:00
Ting SUN and Flora Feng
4a705c7b54
[Bugfix][Frontend] Preserve logprobs when top_logprobs is null ( #47838 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-05 22:43:31 +00:00
Raphaël Rialland and Simon Veitner
c32495d58f
[watermark] compatibility validation ( #56801 )
...
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai >
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Simon Veitner <sveitner@redhat.com >
2026-10-05 14:37:13 -04:00
Juan Pérez de Algaba and Cursor
31f2e70cd3
[Security] Gate per-request multimodal processor kwargs ( #58830 )
...
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-26 11:10:35 +00:00
Nick Hill and Kimi Code
de2d478d9a
[Bugfix][Frontend] Document 404 response for /generative_scoring ( #58788 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Kimi Code <noreply@moonshot.cn >
2026-09-25 14:08:00 -07:00
Taneem Ibrahim and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
50f78084da
[Bugfix] Fix generative scoring body cancellation ( #57729 )
...
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-25 11:01:31 -04:00
5a2ed2e7ee
[Frontend] Handle Disable Thinking in /v1/messages ( #58613 )
...
Signed-off-by: jryberg <johan.ryberg@security.ntt >
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: jryberg <johan.ryberg@security.ntt >
Co-authored-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-09-25 12:52:08 +08:00
Wentao Ye
fae5cbfc8b
[Refactor] Remove dead code multiple places ( #58002 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-22 16:36:03 -04:00
dee37d8911
[Frontend] Add --tool-strict-level for server-side control for structural tag activation ( #56268 )
...
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: lazymio <mio@lazym.io >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-09-18 04:32:39 +00:00
xinnywinne and Benjamin Chislett
fbf2c5e8be
[Frontend] Add per-request metrics to Responses API ( #55084 )
...
Signed-off-by: wxin <wxin@nvidia.com >
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
2026-09-16 22:52:05 +00:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
shaohuaxi and Codex
ca67438c08
[Bugfix][Frontend] Handle aborted requests in beam search ( #56249 )
...
Signed-off-by: 子华 <huaxi.shx@alibaba-inc.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-15 08:17:30 +00:00
Alexander Lee and OpenAI Codex
10e8d5614c
[Bugfix][Beam Search] Respect skip_special_tokens during decoding ( #56211 )
...
Signed-off-by: lxy-alexander <lxy_alexander@outlook.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-09-14 05:49:44 +00:00
ea40bb9e90
[feature] Watermarked generation and detection (Gumbel-max algorithm) ( #54053 )
...
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-09 21:00:15 -07:00
Taneem Ibrahim
4ebf61ebda
[Cohere] Bound remaining request priorities to the MessagePack int64 range ( #56146 )
...
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
2026-09-09 18:57:01 -04:00
Francisco Javier Arceo and OpenAI Codex
6a2a2bb02b
[Frontend] Add stateless /v1/responses/render endpoint ( #50195 )
...
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-07 12:50:42 +00:00
e962733e08
[Security] Validate cache salts before they reach LMCache ( #51444 )
...
Signed-off-by: Clinton Thomas <1033162+KernelClint@users.noreply.github.com >
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-09-05 03:56:06 +00:00
Nicolò Lucchesi
dafbef15a1
[Core] Add max_num_queued_reqs and max_num_queued_tokens for queue size management ( #49445 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-08-31 14:26:49 +00:00
wang.yuqi and Cyrus Leung
eeb549a74d
[Frontend] Move engine/protocol.py out openai folder ( #54492 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com >
2026-08-31 10:13:50 +00:00
7cfb97e337
[Frontend][Core][Spec Decode] Per-request acceptance stats in OpenAI API responses ( #48915 )
...
Signed-off-by: Matthew Kotila <7692737+matthewkotila@users.noreply.github.com >
Co-authored-by: Reed Meyerson <31574681+reed-meyerson@users.noreply.github.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-08-20 15:42:49 -07:00
Myeongwon Kim
8977ea8895
[Perf] Skip detokenization in offline beam search ( #50333 )
...
Signed-off-by: samuelkim7 <samuelmwkim@gmail.com >
2026-08-10 16:17:38 -07:00
Chauncey
22170354d8
[Feature] Parse request priority from HTTP header ( #51089 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
2026-08-05 21:10:29 -07:00
Karen Chung
f57123aa2d
feat(frontend): session id plumbing into requests ( #48048 )
...
Signed-off-by: Karen Chung <karenc@nvidia.com >
2026-08-03 22:18:58 +00:00
andrewbcohere and Cyrus Leung
9c110fa522
[Frontend] Cohere chat v2 api support ( #47189 )
...
Signed-off-by: Andrew Berneshawi <andrewberneshawi@cohere.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-07-31 22:53:54 -07:00
wang.yuqi
f5ffc59b6a
[Renderer] Warm up the renderer properly. ( #50408 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-07-31 11:46:53 +00:00
fae543015c
[Frontend]Flatten beam-search beams with itertools.chain instead of sum ( #48829 )
...
Signed-off-by: Wang Xingda <wangxingda1993@126.com >
Co-authored-by: 王兴达 <wangxingda@360itdeMacBook-Pro.local >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-17 23:09:14 +01:00