MOVZX and Hinata
2066d3143f
[Mini-VLLM] Cut non-OpenAI surfaces, dead kernels, Mistral stack
...
- Remove Anthropic, Cohere, SageMaker, decisions, structured-decisions,
rl, render-launcher and batch entrypoints; keep scale-out token routes
- Drop the Swagger static bundle and the /docs, /redoc routes
- Delete device-keyed tuning configs for fused_moe, quant utils, mamba
select-state and helion; no sm86 entry ever existed for any of them
- Remove ten unused rope variants plus the fbgemm_fp8 and fp_quant
quantizers and their config, metrics and loader plumbing
- Cut the CPU device build: csrc/cpu, cpu_extension.cmake, cpu
requirements and the setup.py cpu target path
- Delete 29 environment variables nothing reads and 14 chat-template
fallbacks for models this fork no longer registers
- Remove nine dead CUDA ops (deepseek_v4, kimi_k3, minimax, muse_spark,
dsv3 gemm) with their kernels, bindings, wrappers and CMake entries
- Delete the connector test suites that targeted removed connectors
- Cut the Mistral tokenizer and parser stack, its tests, and the
mistral_common and ijson dependency pins
- Restore the Qwen4Exp registry rows and nvidia/mtp.py that an earlier
cut deleted while keeping the model tree
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 17:10:56 +07:00
MOVZX and Hinata
bd70a61c20
[Mini-VLLM] Remove orphan subsystems: snapshot, CPU workers, dead decoders
...
- Delete the snapshot package, CLI command, and its launch tests
- Delete CPU workers, cpu_communicator, and the hw_agnostic layer package
- Delete the EC CPU connector, keeping the EC base glue intact
- Delete dead composite/triton_flashinfer backends and third_party/flashmla
- Delete dead video decoders, video-bound tests, and orphan utils
- Remove the deep_gemm_mega_moe option and dead allreduce fusion pair
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 09:27:06 +07:00
MOVZX and Hinata
81074e94a3
[Mini-vLLM] Cut dead subsystems: models, entrypoints, backends, utils, tests
...
- Drop dead model modules, keeping files live imports require
- Remove anthropic, cohere, codegen entrypoints and batch STT modes
- Delete unused attention backends, MoE experts, and NPU/XPU util leaves
- Strip dead third_party trees and ROCm leaves with no CUDA path
- Remove mooncake EC connectors: dependency missing since branch base
- Re-audit verified by full import sweep and Qwen3.5 smoke test
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-19 00:20:48 +07:00
MOVZX and Hinata
e61dcb9143
[Mini-vLLM] SM86-only cut: models, backends, Rust, tests
...
- Keep only Qwen3.5, Qwen3.5-MTP and Qwen4Exp model paths; trim the model registry to 41 entries
- Delete idle attention backends (MLA, FlashInfer, ROCm-only, quant backends) and dead GDN CuTeDSL kernels
- Remove inc/quark quant loaders; keep compressed-tensors and Marlin
- Drop the Rust frontend, pooling entrypoints, and idle test suites
- Rebuild kernels for sm_80;sm_86 and strip the extensions
- Verified: full import sweep clean (optional deps only) and a dummy-load Qwen3.5 engine run on RTX 3090 via the Triton GDN path
Co-authored-by: Hinata <hinata@idihore.id >
Signed-off-by: MOVZX <movzx@yahoo.com >
2026-10-18 19:19:56 +07:00
dskwe
b05ede5616
[Bugfix][Frontend] Preserve token IDs in completion derender responses ( #62114 )
...
Signed-off-by: Yuan Li <dskwelmcy@163.com >
2026-10-17 08:27:48 +00:00
Cy Cheng and Cy
d74f4180cd
[Bugfix][Frontend] Detokenize echoed prompts after truncation ( #61876 )
...
Signed-off-by: Cy <15118312+cycancode@users.noreply.github.com >
Co-authored-by: Cy <15118312+cycancode@users.noreply.github.com >
2026-10-17 05:20:38 +00:00
153a17166e
[Frontend] Add the Beam reasoning and tool parsers ( #62052 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Yoni Gozlan <74535834+yonigozlan@users.noreply.github.com >
Co-authored-by: neilkg <33635204+neilkg@users.noreply.github.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-16 14:08:29 -07:00
Martin Hickey
64175d3827
[Bugfix][Frontend] Keep the parser's tool call IDs in batch derender ( #61852 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-10-16 10:27:17 +00:00
024c5188bf
[Frontend] Resolve logprobs on the parsed streaming derender path ( #60437 )
...
Signed-off-by: Shimi Bandiel <shimib@google.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
2026-10-16 18:04:15 +08:00
maithilijoshi20 and Cyrus Leung
92dc3dd393
[Bugfix] Honor OPT token embedding width for prompt embeddings ( #61736 )
...
Signed-off-by: maithilijoshi20 <maithilij2003@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-16 04:41:50 +00:00
3dcfaeb4e8
[Frontend] Apply each endpoint's top_logprobs cut in derender ( #59518 )
...
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com >
Co-authored-by: Yin Li <kevinli.ai.work@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-15 15:04:04 +00:00
b7b20b1b8f
[Bugfix][Frontend] Leave the final stop token out of chat derender text ( #61757 )
...
Signed-off-by: khluu <khluu000@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-10-15 21:51:34 +08:00
Flora Feng and Chauncey
80dd705c1d
[Frontend] Make finish_reason and Responses item status truthful ( #59324 )
...
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-10-15 10:12:02 +01:00
Bugen Zhao and Claude Opus 5.5
8fa5ae844a
[Frontend] Let an unset --tool-strict-level fall back to the tool parser's default ( #61328 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 15:48:38 +08:00
bf6c80c7e8
[Misc] Remove tool name from warn logs ( #59294 )
...
Signed-off-by: Benjamin Merkel <benjamin.merkel@tngtech.com >
Co-authored-by: Benjamin Merkel <benjamin.merkel@tngtech.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-10-14 08:03:11 -07:00
wang.yuqi
669f85e686
[Misc] Break chat_utils into more maintainable modules. ( #60972 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-10-13 15:08:07 +00:00
Juan Pérez de Algaba and OpenAI Codex
7173c64e2e
[Security] Bound decoded and downloaded media per request ( #61226 )
...
Signed-off-by: juanpas97 <juan.perezalgaba@gmail.com >
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-10-13 10:17:14 +00:00
8d7b93fec2
[Frontend] Report stop_reason on /inference/v1/generate and pass it through derender ( #60251 )
...
Signed-off-by: Hyeonki Hong <hhk7734@gmail.com >
Signed-off-by: Shimi Bandiel <shimib@google.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Hyeonki Hong <hhk7734@gmail.com >
Co-authored-by: Mimir <mimir@nousresearch.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
2026-10-12 19:37:53 +08:00
Ting SUN
f4114e068e
[Bugfix][Frontend] Preserve token IDs in chat derender responses ( #61065 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
2026-10-12 07:55:06 +00:00
wang.yuqi
c1692c85f1
[Misc] mv chat_utils.py to renderers. ( #60522 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-10-10 06:50:34 +00:00
1a3a7a8f01
[Bugfix] Treat max_tokens as output upper bound, not input-space reservation ( #57035 )
...
Signed-off-by: YuvrajSinghBhadoria2 <245782664+YuvrajSinghBhadoria2@users.noreply.github.com >
Signed-off-by: YuvrajSinghBhadoria2 <yuvrajsinghbhado2030@gmail.com >
Signed-off-by: Yuvraj Singh Bhadoria <yuvrajsinghbhado2030@gmail.com >
Co-authored-by: YuvrajSinghBhadoria2 <245782664+YuvrajSinghBhadoria2@users.noreply.github.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-10-10 01:15:22 +08:00
Harry Mellor and Claude Opus 5.5
21d9758459
Remove Python 3.10 support now that it is EOL ( #60402 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 02:30:52 -07:00
Junyi Yao and sfeng33
fdfc171a74
[Bugfix][ToolParser] Stream every tool call when one delta completes several ( #60354 )
...
Signed-off-by: Junyi Yao <j.yao@wustl.edu >
Signed-off-by: sfeng33 <4florafeng@gmail.com >
Co-authored-by: sfeng33 <4florafeng@gmail.com >
2026-10-08 01:07:35 +00:00
7b665f7a58
[Frontend] Integer token IDs for generate output logprobs (GenerateLogProbs) ( #58181 )
...
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com >
Co-authored-by: Yin Li <kevinli.ai.work@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-06 17:22:47 +00:00
Martin Hickey and Flora Feng
3fb1cd796b
[Frontend] Extract parse and assembly out of batch chat derender ( #60073 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-05 21:57:45 +00:00
Yida Weng and Flora Feng
18c8a65edd
[BugFix][Frontend] Pass reasoning_ended through render → generate ( #60059 ) ( #60062 )
...
Signed-off-by: Yida Weng <84164537+YidaWeng@users.noreply.github.com >
Signed-off-by: Flora Feng <4florafeng@gmail.com >
Co-authored-by: Flora Feng <4florafeng@gmail.com >
2026-10-05 20:39:25 +00:00
Martin Hickey
bc21cba967
[Frontend] Add output_mode to /inference/v1/generate (RFC #56851 Phase 1) ( #58588 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-10-03 19:38:25 +08:00
Martin Hickey
fcfacc3499
[Bugfix][Frontend] Seed derender detokenization from the prompt ( #59046 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-10-02 03:17:56 +00:00
9bd993564f
[Frontend] Parse tool calls and reasoning from checkpoint response templates ( #58604 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Signed-off-by: Yoni Gozlan <yonigozlan@users.noreply.github.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-10-02 02:33:00 +00:00
Shijie Lyu
9aaef4296f
[Bugfix][Frontend] Check reused prompt token ids against the vocab before streaming ( #59555 )
...
Signed-off-by: Shijie Lyu <lshjhf@gmail.com >
2026-10-01 09:26:55 -07:00
Juan Pérez de Algaba and Cursor
cff08b461e
[Security] Fix chat template resource-exhaustion DoS (GHSA-4hhp-h66f-… ( #50300 )
...
Signed-off-by: jperezde <jperezde@redhat.com >
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-30 08:32:00 -07:00
Shijie Lyu
ac68c30872
[Bugfix][Frontend] Ignore reused prompt token ids for media in /v1/responses ( #59173 )
...
Signed-off-by: Shijie Lyu <lshjhf@gmail.com >
2026-09-30 05:21:09 +00:00
Shimi Bandiel and Claude
4dc57d44c4
[Frontend] Attach resolved logprobs to streaming derender chunks ( #55029 )
...
Signed-off-by: Shimi Bandiel <shimib@google.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-29 18:43:44 +00:00
c6843d397c
[Bugfix][Frontend] Validate reused prompt token ids; render messages for media and echo ( #55771 )
...
Signed-off-by: Shijie Lyu <sjlyu@amazon.com >
Co-authored-by: Shijie Lyu <sjlyu@amazon.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-29 10:36:22 +08:00
cd63bf9996
[Bugfix] Use the correct repository revision for secondary artifact loaders ( #57461 )
...
Signed-off-by: Clinton Thomas <1033162+KernelClint@users.noreply.github.com >
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-28 14:25:40 +00:00
Flora Feng
6f65fde419
[Bugfix][Frontend] Apply Harmony adjust_request in batched chat completions ( #58958 )
...
Signed-off-by: sfeng33 <4florafeng@gmail.com >
2026-09-28 04:12:04 +00:00
Juan Pérez de Algaba and Cursor
31f2e70cd3
[Security] Gate per-request multimodal processor kwargs ( #58830 )
...
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-26 11:10:35 +00:00
shallow10
3a58aeb1a1
[Bugfix] Stop leaking the internal field name in the max_tokens validation error ( #58336 )
...
Signed-off-by: shallow10 <495593563@qq.com >
2026-09-24 21:55:16 -07:00
Harry Mellor and Claude Opus 5.5
d636456b1f
[Frontend] Remove the slow tokenizer mode ( #58545 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-09-24 08:48:28 -07:00
SHREY GAJJAR
5747d4500a
[Perf][Frontend] Offload streaming derender detokenization ( #57528 )
...
Signed-off-by: Shrey Gajjar <shreygajjar007@gmail.com >
2026-09-24 10:39:21 +00:00
d95a896ea6
[Feature][Frontend] Add DeepSeek-V4 FIM completion rendering ( #44229 )
...
Signed-off-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com >
Co-authored-by: QwertyJack <7554089+QwertyJack@users.noreply.github.com >
Co-authored-by: Chauncey <chaunceyjiang@gmail.com >
2026-09-23 13:02:00 +00:00
Cyrus Leung
c8824a1e42
[MM] Move supports_multimodal_inputs and cache out of registry ( #57913 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-09-21 07:43:07 -07:00
10e6a7f210
[Frontend] Expose per-request spec decode metrics in generate API ( #43310 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <codex@openai.com >
2026-09-20 12:21:33 +00:00
Dilber
27b7757f63
[Bugfix][Frontend] Bound the prompt after multimodal expansion ( #57076 )
...
Signed-off-by: Dilber P Shakir <dilbersha@gmail.com >
2026-09-20 08:52:27 +00:00
Cyrus Leung
751f6807d9
[Refactor] Use dynamic MM cache in processor ( #57674 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-09-19 15:38:00 +00:00
Cyrus Leung
56cac80b14
[Deprecation] Remove assistant_token_mask support ( #57520 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-09-19 00:17:51 +08:00
Martin Hickey
70164bdadc
[Frontend] Add stream reasoning and tool calls from the derender endpoint ( #50550 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-09-18 16:36:27 +02:00
cc09352e5c
[Bugfix][Multimodal] Preserve DeepSeek V4 image block spacing ( #56882 )
...
Signed-off-by: Xucheng Zhou <aden1350@outlook.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-18 18:44:39 +08:00
dee37d8911
[Frontend] Add --tool-strict-level for server-side control for structural tag activation ( #56268 )
...
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: lazymio <mio@lazym.io >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
2026-09-18 04:32:39 +00:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00