iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
23,331 Commits 3 Branches 0 Tags
6a95f62e2fbfebca86ec1e476e3b2e6fd8a574ab
Commit Graph
1329 Commits
Author SHA1 Message Date
MOVZXandHinata e61dcb9143 [Mini-vLLM] SM86-only cut: models, backends, Rust, tests 
- Keep only Qwen3.5, Qwen3.5-MTP and Qwen4Exp model paths; trim the model registry to 41 entries
- Delete idle attention backends (MLA, FlashInfer, ROCm-only, quant backends) and dead GDN CuTeDSL kernels
- Remove inc/quark quant loaders; keep compressed-tensors and Marlin
- Drop the Rust frontend, pooling entrypoints, and idle test suites
- Rebuild kernels for sm_80;sm_86 and strip the extensions
- Verified: full import sweep clean (optional deps only) and a dummy-load Qwen3.5 engine run on RTX 3090 via the Triton GDN path

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
 2026-10-18 19:19:56 +07:00
dskwe b05ede5616 [Bugfix][Frontend] Preserve token IDs in completion derender responses (#62114) 
Signed-off-by: Yuan Li <dskwelmcy@163.com>
 2026-10-17 08:27:48 +00:00
JovanandOpenAI Codex 5be4266f5c [Bugfix][Frontend] Preserve Cohere and Anthropic streaming errors (#61947) 
Signed-off-by: Kevin <jia__kuan@163.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-10-17 06:10:16 +00:00
Shuolei Wangandaoshen02 b3a7e2ff6e [RL][Frontend] Return weight-version spans on OpenAI completions and chat (#61347) 
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
 2026-10-17 02:08:07 +00:00
Martin Hickey 64175d3827 [Bugfix][Frontend] Keep the parser's tool call IDs in batch derender (#61852) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-10-16 10:27:17 +00:00
alpandmottopanikeiku 4ab99ff136 [Bugfix][Frontend] Honor the constructor chat template in offline scoring (#61990) 
Signed-off-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com>
Co-authored-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com>
 2026-10-16 10:20:46 +00:00
Shimi BandielClaudewang.yuqiKevin H. Luu
 		024c5188bf [Frontend] Resolve logprobs on the parsed streaming derender path (#60437) 
Signed-off-by: Shimi Bandiel <shimib@google.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
 2026-10-16 18:04:15 +08:00
maithilijoshi20andCyrus Leung 92dc3dd393 [Bugfix] Honor OPT token embedding width for prompt embeddings (#61736) 
Signed-off-by: maithilijoshi20 <maithilij2003@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-10-16 04:41:50 +00:00
Jovan a9e90b9d1d [Bugfix][Frontend] Reject Cohere REQUIRED tool choice without tools (#61825) 
Signed-off-by: Kevin <jia__kuan@163.com>
 2026-10-15 16:37:08 +00:00
Yin LiYin LiCyrus Leung
 		3dcfaeb4e8 [Frontend] Apply each endpoint's top_logprobs cut in derender (#59518) 
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com>
Co-authored-by: Yin Li <kevinli.ai.work@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-10-15 15:04:04 +00:00
Kevin H. LuuClaude Opus 5.5Cyrus Leung
 		b7b20b1b8f [Bugfix][Frontend] Leave the final stop token out of chat derender text (#61757) 
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-10-15 21:51:34 +08:00
Martin Hickey 252be28c18 [Frontend] Replace return_mm_kwargs with an mm_payload enum on render requests (#61427) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-10-15 12:41:46 +00:00
Martin Hickey c6b772cf6b [Bugfix][Frontend] Emit kv/ec transfer params at the top level of render output (#61241) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-10-15 12:34:15 +00:00
Talor Abramovich 81e42c9987 [Feature][Frontend] Control the log level of request JSON body using env var (#60630) 
Signed-off-by: talora <talora@nvidia.com>
 2026-10-15 11:07:19 +00:00
Flora FengandChauncey 80dd705c1d [Frontend] Make finish_reason and Responses item status truthful (#59324) 
Signed-off-by: Flora Feng <4florafeng@gmail.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
 2026-10-15 10:12:02 +01:00
FeanorandTaneem Ibrahim 5d5c3d7c43 [Bugfix] Preserve padding token types for cross encoders (#58143) 
Signed-off-by: zhaoye <772971548@qq.com>
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
 2026-10-15 03:09:24 +00:00
alpandmottopanikeiku e73fa00981 [Bugfix][Frontend] Apply the JSON schema in Cohere Chat v2 response_format (#60586) 
Signed-off-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com>
Co-authored-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com>
 2026-10-14 19:42:20 +00:00
Robert ShawandClaude d3f6b2a7b3 [Frontend] Clean Client Facing 500 Report (#58043) 
Co-authored-by: Claude <noreply@anthropic.com>
 2026-10-14 14:27:58 -04:00
Xunzhuo 89cab55086 [Model] Support Decision 3.0 decision models (vllm-sr/d3) (#61565) 
Signed-off-by: Xunzhuo Liu <xunzhuo.liu@amd.com>
 2026-10-14 17:10:55 +00:00
AlecClaude Opus 5.5Bugen Zhao
 		aa0c433a3e [Frontend] Accept --reasoning-parser none as the unset default (#61525) 
Signed-off-by: Alec Flowers <aflowers@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
 2026-10-14 15:50:35 +08:00
Kevin H. LuuClaude Opus 5.5Bugen Zhao
 		ec317a4618 [CI] Keep vLLM log propagation on through the first LLM init in caplog_vllm (#61535) 
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
 2026-10-14 15:10:35 +08:00
Francisco Javier ArceoCodexMisha Goin
 		c923e7769b [Feat] Add text-only OpenAI Decisions API MVP (#60465) 
Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
 2026-10-14 07:51:08 +08:00
Kevin H. LuuandCodex 6a03954b45 [CI] Shard quantization, pooling, multimodal APIs, core kernels and PCP eval automatically (#61477) 
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-13 16:15:30 -07:00
GuanLuoandsfeng33 6d1daa9fe5 [Bugfix][Frontend] Expose completion response schemas in OpenAPI (#60926) 
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: sfeng33 <4florafeng@gmail.com>
 2026-10-13 18:54:06 +00:00
wang.yuqi 669f85e686 [Misc] Break chat_utils into more maintainable modules. (#60972) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-10-13 15:08:07 +00:00
Linze Shi e6e84f350c [Feature][Pooling] Add offline fixed-length late chunking (#60164) 
Signed-off-by: Linze-Shi <linzeshi0@gmail.com>
 2026-10-13 11:16:21 +00:00
Tianyu GuoCodexIsotr0py
 		380638afbb [Model] Add Clef-Flash support (#61308) 
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
 2026-10-13 05:46:37 +00:00
xinnywinneBugen ZhaoClaude Opus 5.5
 		1bc6e072b0 [Rust Frontend] Add per-request timing metrics to chat and completions (#59314) 
Signed-off-by: wxin <wxin@nvidia.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-12 15:39:10 -07:00
Martin Hickey 7039c7f17b [Bugfix][Rust Frontend] Omit unset sampling params in render output (#61220) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-10-12 19:31:03 +00:00
Martin Hickey 08582942d8 [Bugfix][Frontend] Honor request_id and X-Request-Id in chat and completions render (#61236) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-10-12 19:17:12 +00:00
Chester c599b5971f [Bugfix][Frontend] Return all logprobs when completions requests logprobs=-1 (#55717) 
Signed-off-by: Chester <42577861+superdiaodiao@users.noreply.github.com>
 2026-10-12 17:05:55 +00:00
Misha GoinSimon VeitnerClaude
 		fa39582756 [Frontend] Serve LiquidAI/d1-3B on /v1/systemone (#60676) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-10-12 13:00:47 -04:00
bzsuni f022975964 [Bugfix] Reject beam search in completion render (#61212) 
Signed-off-by: bzsuni <bingzhe.sun@daocloud.io>
 2026-10-12 13:49:46 +00:00
Jiangyun ZhuClaude Opus 5.5Isotr0py
 		092c35dd15 [Model] Support Cloudflare/clef-omni on /v1/systemone (#61064) 
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
 2026-10-12 21:27:34 +08:00
Shimi BandielHyeonki HongMimirClaudeBugen Zhao
 		8d7b93fec2 [Frontend] Report stop_reason on /inference/v1/generate and pass it through derender (#60251) 
Signed-off-by: Hyeonki Hong <hhk7734@gmail.com>
Signed-off-by: Shimi Bandiel <shimib@google.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Hyeonki Hong <hhk7734@gmail.com>
Co-authored-by: Mimir <mimir@nousresearch.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
 2026-10-12 19:37:53 +08:00
Shijin ZhangandChauncey 83773ecdf3 [Frontend] Support CustomToolCall in the non-Harmony Responses API (#57322) 
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
 2026-10-12 17:22:55 +08:00
Thuongvu HoandNicolò Lucchesi 99b44c7a42 [Metrics] Add admission rejection metrics (#55812) 
Signed-off-by: Thuongvu Ho <thuongvuho@gmail.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
 2026-10-12 08:31:07 +00:00
Ting SUN f4114e068e [Bugfix][Frontend] Preserve token IDs in chat derender responses (#61065) 
Signed-off-by: Ting Sun <suntcrick@gmail.com>
 2026-10-12 07:55:06 +00:00
Ting SUNandCyrus Leung b99fdbe1f8 [Bugfix] Respect the configured offline chat template (#61061) 
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-10-11 16:27:21 +00:00
aoshen02 bd5bbce6e4 [RL] Release weight-transfer communicators without stopping inference (#61060) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: fh3 <a@b>
 2026-10-11 21:15:14 +08:00
Subhash Polisetti f425a21208 [Bugfix][Frontend] Support scalar priority and wire priority into LLM.chat() (#57243) 
Signed-off-by: subhash polisetti <subhashr161347@gmail.com>
 2026-10-11 10:03:33 +00:00
aoshen02andOpenAI Codex 1789f09588 [Frontend] Explain disabled scale-out endpoints with opt-in hints (#61042) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-10-11 08:58:07 +00:00
Juan Pérez de AlgabaandCyrus Leung fcae5c9887 [Security] Restrict default CORS origins (#61009) 
Signed-off-by: Juan Pérez de Algaba <jperezde@redhat.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-10-11 08:26:02 +00:00
Kay YanandCodex c41b2639e2 [Bugfix][Responses API] Complete cache-write token accounting (#60973) 
Signed-off-by: Kay Yan <kay.yan@daocloud.io>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-10 09:08:36 +00:00
wang.yuqi c1692c85f1 [Misc] mv chat_utils.py to renderers. (#60522) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-10-10 06:50:34 +00:00
Chauncey fa5cfd48d5 [Refactor] Deduplicate trace header handling in BaseServing (#60828) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-10-10 05:27:58 +00:00
Misha GoinandClaude Opus 5.5 29e4d2d42e [Frontend][Build] Add vllm download-kernels for FlashInfer precompiled kernels (#58765) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 21:22:35 -07:00
Matt MastracciClaude Opus 5.5Simon VeitnerMisha Goin
 		3c802af3cf [Feat][Frontend] structured decisions: DiffusionGemma read strategy (#60466) 
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
 2026-10-10 09:33:22 +08:00
premsurawutandClaude Opus 5.5 c811943137 [Bugfix][Frontend] Reject beam search with echo and logprobs in /v1/completions (#60876) 
Signed-off-by: surawut.j <surawut.jirasaktavee@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 18:06:26 +00:00
Yuvraj Singh BhadoriaYuvrajSinghBhadoria2Cyrus LeungClaude Opus 4.8
 		1a3a7a8f01 [Bugfix] Treat max_tokens as output upper bound, not input-space reservation (#57035) 
Signed-off-by: YuvrajSinghBhadoria2 <245782664+YuvrajSinghBhadoria2@users.noreply.github.com>
Signed-off-by: YuvrajSinghBhadoria2 <yuvrajsinghbhado2030@gmail.com>
Signed-off-by: Yuvraj Singh Bhadoria <yuvrajsinghbhado2030@gmail.com>
Co-authored-by: YuvrajSinghBhadoria2 <245782664+YuvrajSinghBhadoria2@users.noreply.github.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-10-10 01:15:22 +08:00