iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
22,991 Commits 3 Branches 0 Tags
c19c263ac8ea3fd6e1d2d689f0e4e725855d89cf
Commit Graph
59 Commits
Author SHA1 Message Date
Martin Hickey 08582942d8 [Bugfix][Frontend] Honor request_id and X-Request-Id in chat and completions render (#61236) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-10-12 19:17:12 +00:00
bzsuni f022975964 [Bugfix] Reject beam search in completion render (#61212) 
Signed-off-by: bzsuni <bingzhe.sun@daocloud.io>
 2026-10-12 13:49:46 +00:00
Shimi BandielHyeonki HongMimirClaudeBugen Zhao
 		8d7b93fec2 [Frontend] Report stop_reason on /inference/v1/generate and pass it through derender (#60251) 
Signed-off-by: Hyeonki Hong <hhk7734@gmail.com>
Signed-off-by: Shimi Bandiel <shimib@google.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Hyeonki Hong <hhk7734@gmail.com>
Co-authored-by: Mimir <mimir@nousresearch.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
 2026-10-12 19:37:53 +08:00
aoshen02andOpenAI Codex 1789f09588 [Frontend] Explain disabled scale-out endpoints with opt-in hints (#61042) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-10-11 08:58:07 +00:00
Shuolei Wangandaoshen02 b89852aad0 [RL][Frontend] Return token-level weight-version spans on final outputs (#59582) 
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com>
Signed-off-by: Shuolei Wang <948904026@qq.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
 2026-10-11 11:33:29 +08:00
wang.yuqi c1692c85f1 [Misc] mv chat_utils.py to renderers. (#60522) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-10-10 06:50:34 +00:00
Maroon AyoubandClaude 8a03ccf7b3 [Frontend] Add return_mm_kwargs to render requests (#60728) 
Signed-off-by: Maroon Ayoub <mayoub@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-10-09 08:14:13 +00:00
Yin LiYin LiClaude
 		7b665f7a58 [Frontend] Integer token IDs for generate output logprobs (GenerateLogProbs) (#58181) 
Signed-off-by: Yin Li <kevinli.ai.work@gmail.com>
Co-authored-by: Yin Li <kevinli.ai.work@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-10-06 17:22:47 +00:00
Yida WengandFlora Feng 18c8a65edd [BugFix][Frontend] Pass reasoning_ended through render → generate (#60059) (#60062) 
Signed-off-by: Yida Weng <84164537+YidaWeng@users.noreply.github.com>
Signed-off-by: Flora Feng <4florafeng@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
 2026-10-05 20:39:25 +00:00
Ting SUNandFlora Feng 50b404e71e [Bugfix][Frontend] Preserve abort finish_reason for scale-out token streams (#47933) 
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Signed-off-by: Flora Feng <4florafeng@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
 2026-10-04 02:57:26 +00:00
Martin Hickey bc21cba967 [Frontend] Add output_mode to /inference/v1/generate (RFC #56851 Phase 1) (#58588) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-10-03 19:38:25 +08:00
Martin Hickey fcfacc3499 [Bugfix][Frontend] Seed derender detokenization from the prompt (#59046) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-10-02 03:17:56 +00:00
aoshen02Codexfh3Claude Fable 5.1
 		4a9d6f9572 [Bugfix] Preserve sampling masks in DELTA and TITO streaming outputs (#55935) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: fh3 <a@b>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: fh3 <a@b>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
 2026-10-01 14:43:38 +08:00
Clinton ThomasLucas BourtouleCyrus Leung
 		3d5625cc3c [Security] Authenticate shared-memory multimodal cache handles (#59357) 
Signed-off-by: Clinton Thomas <1033162+KernelClint@users.noreply.github.com>
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-09-30 05:16:26 -07:00
Shimi BandielandClaude 4dc57d44c4 [Frontend] Attach resolved logprobs to streaming derender chunks (#55029) 
Signed-off-by: Shimi Bandiel <shimib@google.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-09-29 18:43:44 +00:00
aoshen02aoshen02Claude Opus 5
 		b721a4c709 [Feature] Add fixed-token prefill scoring (#54335) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
 2026-09-28 07:50:09 -07:00
Robert ShawandRobert Shaw dfab504333 [Bugfix][Frontend] Detect Anthropic inline-system merge against the resolved chat template (#58754) 
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
 2026-09-26 14:00:56 +00:00
aoshen02 7f1a5398e9 Fix full logprobs in token-in/token-out responses (#58488) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
 2026-09-24 04:46:16 +00:00
Martin Hickey 8b98b7d0b4 [Frontend] Add streaming parity tests and docs for derender (#57922) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-09-21 14:13:52 +00:00
Hongxin Xuaoshen02Codex
 		10e6a7f210 [Frontend] Expose per-request spec decode metrics in generate API (#43310) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <codex@openai.com>
 2026-09-20 12:21:33 +00:00
Cyrus Leung 56cac80b14 [Deprecation] Remove assistant_token_mask support (#57520) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-09-19 00:17:51 +08:00
Martin Hickey 70164bdadc [Frontend] Add stream reasoning and tool calls from the derender endpoint (#50550) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-09-18 16:36:27 +02:00
Shimi BandielClaudeCyrus Leungmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		e8a2a0a8bb [Bugfix][Frontend] Reject stop strings on --tokens-only servers instead of silently ignoring them (#57058) 
Signed-off-by: Shimi Bandiel <shimib@google.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-09-17 22:02:00 +00:00
aoshen02 b978df3299 Revert "[Frontend] Omit absent fields from /inference/v1/generate stream chunks" (#57302) 2026-09-16 23:31:59 -07:00
aoshen02andClaude Fable 5.1 8538017f4b [Frontend] Omit absent fields from /inference/v1/generate stream chunks (#57264) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
 2026-09-17 05:58:56 +00:00
aoshen02 0983aef8da [Frontend] Return prompt metadata from /inference/v1/generate (#53187) 2026-09-16 15:29:07 -07:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 16:44:29 +00:00
Martin Hickey 934b1fcbcb [Frontend] Replace VLLM_ENABLE_SCALE_OUT_ENDPOINTS with --enable-scale-out (#55176) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-09-14 10:02:14 +00:00
shaohuaxiandCodex ae71862c51 [Bugfix][Frontend] Check EC requirements for each metadata item (#56070) 
Signed-off-by: 子华 <huaxi.shx@alibaba-inc.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-10 19:23:34 +00:00
Francisco Javier ArceoandOpenAI Codex 6a2a2bb02b [Frontend] Add stateless /v1/responses/render endpoint (#50195) 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-09-07 12:50:42 +00:00
zhouyouandCursor Grok 4.6 52358e6e19 [Frontend] Expose multimodal metadata for disaggregated prefill (#54659) 
Signed-off-by: zhouyou9505 <zhouyou9505@gmail.com>
Co-authored-by: Cursor Grok 4.6 <noreply@cursor.com>
 2026-09-06 14:46:52 +00:00
Clinton ThomasandLucas Bourtoule 1970f3ed4b Validate scale-out multimodal data before engine handoff (#51898) 
Signed-off-by: Clinton Thomas <1033162+KernelClint@users.noreply.github.com>
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com>
 2026-09-06 04:58:04 +00:00
Clinton ThomasLucas BourtouleCyrus Leung
 		e962733e08 [Security] Validate cache salts before they reach LMCache (#51444) 
Signed-off-by: Clinton Thomas <1033162+KernelClint@users.noreply.github.com>
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-09-05 03:56:06 +00:00
Francisco Javier Arceo c00091e026 [Frontend] Gate scale-out endpoints behind opt-in flag (#54579) 
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com>
 2026-09-02 14:31:14 +08:00
Nicolò Lucchesi dafbef15a1 [Core] Add max_num_queued_reqs and max_num_queued_tokens for queue size management (#49445) 
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
 2026-08-31 14:26:49 +00:00
wang.yuqiandCyrus Leung eeb549a74d [Frontend] Move engine/protocol.py out openai folder (#54492) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2026-08-31 10:13:50 +00:00
eligottsandOpenAI Codex 4ae172231c [Frontend] Forward cache salt for content parts (#54315) 
Signed-off-by: Eli Gotts <78387377+eligotts@users.noreply.github.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-08-30 23:29:53 -07:00
Taneem Ibrahim fe6db3ed5e [Bugfix] Validate scale-out transfer params (#54324) 
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
 2026-08-30 09:46:12 -04:00
Juan Pérez de Algaba 1dc464d426 [Bugfix] Bound cache_salt length to prevent DoS via scheduler CPU exhaustion (#54353) 
Signed-off-by: jperezde <jperezde@redhat.com>
 2026-08-30 04:28:16 +00:00
Konstantin DunasandOpenAI Codex c71f6f8a81 [Bugfix] Thread kv_transfer_params into engine for /inference/v1/generate (disagg) (#42644) 
Signed-off-by: hallerite <git@hallerite.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-08-26 08:42:52 -07:00
Hyunkyun Moon f25c2692fe [Frontend] Add /v1/messages/render endpoint for the Anthropic Messages API (#45803) 
Signed-off-by: HyunKyun Moon <mhg5303@gmail.com>
 2026-08-26 11:33:59 +08:00
Wei-Cheng (Wayne) Chiu b41dc4ec72 [Bugfix] Return HTTP 500 for non-streaming generate errors (#49195) 
Signed-off-by: WEI CHENG CHIU <waynehacking8@gmail.com>
 2026-08-21 09:07:17 -07:00
Quentin Gallouédec 92bdee05cb [Bugfix][Frontend] Return all choices from /inference/v1/generate when n > 1 (#52399) 
Signed-off-by: Quentin Gallouédec <quentin.gallouedec@huggingface.co>
 2026-08-19 20:48:45 +08:00
aoshen02OpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		7553aac77b [Frontend] Add routed-experts prompt offset (#51906) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-08-13 08:09:00 -07:00
vx120aoshen02CodexClaude Opus 4.6OpenAI Codex
 		50ba4bc6b2 [Feature] Mask Replay (#49577) 
Signed-off-by: vx120 <893600387@qq.com>
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-08-13 08:44:31 +00:00
Vinay R DamodaranandClaude Opus 5 3ee2df3033 [Frontend] Make model optional on all /derender request classes (#51463) 
Signed-off-by: Vinay Damodaran <vrdn@hey.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-12 04:48:58 +00:00
aoshen02Claude Opus 4.6mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		33584901ba [Frontend] Add content_parts to /inference/v1/generate for raw multim… (#51478) 
Signed-off-by: Allen Shen <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-08-11 10:00:12 +08:00
Karen Chung f57123aa2d feat(frontend): session id plumbing into requests (#48048) 
Signed-off-by: Karen Chung <karenc@nvidia.com>
 2026-08-03 22:18:58 +00:00
Cyrus Leung b3f97dae24 [Frontend] Require cache_salt to be non-empty via schema (#50816) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-08-03 06:51:27 +00:00