Cyrus Leung
c8824a1e42
[MM] Move supports_multimodal_inputs and cache out of registry ( #57913 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-09-21 07:43:07 -07:00
Martin Hickey
8b98b7d0b4
[Frontend] Add streaming parity tests and docs for derender ( #57922 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-09-21 14:13:52 +00:00
0b7f11a1ee
[AuxOutput] Add block-keyed storage for routed-expert outputs ( #45635 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-21 10:25:05 +00:00
10e6a7f210
[Frontend] Expose per-request spec decode metrics in generate API ( #43310 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <codex@openai.com >
2026-09-20 12:21:33 +00:00
Cyrus Leung
56cac80b14
[Deprecation] Remove assistant_token_mask support ( #57520 )
...
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-09-19 00:17:51 +08:00
Martin Hickey
70164bdadc
[Frontend] Add stream reasoning and tool calls from the derender endpoint ( #50550 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-09-18 16:36:27 +02:00
e8a2a0a8bb
[Bugfix][Frontend] Reject stop strings on --tokens-only servers instead of silently ignoring them ( #57058 )
...
Signed-off-by: Shimi Bandiel <shimib@google.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-17 22:02:00 +00:00
aoshen02
b978df3299
Revert "[Frontend] Omit absent fields from /inference/v1/generate stream chunks" ( #57302 )
2026-09-16 23:31:59 -07:00
aoshen02 and Claude Fable 5.1
8538017f4b
[Frontend] Omit absent fields from /inference/v1/generate stream chunks ( #57264 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-17 05:58:56 +00:00
aoshen02
0983aef8da
[Frontend] Return prompt metadata from /inference/v1/generate ( #53187 )
2026-09-16 15:29:07 -07:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
caf56ce9da
[CI] Pass the scale-out endpoint flag in EC E2E ( #56819 )
...
Signed-off-by: Kevin Luu <51931015+khluu@users.noreply.github.com >
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
2026-09-15 12:25:30 +08:00
Martin Hickey
934b1fcbcb
[Frontend] Replace VLLM_ENABLE_SCALE_OUT_ENDPOINTS with --enable-scale-out ( #55176 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-09-14 10:02:14 +00:00
shaohuaxi and Codex
ae71862c51
[Bugfix][Frontend] Check EC requirements for each metadata item ( #56070 )
...
Signed-off-by: 子华 <huaxi.shx@alibaba-inc.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-10 19:23:34 +00:00
9ffb8cea96
[CI] Add e2e test for scale-out EC connector flow ( #54973 )
...
Signed-off-by: zhouyou9505 <zhouyou9505@gmail.com >
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
Co-authored-by: zhouyou9505 <zhouyou9505@gmail.com >
Co-authored-by: Cursor Grok 4.6 <noreply@cursor.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-09-09 07:38:32 -07:00
Francisco Javier Arceo and OpenAI Codex
6a2a2bb02b
[Frontend] Add stateless /v1/responses/render endpoint ( #50195 )
...
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-07 12:50:42 +00:00
zhouyou and Cursor Grok 4.6
52358e6e19
[Frontend] Expose multimodal metadata for disaggregated prefill ( #54659 )
...
Signed-off-by: zhouyou9505 <zhouyou9505@gmail.com >
Co-authored-by: Cursor Grok 4.6 <noreply@cursor.com >
2026-09-06 14:46:52 +00:00
Clinton Thomas and Lucas Bourtoule
1970f3ed4b
Validate scale-out multimodal data before engine handoff ( #51898 )
...
Signed-off-by: Clinton Thomas <1033162+KernelClint@users.noreply.github.com >
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com >
2026-09-06 04:58:04 +00:00
560ef78bfe
[Perf][Model Runner V2] Compact sampling masks on GPU instead of unpacking the full-vocab bitmask on CPU ( #54901 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: vx120 <893600387@qq.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: vx120 <893600387@qq.com >
2026-09-03 19:03:36 -07:00
Francisco Javier Arceo
c00091e026
[Frontend] Gate scale-out endpoints behind opt-in flag ( #54579 )
...
Signed-off-by: Francisco Javier Arceo <farceo@redhat.com >
2026-09-02 14:31:14 +08:00
Hotragn Pettugani
ff0c3cb03c
[Bugfix][Frontend] Truncate the assistant tokens mask with the prompt ( #54539 )
...
Signed-off-by: hotragn <hotragn.pettugani_2024@woxsen.edu.in >
2026-09-01 05:53:22 +00:00
wang.yuqi and Cyrus Leung
eeb549a74d
[Frontend] Move engine/protocol.py out openai folder ( #54492 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com >
2026-08-31 10:13:50 +00:00
Hotragn Pettugani
8fd9eb85d5
[Bugfix][Frontend] Truncate prompt_token_offsets with the prompt ( #54407 )
...
Signed-off-by: hotragn <hotragn.pettugani_2024@woxsen.edu.in >
2026-08-31 06:13:37 +00:00
Hyunkyun Moon
f25c2692fe
[Frontend] Add /v1/messages/render endpoint for the Anthropic Messages API ( #45803 )
...
Signed-off-by: HyunKyun Moon <mhg5303@gmail.com >
2026-08-26 11:33:59 +08:00
Wei-Cheng (Wayne) Chiu
b41dc4ec72
[Bugfix] Return HTTP 500 for non-streaming generate errors ( #49195 )
...
Signed-off-by: WEI CHENG CHIU <waynehacking8@gmail.com >
2026-08-21 09:07:17 -07:00
Quentin Gallouédec
92bdee05cb
[Bugfix][Frontend] Return all choices from /inference/v1/generate when n > 1 ( #52399 )
...
Signed-off-by: Quentin Gallouédec <quentin.gallouedec@huggingface.co >
2026-08-19 20:48:45 +08:00
50ba4bc6b2
[Feature] Mask Replay ( #49577 )
...
Signed-off-by: vx120 <893600387@qq.com >
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-13 08:44:31 +00:00
Vinay R Damodaran and Claude Opus 5
3ee2df3033
[Frontend] Make model optional on all /derender request classes ( #51463 )
...
Signed-off-by: Vinay Damodaran <vrdn@hey.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-08-12 04:48:58 +00:00
Andreas Karatzas and OpenAI Codex
1ab2801dde
[ROCm] Remove stale SDPA and skinny GEMM workarounds ( #50907 )
...
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-11 14:19:16 -04:00
33584901ba
[Frontend] Add content_parts to /inference/v1/generate for raw multim… ( #51478 )
...
Signed-off-by: Allen Shen <aoshen@inferact.ai >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-11 10:00:12 +08:00
Karen Chung
f57123aa2d
feat(frontend): session id plumbing into requests ( #48048 )
...
Signed-off-by: Karen Chung <karenc@nvidia.com >
2026-08-03 22:18:58 +00:00
Martin Hickey
0a31372e5f
[Frontend] Add detokenization streaming derender for disaggregated serving ( #47301 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-07-29 18:01:20 -07:00
Martin Hickey
4d4e04f452
[Render] Add round trip parity test and docs for derender ( #48617 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-07-17 05:02:46 +00:00
Juan Pérez de Algaba and Cursor
b3e85be663
fix: use configured max_logprobs instead of hardcoded 20 in derender validation ( #47834 )
...
Signed-off-by: jperezde <jperezde@redhat.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-07-07 09:42:47 +00:00
Juan Pérez de Algaba and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
8e61b646e2
fix(security): add resource bounds validation to derender endpoints ( #47260 )
...
Signed-off-by: jperezde <jperezde@redhat.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-07 14:58:26 +08:00
wang.yuqi
3483240b7e
[Frontend] Consolidate scale out entrypoints ( #44512 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-06-29 03:18:53 -07:00
Cyrus Leung