iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[Docs] Update structured decisions docs. (#61588)

Browse Source 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
This commit is contained in:
wang.yuqi
2026-10-14 10:28:22 +00:00
committed by GitHub
parent 19aba3e4d5
commit 6ecb39eaa7
5 changed files with 135 additions and 99 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/.nav.yml
+1
View File
@@ -26,6 +26,7 @@ nav:
- models/supported_models.md
- models/generative_models.md
- Pooling Models: models/pooling_models
- models/structured_decisions_models.md
- models/extensions
- Hardware Supported Models:
- models/hardware_supported_models/*
docs/serving/online_serving/structured_decisions.md → docs/models/structured_decisions_models.md
+130 -27
View File
@@ -1,4 +1,34 @@
# Structured Decisions
# Structured Decisions Models
Structured Decisions Models (aka JEV Models) map inputs directly to predefined
options or categories and return probabilities and confidence scores, commonly
used for classification and routing.
## Supported Models
### Text-only Models
| Architecture | ReadStrategy | Task | Head | Example HF Models |
|---------------------------|---------------------|------------------|-------------------|-------------------------|
| `ClefFlashForJointSchema` | `ClefFlashStrategy` | `token_classify` | `JointSchemaHead` | `Cloudflare/clef-flash` |
| `Qwen3ForCausalLM` | `NextTokenStrategy` | `generate` | N/A | `Qwen/Qwen3-0.6B`, etc. |
### Multimodal Models
!!! note
For more information about multimodal models inputs, see [this page](./supported_models.md#list-of-multimodal-language-models).
| Architecture | Inputs | ReadStrategy | Task | Head | Example HF Models |
|--------------------------------------|-------------------------------------|--------------------------------|------------------|-------------------|------------------------------------------|
| `ClefOmniForJointSchema` | T + I + V | `JointSchemaStrategy` | `token_classify` | `JointSchemaHead` | `Cloudflare/clef-omni` |
| `Lfm2VlForConditionalGeneration` | T + I<sup>+</sup> | `LiquidStrategy` | `generate` | N/A | `LiquidAI/d1-3B` |
| `Qwen3_5ForConditionalGeneration` | T + I<sup>E+</sup> + V<sup>E+</sup> | `NextTokenStrategy` | `generate` | N/A | `Qwen/Qwen3.5-0.8B`, etc. |
| `Qwen3_5MoeForConditionalGeneration` | T + I<sup>E+</sup> + V<sup>E+</sup> | `NextTokenStrategy` | `generate` | N/A | `Qwen/Qwen3.5-35B-A3B`, etc. |
| `DiffusionGemmaForBlockDiffusion` | T + I<sup>+</sup> + V<sup>+</sup> | `DiffusionGemmaCanvasStrategy` | `generate` | N/A | `google/diffusiongemma-26B-A4B-it`, etc. |
## Online Serving
### TypeSafe System One Endpoint
The `/v1/systemone` endpoint answers a set of typed questions about a state and
returns a probability for every allowed answer.
@@ -8,7 +38,7 @@ supported read strategy return 501. Supported architectures require
`--logprobs-mode raw_logprobs` or `processed_logprobs`, since `label_mass` needs
log probabilities.
## How it works
#### How it works
1. Each question is one read. The user message is the state, followed by the
question and its options with the type's labels, and an instruction to
@@ -29,7 +59,7 @@ Every read of a request starts with the state, so with
`--enable-prefix-caching` the state is prefilled once and each further
question prefills only its own text, then reads one token.
## Question types
#### Question types
| type | criteria | answer |
| --- | --- | --- |
@@ -40,7 +70,7 @@ question prefills only its own text, then reads one token.
A choice has at least one option, a score at least two levels. A question id
is a non-empty string.
## Example
#### Example
```bash
curl -s http://localhost:8000/v1/systemone \
@@ -116,13 +146,13 @@ its whole vocabulary. A low value means most of the probability went to tokens
that are not labels. `confidence` is the chosen option's probability over the
whole vocabulary: its share of the labels times `label_mass`.
## Labels
#### Labels
Labels follow the type: a `choice` labels its options `A` to `Z` in the order
the request lists them, a `noul` labels its options `yes` and `no`, and a
`score` labels its levels `0` to `9`, so a level's label is its score.
## Limits
#### Limits
| limit | value |
| --- | --- |
@@ -130,7 +160,98 @@ the request lists them, a `noul` labels its options `yes` and `no`, and a
| options per `choice` | 26 |
| levels per `score` | 10 |
## Clef-Flash
#### Request fields
| field | meaning |
| --- | --- |
| `state` | what the questions are about: a string, or JSON that is sent as its JSON text |
| `questions` | question id to `{type, instructions, criteria}`, asked in this order |
| `instructions` | optional context placed ahead of the questions |
| `chat_template_kwargs` | passed to the chat template, for example `{"enable_thinking": false}` |
| `cache_salt` | optional [prefix-cache salt](../usage/security.md#cache-salting) applied to every question |
| `images`, `audio`, `videos` | optional media URLs or data URLs, for models that take them ([Clef-Flash](#clef-flash), images/video; [Clef-Omni](#clef-omni); [Liquid d1](#liquid-d1), images only) |
A question with any field other than `type`, `instructions` and `criteria` is
rejected with a 400.
When [tracing](../../examples/observability/opentelemetry/README.md) is enabled
with `--otlp-traces-endpoint`, the HTTP `traceparent` and `tracestate` headers are
forwarded to every question's generation request.
### OpenAI Decisions API
vLLM provides a text-only implementation of OpenAI's
[`POST /v1/decisions`](https://developers.openai.com/api/docs/guides/decisions)
API. Start a supported model:
```bash
vllm serve Qwen/Qwen3-0.6B
```
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
decision = client.decisions.create(
model="Qwen/Qwen3-0.6B",
input="The package arrived with a broken screen.",
questions=[
{
"type": "predicate",
"name": "damaged",
"instructions": "Does the customer report a damaged item?",
},
{
"type": "choice",
"name": "department",
"instructions": "Which department should handle this complaint?",
"choices": [{"value": "returns"}, {"value": "billing"}],
},
{
"type": "score",
"instructions": "How severe is the damage?",
"levels": [{"label": "cosmetic"}, {"label": "broken"}],
},
],
)
print(decision.answers)
```
Use OpenAI Python SDK 3.26.0 or newer for `client.decisions.create`.
`input` accepts a string or an array of user messages with string content or
`input_text` parts. Answers retain question order and echo `name`, including
`null` for unnamed questions. Choice values preserve strings and booleans.
Scores are probability-weighted averages of zero-based level indices.
The request contract follows the OpenAI OpenAPI schemas retrieved on
October 7, 2026. This MVP supports 1–64 questions, 2–26 choices, and 2–10 score
levels. Upstream permits 200 questions and 255 choices; this backend returns
a 400 when either model limit is exceeded.
The optional `safety_identifier` is caller metadata, available in request
body logging with `--enable-log-requests` and DEBUG logging; it does not
authenticate callers.
#### Implementation and limits
This MVP uses the same label-reading backend as
[structured decisions](structured_decisions_models.md). It supports generation
architectures with a registered read strategy, including Qwen, DiffusionGemma,
and Liquid d1; unsupported models return 501. Each question reads a model's
label tokens. Usage reports the actual prompt and output tokens across these
reads, including prefix-cache counters.
Probabilities are normalized over the supplied options. `confidence` is the
winning label's probability over the full vocabulary; it is not calibrated
to OpenAI's models. Thinking is disabled for these reads.
Image inputs, other message roles, tools, streaming, and per-question refusal
scoring are outside this MVP. Unsupported request fields and input types are
rejected. The `/v1/systemone` endpoint retains its existing request format and
is available by default.
## Examples
### Clef-Flash
[Clef-Flash](https://huggingface.co/Cloudflare/clef-flash) uses a Qwen3.5-9B
backbone and a joint schema head to score all questions in one forward pass.
@@ -154,7 +275,7 @@ examples above. `choice`, `noul` and `score` questions are supported.
- Each request performs one pooling read. `usage.input_tokens` counts the
prompt once, `output_tokens` is 0, and `label_mass` is 1.
## Clef-Omni
### Clef-Omni
[Clef-Omni](https://huggingface.co/Cloudflare/clef-omni) is a decision model:
a joint schema head on the Qwen3-Omni thinker scores every option of every
@@ -180,7 +301,7 @@ vllm serve Cloudflare/clef-omni
with it when every video has one. Audio clips cannot be sent together with
videos that have soundtracks.
## Liquid d1
### Liquid d1
[`LiquidAI/d1-3B`](https://huggingface.co/LiquidAI/d1-3B) is LFM2.5-VL-3B
post-trained to answer decisions as the first token of its reply. vLLM serves
@@ -202,21 +323,3 @@ vllm serve LiquidAI/d1-3B
letters.
- When each request carries one question over a state that changes from
request to request, `--no-enable-prefix-caching` lowers latency.
## Request fields
| field | meaning |
| --- | --- |
| `state` | what the questions are about: a string, or JSON that is sent as its JSON text |
| `questions` | question id to `{type, instructions, criteria}`, asked in this order |
| `instructions` | optional context placed ahead of the questions |
| `chat_template_kwargs` | passed to the chat template, for example `{"enable_thinking": false}` |
| `cache_salt` | optional [prefix-cache salt](../../usage/security.md#cache-salting) applied to every question |
| `images`, `audio`, `videos` | optional media URLs or data URLs, for models that take them ([Clef-Flash](#clef-flash), images/video; [Clef-Omni](#clef-omni); [Liquid d1](#liquid-d1), images only) |
A question with any field other than `type`, `instructions` and `criteria` is
rejected with a 400.
When [tracing](../../../examples/observability/opentelemetry/README.md) is enabled
with `--otlp-traces-endpoint`, the HTTP `traceparent` and `tracestate` headers are
forwarded to every question's generation request.
docs/serving/online_serving/README.md
+2 -2
View File
@@ -16,7 +16,7 @@ We currently support the following OpenAI APIs:
- [Chat Completions batch API](./openai_compatible_server.md#chat-api) (`/v1/chat/completions/batch`)
- [Responses API](./openai_compatible_server.md#responses-api) (`/v1/responses`, `/v1/responses/{response_id}`, `/v1/responses/{response_id}/cancel`)
- Only applicable to [text generation models](../../models/generative_models.md).
- [Decisions API](decisions.md) (`/v1/decisions`)
- [Decisions API](../../models/structured_decisions_models.md#openai-decisions-api) (`/v1/decisions`)
- Supports text-only predicate, choice, and score questions on supported generation models.
- [Embeddings API](../../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`)
- Only applicable to [embedding models](../../models/pooling_models/embed.md).
@@ -78,7 +78,7 @@ For further details on speech to text, please refer to [this page](speech_to_tex
- [Generative Scoring API](generative_scoring.md) (`/generative_scoring`)
- Applicable to [CausalLM models](../../models/generative_models.md) (task `"generate"`).
- Computes next-token probabilities for specified `label_token_ids`.
- [Structured Decisions API](structured_decisions.md) (`/v1/systemone`)
- [Structured Decisions API](../../models/structured_decisions_models.md#typesafe-system-one-endpoint) (`/v1/systemone`)
- Applicable to [text generation models](../../models/generative_models.md) with a [chat template](#chat-template).
- Answers typed questions about a state with a probability for every allowed answer.
docs/serving/online_serving/decisions.md
-70
View File
@@ -1,70 +0,0 @@
# Decisions API
vLLM provides a text-only implementation of OpenAI's
[`POST /v1/decisions`](https://developers.openai.com/api/docs/guides/decisions)
API. Start a supported model:
```bash
vllm serve Qwen/Qwen3-0.6B
```
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
decision = client.decisions.create(
model="Qwen/Qwen3-0.6B",
input="The package arrived with a broken screen.",
questions=[
{
"type": "predicate",
"name": "damaged",
"instructions": "Does the customer report a damaged item?",
},
{
"type": "choice",
"name": "department",
"instructions": "Which department should handle this complaint?",
"choices": [{"value": "returns"}, {"value": "billing"}],
},
{
"type": "score",
"instructions": "How severe is the damage?",
"levels": [{"label": "cosmetic"}, {"label": "broken"}],
},
],
)
print(decision.answers)
```
Use OpenAI Python SDK 3.26.0 or newer for `client.decisions.create`.
`input` accepts a string or an array of user messages with string content or
`input_text` parts. Answers retain question order and echo `name`, including
`null` for unnamed questions. Choice values preserve strings and booleans.
Scores are probability-weighted averages of zero-based level indices.
The request contract follows the OpenAI OpenAPI schemas retrieved on
October 7, 2026. This MVP supports 1–64 questions, 2–26 choices, and 2–10 score
levels. Upstream permits 200 questions and 255 choices; this backend returns
a 400 when either model limit is exceeded.
The optional `safety_identifier` is caller metadata, available in request
body logging with `--enable-log-requests` and DEBUG logging; it does not
authenticate callers.
## Implementation and limits
This MVP uses the same label-reading backend as
[structured decisions](structured_decisions.md). It supports generation
architectures with a registered read strategy, including Qwen, DiffusionGemma,
and Liquid d1; unsupported models return 501. Each question reads a model's
label tokens. Usage reports the actual prompt and output tokens across these
reads, including prefix-cache counters.
Probabilities are normalized over the supplied options. `confidence` is the
winning label's probability over the full vocabulary; it is not calibrated
to OpenAI's models. Thinking is disabled for these reads.
Image inputs, other message roles, tools, streaming, and per-question refusal
scoring are outside this MVP. Unsupported request fields and input types are
rejected. The `/v1/systemone` endpoint retains its existing request format and
is available by default.
mkdocs.yaml
+2
View File
@@ -104,6 +104,8 @@ plugins:
examples/others/lmcache.md: examples/disaggregated/lmcache.md
examples/features/batch_invariance.md: examples/rl/batch_invariance.md
examples/features/pause_resume.md: examples/rl/pause_resume.md
serving/online_serving/decisions.md: models/structured_decisions_models.md
serving/online_serving/structured_decisions.md: models/structured_decisions_models.md
markdown_extensions:
- attr_list