[Docs] Update structured decisions docs. (#61588)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
This commit is contained in:
@@ -16,7 +16,7 @@ We currently support the following OpenAI APIs:
|
||||
- [Chat Completions batch API](./openai_compatible_server.md#chat-api) (`/v1/chat/completions/batch`)
|
||||
- [Responses API](./openai_compatible_server.md#responses-api) (`/v1/responses`, `/v1/responses/{response_id}`, `/v1/responses/{response_id}/cancel`)
|
||||
- Only applicable to [text generation models](../../models/generative_models.md).
|
||||
- [Decisions API](decisions.md) (`/v1/decisions`)
|
||||
- [Decisions API](../../models/structured_decisions_models.md#openai-decisions-api) (`/v1/decisions`)
|
||||
- Supports text-only predicate, choice, and score questions on supported generation models.
|
||||
- [Embeddings API](../../models/pooling_models/embed.md#openai-compatible-embeddings-api) (`/v1/embeddings`)
|
||||
- Only applicable to [embedding models](../../models/pooling_models/embed.md).
|
||||
@@ -78,7 +78,7 @@ For further details on speech to text, please refer to [this page](speech_to_tex
|
||||
- [Generative Scoring API](generative_scoring.md) (`/generative_scoring`)
|
||||
- Applicable to [CausalLM models](../../models/generative_models.md) (task `"generate"`).
|
||||
- Computes next-token probabilities for specified `label_token_ids`.
|
||||
- [Structured Decisions API](structured_decisions.md) (`/v1/systemone`)
|
||||
- [Structured Decisions API](../../models/structured_decisions_models.md#typesafe-system-one-endpoint) (`/v1/systemone`)
|
||||
- Applicable to [text generation models](../../models/generative_models.md) with a [chat template](#chat-template).
|
||||
- Answers typed questions about a state with a probability for every allowed answer.
|
||||
|
||||
|
||||
@@ -1,70 +0,0 @@
|
||||
# Decisions API
|
||||
|
||||
vLLM provides a text-only implementation of OpenAI's
|
||||
[`POST /v1/decisions`](https://developers.openai.com/api/docs/guides/decisions)
|
||||
API. Start a supported model:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-0.6B
|
||||
```
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
decision = client.decisions.create(
|
||||
model="Qwen/Qwen3-0.6B",
|
||||
input="The package arrived with a broken screen.",
|
||||
questions=[
|
||||
{
|
||||
"type": "predicate",
|
||||
"name": "damaged",
|
||||
"instructions": "Does the customer report a damaged item?",
|
||||
},
|
||||
{
|
||||
"type": "choice",
|
||||
"name": "department",
|
||||
"instructions": "Which department should handle this complaint?",
|
||||
"choices": [{"value": "returns"}, {"value": "billing"}],
|
||||
},
|
||||
{
|
||||
"type": "score",
|
||||
"instructions": "How severe is the damage?",
|
||||
"levels": [{"label": "cosmetic"}, {"label": "broken"}],
|
||||
},
|
||||
],
|
||||
)
|
||||
print(decision.answers)
|
||||
```
|
||||
|
||||
Use OpenAI Python SDK 3.26.0 or newer for `client.decisions.create`.
|
||||
`input` accepts a string or an array of user messages with string content or
|
||||
`input_text` parts. Answers retain question order and echo `name`, including
|
||||
`null` for unnamed questions. Choice values preserve strings and booleans.
|
||||
Scores are probability-weighted averages of zero-based level indices.
|
||||
|
||||
The request contract follows the OpenAI OpenAPI schemas retrieved on
|
||||
October 7, 2026. This MVP supports 1–64 questions, 2–26 choices, and 2–10 score
|
||||
levels. Upstream permits 200 questions and 255 choices; this backend returns
|
||||
a 400 when either model limit is exceeded.
|
||||
The optional `safety_identifier` is caller metadata, available in request
|
||||
body logging with `--enable-log-requests` and DEBUG logging; it does not
|
||||
authenticate callers.
|
||||
|
||||
## Implementation and limits
|
||||
|
||||
This MVP uses the same label-reading backend as
|
||||
[structured decisions](structured_decisions.md). It supports generation
|
||||
architectures with a registered read strategy, including Qwen, DiffusionGemma,
|
||||
and Liquid d1; unsupported models return 501. Each question reads a model's
|
||||
label tokens. Usage reports the actual prompt and output tokens across these
|
||||
reads, including prefix-cache counters.
|
||||
|
||||
Probabilities are normalized over the supplied options. `confidence` is the
|
||||
winning label's probability over the full vocabulary; it is not calibrated
|
||||
to OpenAI's models. Thinking is disabled for these reads.
|
||||
|
||||
Image inputs, other message roles, tools, streaming, and per-question refusal
|
||||
scoring are outside this MVP. Unsupported request fields and input types are
|
||||
rejected. The `/v1/systemone` endpoint retains its existing request format and
|
||||
is available by default.
|
||||
@@ -1,222 +0,0 @@
|
||||
# Structured Decisions
|
||||
|
||||
The `/v1/systemone` endpoint answers a set of typed questions about a state and
|
||||
returns a probability for every allowed answer.
|
||||
|
||||
The endpoint is available by default on generation models. Models without a
|
||||
supported read strategy return 501. Supported architectures require
|
||||
`--logprobs-mode raw_logprobs` or `processed_logprobs`, since `label_mass` needs
|
||||
log probabilities.
|
||||
|
||||
## How it works
|
||||
|
||||
1. Each question is one read. The user message is the state, followed by the
|
||||
question and its options with the type's labels, and an instruction to
|
||||
answer with one label.
|
||||
2. The read is the reply's first token. Each label must be one token there,
|
||||
or the request returns 400.
|
||||
3. The read returns the logprobs of that question's label tokens. Softmax over
|
||||
the labels gives the answer's probabilities.
|
||||
|
||||
Thinking is off unless `chat_template_kwargs` turns it on.
|
||||
|
||||
Probabilities can differ between identical requests. They depend on which
|
||||
requests share a batch and on whether a read hits the prefix cache. Set
|
||||
`VLLM_BATCH_INVARIANT=1` for reads that repeat exactly, at a cost in
|
||||
throughput.
|
||||
|
||||
Every read of a request starts with the state, so with
|
||||
`--enable-prefix-caching` the state is prefilled once and each further
|
||||
question prefills only its own text, then reads one token.
|
||||
|
||||
## Question types
|
||||
|
||||
| type | criteria | answer |
|
||||
| --- | --- | --- |
|
||||
| `choice` | map of option name to a description or `null` | `choice`, `probabilities` by option name, `confidence` |
|
||||
| `noul` | `null`, or an object with optional `true` and `false` descriptions | `noul` (probability of yes), `probabilities`, `confidence` |
|
||||
| `score` | ordered list of level names | `score` (expected 0-indexed level), `legend`, `probabilities` by level index, `confidence` |
|
||||
|
||||
A choice has at least one option, a score at least two levels. A question id
|
||||
is a non-empty string.
|
||||
|
||||
## Example
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:8000/v1/systemone \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"state": {"ticket": "My card was charged twice for one order."},
|
||||
"questions": {
|
||||
"team": {
|
||||
"type": "choice",
|
||||
"instructions": "Which team should handle this ticket?",
|
||||
"criteria": {
|
||||
"billing": "payments and refunds",
|
||||
"shipping": "deliveries",
|
||||
"security": "account access"
|
||||
}
|
||||
},
|
||||
"urgent": {
|
||||
"type": "noul",
|
||||
"instructions": "Does the customer need a reply within the hour?",
|
||||
"criteria": {"true": "the account is locked or money is moving"}
|
||||
},
|
||||
"tone": {
|
||||
"type": "score",
|
||||
"instructions": "How upset is the customer?",
|
||||
"criteria": ["calm", "annoyed", "furious"]
|
||||
}
|
||||
},
|
||||
"chat_template_kwargs": {"enable_thinking": false}
|
||||
}'
|
||||
```
|
||||
|
||||
Response, with probabilities that depend on the model:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "decision-...",
|
||||
"object": "structured_decision",
|
||||
"created": 1790000000,
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"answers": {
|
||||
"team": {
|
||||
"type": "choice",
|
||||
"choice": "billing",
|
||||
"probabilities": {"billing": 0.97, "shipping": 0.01, "security": 0.02},
|
||||
"confidence": 0.90
|
||||
},
|
||||
"urgent": {
|
||||
"type": "noul",
|
||||
"noul": 0.02,
|
||||
"probabilities": {"yes": 0.02, "no": 0.98},
|
||||
"confidence": 0.95
|
||||
},
|
||||
"tone": {
|
||||
"type": "score",
|
||||
"score": 0.6,
|
||||
"legend": {"0": "calm", "1": "annoyed", "2": "furious"},
|
||||
"probabilities": {"0": 0.7, "1": 0.2, "2": 0.1},
|
||||
"confidence": 0.65
|
||||
}
|
||||
},
|
||||
"usage": {"input_tokens": 142, "output_tokens": 3},
|
||||
"diagnostics": {
|
||||
"team": {"label_mass": 0.93, "argmax_is_label": true},
|
||||
"urgent": {"label_mass": 0.98, "argmax_is_label": true},
|
||||
"tone": {"label_mass": 0.91, "argmax_is_label": true}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`diagnostics.label_mass` is the probability the model put on the labels across
|
||||
its whole vocabulary. A low value means most of the probability went to tokens
|
||||
that are not labels. `confidence` is the chosen option's probability over the
|
||||
whole vocabulary: its share of the labels times `label_mass`.
|
||||
|
||||
## Labels
|
||||
|
||||
Labels follow the type: a `choice` labels its options `A` to `Z` in the order
|
||||
the request lists them, a `noul` labels its options `yes` and `no`, and a
|
||||
`score` labels its levels `0` to `9`, so a level's label is its score.
|
||||
|
||||
## Limits
|
||||
|
||||
| limit | value |
|
||||
| --- | --- |
|
||||
| questions per request | 64 |
|
||||
| options per `choice` | 26 |
|
||||
| levels per `score` | 10 |
|
||||
|
||||
## Clef-Flash
|
||||
|
||||
[Clef-Flash](https://huggingface.co/Cloudflare/clef-flash) uses a Qwen3.5-9B
|
||||
backbone and a joint schema head to score all questions in one forward pass.
|
||||
vLLM detects the head and serves the checkpoint as a pooling model.
|
||||
|
||||
```bash
|
||||
vllm serve Cloudflare/clef-flash --max-model-len 16384
|
||||
```
|
||||
|
||||
Send requests to `/v1/systemone` using `state` and `questions`, as in the
|
||||
examples above. `choice`, `noul` and `score` questions are supported.
|
||||
|
||||
- Text and JSON states use the checkpoint's fixed prompt. JSON is compact
|
||||
with sorted keys. `instructions` returns 400; `chat_template_kwargs`
|
||||
has no effect.
|
||||
- The state is truncated to fit 16,384 tokens, or `max_model_len` if smaller.
|
||||
A schema that exceeds this bound returns 400.
|
||||
- `images` and `videos` are placed before the state. Videos are sampled at
|
||||
2 fps and processed by the Qwen3.5 vision processor. Audio inputs return
|
||||
400, and video soundtracks are ignored.
|
||||
- Each request performs one pooling read. `usage.input_tokens` counts the
|
||||
prompt once, `output_tokens` is 0, and `label_mass` is 1.
|
||||
|
||||
## Clef-Omni
|
||||
|
||||
[Clef-Omni](https://huggingface.co/Cloudflare/clef-omni) is a decision model:
|
||||
a joint schema head on the Qwen3-Omni thinker scores every option of every
|
||||
question in one forward pass. vLLM serves it as a pooling model.
|
||||
|
||||
```bash
|
||||
vllm serve Cloudflare/clef-omni
|
||||
```
|
||||
|
||||
- A request is one read. The prompt is the one the checkpoint's
|
||||
`joint_schema_model.py` builds: the state, then every question with its
|
||||
options. A JSON state is written as compact JSON with sorted keys.
|
||||
- A state too long for the model is truncated to fit, as the reference does.
|
||||
A request whose questions alone exceed 64,000 tokens, or `max_model_len`,
|
||||
returns 400.
|
||||
- `instructions` returns 400, since the prompt is fixed. `chat_template_kwargs`
|
||||
has no effect.
|
||||
- There are no labels, so `label_mass` is 1 and `confidence` is the chosen
|
||||
option's probability. `usage.input_tokens` counts the prompt once, and
|
||||
`output_tokens` is 0.
|
||||
- `images`, `audio` and `videos` are read as the reference does: placed
|
||||
before the state, videos sampled at 2 fps, and a video's soundtrack heard
|
||||
with it when every video has one. Audio clips cannot be sent together with
|
||||
videos that have soundtracks.
|
||||
|
||||
## Liquid d1
|
||||
|
||||
[`LiquidAI/d1-3B`](https://huggingface.co/LiquidAI/d1-3B) is LFM2.5-VL-3B
|
||||
post-trained to answer decisions as the first token of its reply. vLLM serves
|
||||
it as a generative model.
|
||||
|
||||
```bash
|
||||
vllm serve LiquidAI/d1-3B
|
||||
```
|
||||
|
||||
- The server writes the prompt and reads the label tokens the way the
|
||||
`prompt.py` shipped with the checkpoint does, so picks and probabilities
|
||||
match `model.system_one()`. `confidence` and `usage` follow the definitions
|
||||
on this page instead.
|
||||
- A JSON state is written as indented JSON. A `null` state leaves the images,
|
||||
or the question alone, as the prompt.
|
||||
- `images` are placed before the state and shrunk to at most 1024 x 1024
|
||||
pixels, as the reference does.
|
||||
- A `choice` whose option names are all single letters is labeled with those
|
||||
letters.
|
||||
- When each request carries one question over a state that changes from
|
||||
request to request, `--no-enable-prefix-caching` lowers latency.
|
||||
|
||||
## Request fields
|
||||
|
||||
| field | meaning |
|
||||
| --- | --- |
|
||||
| `state` | what the questions are about: a string, or JSON that is sent as its JSON text |
|
||||
| `questions` | question id to `{type, instructions, criteria}`, asked in this order |
|
||||
| `instructions` | optional context placed ahead of the questions |
|
||||
| `chat_template_kwargs` | passed to the chat template, for example `{"enable_thinking": false}` |
|
||||
| `cache_salt` | optional [prefix-cache salt](../../usage/security.md#cache-salting) applied to every question |
|
||||
| `images`, `audio`, `videos` | optional media URLs or data URLs, for models that take them ([Clef-Flash](#clef-flash), images/video; [Clef-Omni](#clef-omni); [Liquid d1](#liquid-d1), images only) |
|
||||
|
||||
A question with any field other than `type`, `instructions` and `criteria` is
|
||||
rejected with a 400.
|
||||
|
||||
When [tracing](../../../examples/observability/opentelemetry/README.md) is enabled
|
||||
with `--otlp-traces-endpoint`, the HTTP `traceparent` and `tracestate` headers are
|
||||
forwarded to every question's generation request.
|
||||
Reference in New Issue
Block a user