363 lines
16 KiB
Markdown
363 lines
16 KiB
Markdown
# Structured Decisions Models
|
||
|
||
Structured Decisions Models (aka JEV Models) map inputs directly to predefined
|
||
options or categories and return probabilities and confidence scores, commonly
|
||
used for classification and routing.
|
||
|
||
## Supported Models
|
||
|
||
### Text-only Models
|
||
|
||
| Architecture | ReadStrategy | Task | Head | Example HF Models |
|
||
|---------------------------|---------------------|------------------|-------------------|-------------------------|
|
||
| `ClefFlashForJointSchema` | `ClefFlashStrategy` | `token_classify` | `JointSchemaHead` | `Cloudflare/clef-flash` |
|
||
| `Qwen3ForCausalLM` | `NextTokenStrategy` | `generate` | N/A | `Qwen/Qwen3-0.6B`, etc. |
|
||
|
||
### Multimodal Models
|
||
|
||
!!! note
|
||
For more information about multimodal models inputs, see [this page](./supported_models.md#list-of-multimodal-language-models).
|
||
|
||
| Architecture | Inputs | ReadStrategy | Task | Head | Example HF Models |
|
||
|--------------------------------------|-------------------------------------|--------------------------------|------------------|-------------------|------------------------------------------|
|
||
| `ClefOmniForJointSchema` | T + I + V | `JointSchemaStrategy` | `token_classify` | `JointSchemaHead` | `Cloudflare/clef-omni` |
|
||
| `D3ForSequenceClassification` | T + I + V | `D3Strategy` | `classify` | Linear (`score`) | `vllm-sr/d3`, `vllm-sr/d3-mini`, etc. |
|
||
| `Lfm2VlForConditionalGeneration` | T + I<sup>+</sup> | `LiquidStrategy` | `generate` | N/A | `LiquidAI/d1-3B` |
|
||
| `Qwen3_5ForConditionalGeneration` | T + I<sup>E+</sup> + V<sup>E+</sup> | `NextTokenStrategy` | `generate` | N/A | `Qwen/Qwen3.5-0.8B`, etc. |
|
||
| `Qwen3_5MoeForConditionalGeneration` | T + I<sup>E+</sup> + V<sup>E+</sup> | `NextTokenStrategy` | `generate` | N/A | `Qwen/Qwen3.5-35B-A3B`, etc. |
|
||
| `DiffusionGemmaForBlockDiffusion` | T + I<sup>+</sup> + V<sup>+</sup> | `DiffusionGemmaCanvasStrategy` | `generate` | N/A | `google/diffusiongemma-26B-A4B-it`, etc. |
|
||
|
||
## Online Serving
|
||
|
||
### TypeSafe System One Endpoint
|
||
|
||
The `/v1/systemone` endpoint answers a set of typed questions about a state and
|
||
returns a probability for every allowed answer.
|
||
|
||
The endpoint is available by default on generation models. Models without a
|
||
supported read strategy return 501. Supported architectures require
|
||
`--logprobs-mode raw_logprobs` or `processed_logprobs`, since `label_mass` needs
|
||
log probabilities.
|
||
|
||
#### How it works
|
||
|
||
1. Each question is one read. The user message is the state, followed by the
|
||
question and its options with the type's labels, and an instruction to
|
||
answer with one label.
|
||
2. The read is the reply's first token. Each label must be one token there,
|
||
or the request returns 400.
|
||
3. The read returns the logprobs of that question's label tokens. Softmax over
|
||
the labels gives the answer's probabilities.
|
||
|
||
Thinking is off unless `chat_template_kwargs` turns it on.
|
||
|
||
Probabilities can differ between identical requests. They depend on which
|
||
requests share a batch and on whether a read hits the prefix cache. Set
|
||
`VLLM_BATCH_INVARIANT=1` for reads that repeat exactly, at a cost in
|
||
throughput.
|
||
|
||
Every read of a request starts with the state, so with
|
||
`--enable-prefix-caching` the state is prefilled once and each further
|
||
question prefills only its own text, then reads one token.
|
||
|
||
#### Question types
|
||
|
||
| type | criteria | answer |
|
||
| --- | --- | --- |
|
||
| `choice` | map of option name to a description or `null` | `choice`, `probabilities` by option name, `confidence` |
|
||
| `noul` | `null`, or an object with optional `true` and `false` descriptions | `noul` (probability of yes), `probabilities`, `confidence` |
|
||
| `score` | ordered list of level names | `score` (expected 0-indexed level), `legend`, `probabilities` by level index, `confidence` |
|
||
|
||
A choice has at least one option, a score at least two levels. A question id
|
||
is a non-empty string.
|
||
|
||
#### Example
|
||
|
||
```bash
|
||
curl -s http://localhost:8000/v1/systemone \
|
||
-H "Content-Type: application/json" \
|
||
-d '{
|
||
"model": "Qwen/Qwen3-0.6B",
|
||
"state": {"ticket": "My card was charged twice for one order."},
|
||
"questions": {
|
||
"team": {
|
||
"type": "choice",
|
||
"instructions": "Which team should handle this ticket?",
|
||
"criteria": {
|
||
"billing": "payments and refunds",
|
||
"shipping": "deliveries",
|
||
"security": "account access"
|
||
}
|
||
},
|
||
"urgent": {
|
||
"type": "noul",
|
||
"instructions": "Does the customer need a reply within the hour?",
|
||
"criteria": {"true": "the account is locked or money is moving"}
|
||
},
|
||
"tone": {
|
||
"type": "score",
|
||
"instructions": "How upset is the customer?",
|
||
"criteria": ["calm", "annoyed", "furious"]
|
||
}
|
||
},
|
||
"chat_template_kwargs": {"enable_thinking": false}
|
||
}'
|
||
```
|
||
|
||
Response, with probabilities that depend on the model:
|
||
|
||
```json
|
||
{
|
||
"id": "decision-...",
|
||
"object": "structured_decision",
|
||
"created": 1790000000,
|
||
"model": "Qwen/Qwen3-0.6B",
|
||
"answers": {
|
||
"team": {
|
||
"type": "choice",
|
||
"choice": "billing",
|
||
"probabilities": {"billing": 0.97, "shipping": 0.01, "security": 0.02},
|
||
"confidence": 0.90
|
||
},
|
||
"urgent": {
|
||
"type": "noul",
|
||
"noul": 0.02,
|
||
"probabilities": {"yes": 0.02, "no": 0.98},
|
||
"confidence": 0.95
|
||
},
|
||
"tone": {
|
||
"type": "score",
|
||
"score": 0.6,
|
||
"legend": {"0": "calm", "1": "annoyed", "2": "furious"},
|
||
"probabilities": {"0": 0.7, "1": 0.2, "2": 0.1},
|
||
"confidence": 0.65
|
||
}
|
||
},
|
||
"usage": {"input_tokens": 142, "output_tokens": 3},
|
||
"diagnostics": {
|
||
"team": {"label_mass": 0.93, "argmax_is_label": true},
|
||
"urgent": {"label_mass": 0.98, "argmax_is_label": true},
|
||
"tone": {"label_mass": 0.91, "argmax_is_label": true}
|
||
}
|
||
}
|
||
```
|
||
|
||
`diagnostics.label_mass` is the probability the model put on the labels across
|
||
its whole vocabulary. A low value means most of the probability went to tokens
|
||
that are not labels. `confidence` is the chosen option's probability over the
|
||
whole vocabulary: its share of the labels times `label_mass`.
|
||
|
||
#### Labels
|
||
|
||
Labels follow the type: a `choice` labels its options `A` to `Z` in the order
|
||
the request lists them, a `noul` labels its options `yes` and `no`, and a
|
||
`score` labels its levels `0` to `9`, so a level's label is its score.
|
||
|
||
#### Limits
|
||
|
||
| limit | value |
|
||
| --- | --- |
|
||
| questions per request | 64 |
|
||
| options per `choice` | 26 |
|
||
| levels per `score` | 10 |
|
||
|
||
#### Request fields
|
||
|
||
| field | meaning |
|
||
| --- | --- |
|
||
| `state` | what the questions are about: a string, or JSON that is sent as its JSON text |
|
||
| `questions` | question id to `{type, instructions, criteria}`, asked in this order |
|
||
| `instructions` | optional context placed ahead of the questions |
|
||
| `chat_template_kwargs` | passed to the chat template, for example `{"enable_thinking": false}` |
|
||
| `cache_salt` | optional [prefix-cache salt](../usage/security.md#cache-salting) applied to every question |
|
||
| `images`, `audio`, `videos` | optional media URLs or data URLs, for models that take them ([Clef-Flash](#clef-flash), images/video; [Clef-Omni](#clef-omni); [Decision 3.0](#decision-30), images/video; [Liquid d1](#liquid-d1), images only) |
|
||
|
||
A question with any field other than `type`, `instructions` and `criteria` is
|
||
rejected with a 400.
|
||
|
||
When [tracing](../../examples/observability/opentelemetry/README.md) is enabled
|
||
with `--otlp-traces-endpoint`, the HTTP `traceparent` and `tracestate` headers are
|
||
forwarded to every question's generation request.
|
||
|
||
### OpenAI Decisions API
|
||
|
||
vLLM provides a text-only implementation of OpenAI's
|
||
[`POST /v1/decisions`](https://developers.openai.com/api/docs/guides/decisions)
|
||
API. Start a supported model:
|
||
|
||
```bash
|
||
vllm serve Qwen/Qwen3-0.6B
|
||
```
|
||
|
||
```python
|
||
from openai import OpenAI
|
||
|
||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||
decision = client.decisions.create(
|
||
model="Qwen/Qwen3-0.6B",
|
||
input="The package arrived with a broken screen.",
|
||
questions=[
|
||
{
|
||
"type": "predicate",
|
||
"name": "damaged",
|
||
"instructions": "Does the customer report a damaged item?",
|
||
},
|
||
{
|
||
"type": "choice",
|
||
"name": "department",
|
||
"instructions": "Which department should handle this complaint?",
|
||
"choices": [{"value": "returns"}, {"value": "billing"}],
|
||
},
|
||
{
|
||
"type": "score",
|
||
"instructions": "How severe is the damage?",
|
||
"levels": [{"label": "cosmetic"}, {"label": "broken"}],
|
||
},
|
||
],
|
||
)
|
||
print(decision.answers)
|
||
```
|
||
|
||
Use OpenAI Python SDK 3.26.0 or newer for `client.decisions.create`.
|
||
`input` accepts a string or an array of user messages with string content or
|
||
`input_text` parts. Answers retain question order and echo `name`, including
|
||
`null` for unnamed questions. Choice values preserve strings and booleans.
|
||
Scores are probability-weighted averages of zero-based level indices.
|
||
|
||
The request contract follows the OpenAI OpenAPI schemas retrieved on
|
||
October 7, 2026. This MVP supports 1–64 questions, 2–26 choices, and 2–10 score
|
||
levels. Upstream permits 200 questions and 255 choices; this backend returns
|
||
a 400 when either model limit is exceeded.
|
||
The optional `safety_identifier` is caller metadata, available in request
|
||
body logging with `--enable-log-requests` and DEBUG logging; it does not
|
||
authenticate callers.
|
||
|
||
#### Implementation and limits
|
||
|
||
This MVP uses the same label-reading backend as
|
||
[structured decisions](structured_decisions_models.md). It supports generation
|
||
architectures with a registered read strategy, including Qwen, DiffusionGemma,
|
||
and Liquid d1; unsupported models return 501. Each question reads a model's
|
||
label tokens. Usage reports the actual prompt and output tokens across these
|
||
reads, including prefix-cache counters.
|
||
|
||
Probabilities are normalized over the supplied options. `confidence` is the
|
||
winning label's probability over the full vocabulary; it is not calibrated
|
||
to OpenAI's models. Thinking is disabled for these reads.
|
||
|
||
Image inputs, other message roles, tools, streaming, and per-question refusal
|
||
scoring are outside this MVP. Unsupported request fields and input types are
|
||
rejected. The `/v1/systemone` endpoint retains its existing request format and
|
||
is available by default.
|
||
|
||
## Examples
|
||
|
||
### Clef-Flash
|
||
|
||
[Clef-Flash](https://huggingface.co/Cloudflare/clef-flash) uses a Qwen3.5-9B
|
||
backbone and a joint schema head to score all questions in one forward pass.
|
||
vLLM detects the head and serves the checkpoint as a pooling model.
|
||
|
||
```bash
|
||
vllm serve Cloudflare/clef-flash --max-model-len 16384
|
||
```
|
||
|
||
Send requests to `/v1/systemone` using `state` and `questions`, as in the
|
||
examples above. `choice`, `noul` and `score` questions are supported.
|
||
|
||
- Text and JSON states use the checkpoint's fixed prompt. JSON is compact
|
||
with sorted keys. `instructions` returns 400; `chat_template_kwargs`
|
||
has no effect.
|
||
- The state is truncated to fit 16,384 tokens, or `max_model_len` if smaller.
|
||
A schema that exceeds this bound returns 400.
|
||
- `images` and `videos` are placed before the state. Videos are sampled at
|
||
2 fps and processed by the Qwen3.5 vision processor. Audio inputs return
|
||
400, and video soundtracks are ignored.
|
||
- Each request performs one pooling read. `usage.input_tokens` counts the
|
||
prompt once, `output_tokens` is 0, and `label_mass` is 1.
|
||
|
||
### Clef-Omni
|
||
|
||
[Clef-Omni](https://huggingface.co/Cloudflare/clef-omni) is a decision model:
|
||
a joint schema head on the Qwen3-Omni thinker scores every option of every
|
||
question in one forward pass. vLLM serves it as a pooling model.
|
||
|
||
```bash
|
||
vllm serve Cloudflare/clef-omni
|
||
```
|
||
|
||
- A request is one read. The prompt is the one the checkpoint's
|
||
`joint_schema_model.py` builds: the state, then every question with its
|
||
options. A JSON state is written as compact JSON with sorted keys.
|
||
- A state too long for the model is truncated to fit, as the reference does.
|
||
A request whose questions alone exceed 64,000 tokens, or `max_model_len`,
|
||
returns 400.
|
||
- `instructions` returns 400, since the prompt is fixed. `chat_template_kwargs`
|
||
has no effect.
|
||
- There are no labels, so `label_mass` is 1 and `confidence` is the chosen
|
||
option's probability. `usage.input_tokens` counts the prompt once, and
|
||
`output_tokens` is 0.
|
||
- `images`, `audio` and `videos` are read as the reference does: placed
|
||
before the state, videos sampled at 2 fps, and a video's soundtrack heard
|
||
with it when every video has one. Audio clips cannot be sent together with
|
||
videos that have soundtracks.
|
||
|
||
### Decision 3.0
|
||
|
||
The Decision 3.0 models are a Qwen3.5 backbone with an answer-code readout:
|
||
one forward pass per question, and the readout scores the question's options
|
||
at the last token. vLLM detects the checkpoint's `decision_config.json` and
|
||
serves it as a classification model. All six sizes take the same command:
|
||
[`vllm-sr/d3`](https://huggingface.co/vllm-sr/d3) (27B),
|
||
[`vllm-sr/d3-flash`](https://huggingface.co/vllm-sr/d3-flash) (9B),
|
||
[`vllm-sr/d3-mini`](https://huggingface.co/vllm-sr/d3-mini) (4B),
|
||
[`vllm-sr/d3-nano`](https://huggingface.co/vllm-sr/d3-nano) (2B),
|
||
[`vllm-sr/d3-lite`](https://huggingface.co/vllm-sr/d3-lite) (0.8B) and
|
||
[`vllm-sr/d3-edge`](https://huggingface.co/vllm-sr/d3-edge) (0.6B).
|
||
|
||
```bash
|
||
vllm serve vllm-sr/d3
|
||
```
|
||
|
||
- Each question is one read of the prompt the checkpoint's `d3_format.py`
|
||
writes: the state, the question and one line per option under an answer
|
||
code. A question's probabilities are the softmax of its options' code
|
||
logits at the checkpoint's temperature, so picks and probabilities match
|
||
`model.system_one()`. `confidence` and `usage` follow the definitions on
|
||
this page instead.
|
||
- A choice can have up to 255 options. A JSON state is written as JSON text,
|
||
and an empty state as `(empty)`.
|
||
- The full attention layers read the whole prompt in both directions, as the
|
||
checkpoint declares, so chunked prefill and prefix caching are off.
|
||
- `instructions` returns 400, since the prompt is fixed.
|
||
`chat_template_kwargs` has no effect.
|
||
- `images` and `videos` are placed before the text, images read at up to
|
||
1.6 megapixels and videos at 2 fps, at most 32 frames of up to
|
||
0.2 megapixels each, as the checkpoint's runtime does. The media are read
|
||
once and shared by every question of the request.
|
||
- `/classify` and `/pooling` return the raw logits of all 255 codes; only the
|
||
first codes, one per option, belong to a question.
|
||
|
||
### Liquid d1
|
||
|
||
[`LiquidAI/d1-3B`](https://huggingface.co/LiquidAI/d1-3B) is LFM2.5-VL-3B
|
||
post-trained to answer decisions as the first token of its reply. vLLM serves
|
||
it as a generative model.
|
||
|
||
```bash
|
||
vllm serve LiquidAI/d1-3B
|
||
```
|
||
|
||
- The server writes the prompt and reads the label tokens the way the
|
||
`prompt.py` shipped with the checkpoint does, so picks and probabilities
|
||
match `model.system_one()`. `confidence` and `usage` follow the definitions
|
||
on this page instead.
|
||
- A JSON state is written as indented JSON. A `null` state leaves the images,
|
||
or the question alone, as the prompt.
|
||
- `images` are placed before the state and shrunk to at most 1024 x 1024
|
||
pixels, as the reference does.
|
||
- A `choice` whose option names are all single letters is labeled with those
|
||
letters.
|
||
- When each request carries one question over a state that changes from
|
||
request to request, `--disable-prefix-caching` lowers latency.
|