[Feat] Structured decisions endpoint (/v1/systemone) (#59299)
Signed-off-by: Matt Mastracci <matthew@mastracci.com> Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com> Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
Lucas Wilkinson
Lucas Wilkinson
Francisco Javier Arceo
Chauncey
parent
273e099693
commit
d33872dfeb
@@ -76,6 +76,9 @@ For further details on speech to text, please refer to [this page](speech_to_tex
|
||||
- [Generative Scoring API](generative_scoring.md) (`/generative_scoring`)
|
||||
- Applicable to [CausalLM models](../../models/generative_models.md) (task `"generate"`).
|
||||
- Computes next-token probabilities for specified `label_token_ids`.
|
||||
- [Structured Decisions API](structured_decisions.md) (`/v1/systemone`)
|
||||
- Applicable to [text generation models](../../models/generative_models.md) with a [chat template](#chat-template).
|
||||
- Answers typed questions about a state with a probability for every allowed answer.
|
||||
|
||||
## Instrumentator APIs
|
||||
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
# Structured Decisions
|
||||
|
||||
The `/v1/systemone` endpoint answers a set of typed questions about a state and
|
||||
returns a probability for every allowed answer.
|
||||
|
||||
The endpoint is off unless the server starts with
|
||||
`--enable-structured-decisions`. With the flag, startup fails for a model that
|
||||
does not support structured decisions, and with `--logprobs-mode raw_logits` or
|
||||
`processed_logits`, since `label_mass` needs log probabilities.
|
||||
|
||||
## How it works
|
||||
|
||||
1. Each question is one read. The user message is the state, followed by the
|
||||
question and its options, labeled `A`, `B`, `C`, ... in order, and
|
||||
"Answer with the letter of one option only."
|
||||
2. The read is the reply's first token. Each label must be one token there,
|
||||
or the request returns 400.
|
||||
3. The read returns the logprobs of that question's label tokens. Softmax over
|
||||
the labels gives the answer's probabilities.
|
||||
|
||||
Thinking is off unless `chat_template_kwargs` turns it on.
|
||||
|
||||
This endpoint does not serve diffusion models yet.
|
||||
|
||||
Every read of a request starts with the state, so with
|
||||
`--enable-prefix-caching` the state is prefilled once and each further
|
||||
question prefills only its own text, then reads one token.
|
||||
|
||||
## Question types
|
||||
|
||||
| type | criteria | answer |
|
||||
| --- | --- | --- |
|
||||
| `choice` | map of option name to a description or `null` | `choice`, `probabilities` by option name, `confidence` |
|
||||
|
||||
A choice has at least one option. A question id is a non-empty string.
|
||||
|
||||
## Example
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:8000/v1/systemone \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"state": {"ticket": "My card was charged twice for one order."},
|
||||
"questions": {
|
||||
"team": {
|
||||
"type": "choice",
|
||||
"instructions": "Which team should handle this ticket?",
|
||||
"criteria": {
|
||||
"billing": "payments and refunds",
|
||||
"shipping": "deliveries",
|
||||
"security": "account access"
|
||||
}
|
||||
}
|
||||
},
|
||||
"chat_template_kwargs": {"enable_thinking": false}
|
||||
}'
|
||||
```
|
||||
|
||||
Response, with probabilities that depend on the model:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "decision-...",
|
||||
"object": "structured_decision",
|
||||
"created": 1790000000,
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"answers": {
|
||||
"team": {
|
||||
"type": "choice",
|
||||
"choice": "billing",
|
||||
"probabilities": {"billing": 0.97, "shipping": 0.01, "security": 0.02},
|
||||
"confidence": 0.90
|
||||
}
|
||||
},
|
||||
"usage": {"input_tokens": 142, "output_tokens": 1},
|
||||
"diagnostics": {
|
||||
"team": {"label_mass": 0.93, "argmax_is_label": true}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`diagnostics.label_mass` is the probability the model put on the labels across
|
||||
its whole vocabulary. A low value means most of the probability went to tokens
|
||||
that are not labels. `confidence` is the chosen option's probability over the
|
||||
whole vocabulary: its share of the labels times `label_mass`.
|
||||
|
||||
## Labels
|
||||
|
||||
Options are labeled `A` to `Z` in the order the request lists them, so a
|
||||
`choice` has at most 26 options.
|
||||
|
||||
## Limits
|
||||
|
||||
| limit | value |
|
||||
| --- | --- |
|
||||
| questions per request | 64 |
|
||||
| options per `choice` | 26 |
|
||||
|
||||
## Request fields
|
||||
|
||||
| field | meaning |
|
||||
| --- | --- |
|
||||
| `state` | what the questions are about: a string, or JSON that is sent as its JSON text |
|
||||
| `questions` | question id to `{type, instructions, criteria}`, asked in this order |
|
||||
| `instructions` | optional context placed ahead of the questions |
|
||||
| `chat_template_kwargs` | passed to the chat template, for example `{"enable_thinking": false}` |
|
||||
|
||||
A question with any field other than `type`, `instructions` and `criteria` is
|
||||
rejected with a 400.
|
||||
@@ -175,6 +175,7 @@ When `--api-key` is configured, the following endpoints require Bearer token aut
|
||||
- `/v1/responses/{response_id}/cancel` - Cancel a response
|
||||
- `/v1/score` - Scoring API
|
||||
- `/v1/rerank` - Reranking API
|
||||
- `/v1/systemone` - Structured decisions API
|
||||
- `/v1/load_lora_adapter` - Load a LoRA adapter (can alter model behavior; only available when `--enable-lora` is set and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`)
|
||||
- `/v1/unload_lora_adapter` - Unload a LoRA adapter (can alter model behavior; only available when `--enable-lora` is set and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`)
|
||||
- `/inference/v1/generate` - Generate completions (available when `--enable-scale-out` is set, or with `--tokens-only`)
|
||||
|
||||
Reference in New Issue
Block a user