[Frontend] Add cache salt and tracing to structured decisions (#60786)
Signed-off-by: Taewoong Kim <ktw2172@gmail.com> Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
This commit is contained in:
@@ -135,6 +135,11 @@ the request lists them, a `noul` labels its options `yes` and `no`, and a
|
||||
| `questions` | question id to `{type, instructions, criteria}`, asked in this order |
|
||||
| `instructions` | optional context placed ahead of the questions |
|
||||
| `chat_template_kwargs` | passed to the chat template, for example `{"enable_thinking": false}` |
|
||||
| `cache_salt` | optional [prefix-cache salt](../../usage/security.md#cache-salting) applied to every question |
|
||||
|
||||
A question with any field other than `type`, `instructions` and `criteria` is
|
||||
rejected with a 400.
|
||||
|
||||
When [tracing](../../../examples/observability/opentelemetry/README.md) is enabled
|
||||
with `--otlp-traces-endpoint`, the HTTP `traceparent` and `tracestate` headers are
|
||||
forwarded to every question's generation request.
|
||||
|
||||
@@ -572,7 +572,7 @@ An attacker sharing the same backend can measure differences in Time to First To
|
||||
|
||||
vLLM accepts an optional `cache_salt` parameter on requests. The salt is mixed into the hash of the first KV cache block, so only requests carrying the same salt can share cached prefix blocks. See [Automatic Prefix Caching](../design/prefix_caching.md) for the implementation details.
|
||||
|
||||
`cache_salt` is accepted by the OpenAI-compatible chat completions, completions, responses, and pooling (embeddings, classification, scoring) endpoints, and by the Anthropic `/v1/messages` endpoint.
|
||||
`cache_salt` is accepted by the OpenAI-compatible chat completions, completions, responses, and pooling (embeddings, classification, scoring) endpoints, the Anthropic `/v1/messages` endpoint, and the structured decisions `/v1/systemone` endpoint.
|
||||
|
||||
#### Usage with the OpenAI Python client
|
||||
|
||||
|
||||
Reference in New Issue
Block a user