[Frontend] Skip structural tags for auto tool_choice without strict mode (#45600)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
This commit is contained in:
@@ -109,18 +109,18 @@ vLLM supports the `tool_choice='none'` option in the chat completion API. When t
|
|||||||
|
|
||||||
## Constrained Decoding Behavior
|
## Constrained Decoding Behavior
|
||||||
|
|
||||||
Whether vLLM enforces the tool parameter schema during generation depends on the `tool_choice` mode:
|
Whether vLLM enforces the tool parameter schema during generation depends on the `tool_choice` mode and the per-tool `strict` field:
|
||||||
|
|
||||||
| `tool_choice` value | Schema-constrained decoding | Behavior |
|
| `tool_choice` value | Schema-constrained decoding | Behavior |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| Named function | Yes (via structured outputs backend) | Arguments are guaranteed to be valid JSON conforming to the function's parameter schema. |
|
| Named function | Yes (via structured outputs backend) | Arguments are guaranteed to be valid JSON conforming to the function's parameter schema. |
|
||||||
| `"required"` | Yes (via structured outputs backend) | Same as named function. The model must produce at least one tool call. |
|
| `"required"` | Yes (via structured outputs backend) | Same as named function. The model must produce at least one tool call. |
|
||||||
| `"auto"` | Depends on the parser | Model-specific structural-tag parsers can constrain tool-call arguments with structured outputs. Other parsers generate freely and extract tool calls from raw text. |
|
| `"auto"` | Only when `strict: true` is set on at least one tool | Structural-tag parsers constrain tool-call arguments when a tool opts in with `strict: true`. Without it, the model generates freely and tool calls are extracted from raw text. |
|
||||||
| `"none"` | N/A | No tool calls are produced. |
|
| `"none"` | N/A | No tool calls are produced. |
|
||||||
|
|
||||||
### Strict Mode
|
### Strict Mode
|
||||||
|
|
||||||
Strict tool calling makes function-call arguments adhere to the function schema instead of relying only on best-effort parsing. vLLM implements strict tool calling for structural-tag based tool parsers by using the structured outputs backend under the hood.
|
For `tool_choice="required"` or named function calling, structural-tag constraints are always applied regardless of the `strict` field. For `tool_choice="auto"`, setting `strict: true` on at least one tool opts in to structural-tag constraints; without it, the model generates freely and tool calls are extracted from raw text. The `strict` field is supported across all three API surfaces: Chat Completion, Responses, and Anthropic Messages.
|
||||||
|
|
||||||
For best compatibility with strict schema enforcement, define tool parameter schemas in the OpenAI strict-schema style:
|
For best compatibility with strict schema enforcement, define tool parameter schemas in the OpenAI strict-schema style:
|
||||||
|
|
||||||
@@ -128,16 +128,12 @@ For best compatibility with strict schema enforcement, define tool parameter sch
|
|||||||
* Mark all fields in `properties` as required.
|
* Mark all fields in `properties` as required.
|
||||||
* Represent optional fields by allowing `null`, for example `{"type": ["string", "null"]}`.
|
* Represent optional fields by allowing `null`, for example `{"type": ["string", "null"]}`.
|
||||||
|
|
||||||
vLLM controls structural-tag strict tool calling with the `VLLM_ENFORCE_STRICT_TOOL_CALLING` environment variable. It defaults to `true`.
|
vLLM also provides a global toggle via the `VLLM_ENFORCE_STRICT_TOOL_CALLING` environment variable (defaults to `true`). When set to `false`, vLLM does not attach structural tags for tool calling regardless of the per-tool `strict` field. This environment variable only affects structural-tag based tool calling; it does not change schema-derived structured outputs used by named function calling or `tool_choice="required"`.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
VLLM_ENFORCE_STRICT_TOOL_CALLING=false vllm serve ...
|
VLLM_ENFORCE_STRICT_TOOL_CALLING=false vllm serve ...
|
||||||
```
|
```
|
||||||
|
|
||||||
When this variable is `true`, structural-tag based tool parsers attach a structural tag to the request, so the structured outputs backend can constrain the model-specific tool-call format and function-call arguments. When it is `false`, vLLM does not attach structural tags for tool calling. In that case, `tool_choice="auto"` falls back to best-effort parser extraction from the raw model output, and no structural-tag constraint is applied.
|
|
||||||
|
|
||||||
This environment variable only affects structural-tag based tool calling. It does not change schema-derived structured outputs used by named function calling or `tool_choice="required"`.
|
|
||||||
|
|
||||||
## Automatic Function Calling
|
## Automatic Function Calling
|
||||||
|
|
||||||
To enable this feature, you should set the following flags:
|
To enable this feature, you should set the following flags:
|
||||||
@@ -156,7 +152,7 @@ from HuggingFace; and you can find an example of this in a `tokenizer_config.jso
|
|||||||
If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
|
If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
|
||||||
|
|
||||||
!!! note
|
!!! note
|
||||||
With `tool_choice="auto"`, schema-level constraint depends on the selected parser and `VLLM_ENFORCE_STRICT_TOOL_CALLING`. Structural-tag parsers can enforce tool-call constraints when it is `true`; when it is `false`, or when the selected parser has no structural-tag support, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema.
|
With `tool_choice="auto"`, schema-level constraint requires both `VLLM_ENFORCE_STRICT_TOOL_CALLING=true` (the default) and at least one tool with `strict: true`. When these conditions are met and the selected parser supports structural tags, vLLM constrains tool-call arguments. Otherwise, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema.
|
||||||
|
|
||||||
### Hermes Models (`hermes`)
|
### Hermes Models (`hermes`)
|
||||||
|
|
||||||
|
|||||||
@@ -216,14 +216,32 @@ def test_streaming_emits_incremental_argument_chunks():
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _with_strict(
|
||||||
|
tools: list[ChatCompletionToolsParam],
|
||||||
|
) -> list[ChatCompletionToolsParam]:
|
||||||
|
return [
|
||||||
|
ChatCompletionToolsParam(
|
||||||
|
type=t.type,
|
||||||
|
function=FunctionDefinition(
|
||||||
|
name=t.function.name,
|
||||||
|
description=t.function.description,
|
||||||
|
parameters=t.function.parameters,
|
||||||
|
strict=True,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for t in tools
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def test_get_vllm_registry_structural_tag_returns_structural_tag(
|
def test_get_vllm_registry_structural_tag_returns_structural_tag(
|
||||||
sample_tools: list[ChatCompletionToolsParam],
|
sample_tools: list[ChatCompletionToolsParam],
|
||||||
) -> None:
|
) -> None:
|
||||||
parser = make_parser()
|
parser = make_parser()
|
||||||
|
strict_tools = _with_strict(sample_tools)
|
||||||
req = ChatCompletionRequest(
|
req = ChatCompletionRequest(
|
||||||
messages=[],
|
messages=[],
|
||||||
model="m",
|
model="m",
|
||||||
tools=sample_tools,
|
tools=strict_tools,
|
||||||
tool_choice="auto",
|
tool_choice="auto",
|
||||||
)
|
)
|
||||||
tag = parser.get_structural_tag(req)
|
tag = parser.get_structural_tag(req)
|
||||||
|
|||||||
@@ -14,6 +14,7 @@ from vllm.entrypoints.openai.chat_completion.protocol import (
|
|||||||
ChatCompletionNamedToolChoiceParam,
|
ChatCompletionNamedToolChoiceParam,
|
||||||
ChatCompletionRequest,
|
ChatCompletionRequest,
|
||||||
ChatCompletionToolsParam,
|
ChatCompletionToolsParam,
|
||||||
|
FunctionDefinition,
|
||||||
)
|
)
|
||||||
from vllm.entrypoints.openai.engine.protocol import (
|
from vllm.entrypoints.openai.engine.protocol import (
|
||||||
DeltaMessage,
|
DeltaMessage,
|
||||||
@@ -115,6 +116,23 @@ def sample_tools(request):
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _with_strict(
|
||||||
|
tools: list[ChatCompletionToolsParam],
|
||||||
|
) -> list[ChatCompletionToolsParam]:
|
||||||
|
return [
|
||||||
|
ChatCompletionToolsParam(
|
||||||
|
type=t.type,
|
||||||
|
function=FunctionDefinition(
|
||||||
|
name=t.function.name,
|
||||||
|
description=t.function.description,
|
||||||
|
parameters=t.function.parameters,
|
||||||
|
strict=True,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for t in tools
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def _as_chat_completion_tools(
|
def _as_chat_completion_tools(
|
||||||
tools: list[ChatCompletionToolsParam | FunctionTool],
|
tools: list[ChatCompletionToolsParam | FunctionTool],
|
||||||
) -> list[ChatCompletionToolsParam]:
|
) -> list[ChatCompletionToolsParam]:
|
||||||
@@ -1323,10 +1341,11 @@ def test_get_vllm_registry_structural_tag_returns_structural_tag(
|
|||||||
sample_tools: list[ChatCompletionToolsParam],
|
sample_tools: list[ChatCompletionToolsParam],
|
||||||
) -> None:
|
) -> None:
|
||||||
request_tools = _as_chat_completion_tools(sample_tools)
|
request_tools = _as_chat_completion_tools(sample_tools)
|
||||||
|
strict_tools = _with_strict(request_tools)
|
||||||
req = ChatCompletionRequest(
|
req = ChatCompletionRequest(
|
||||||
messages=[],
|
messages=[],
|
||||||
model="m",
|
model="m",
|
||||||
tools=request_tools,
|
tools=strict_tools,
|
||||||
tool_choice="auto",
|
tool_choice="auto",
|
||||||
)
|
)
|
||||||
tag = qwen3_tool_parser.get_structural_tag(req)
|
tag = qwen3_tool_parser.get_structural_tag(req)
|
||||||
@@ -1364,10 +1383,11 @@ def test_adjust_request_auto_uses_vllm_registry_structural_tag(
|
|||||||
tool_parser_cls = Qwen3EngineToolParser
|
tool_parser_cls = Qwen3EngineToolParser
|
||||||
|
|
||||||
request_tools = _as_chat_completion_tools(sample_tools)
|
request_tools = _as_chat_completion_tools(sample_tools)
|
||||||
|
strict_tools = _with_strict(request_tools)
|
||||||
req = ChatCompletionRequest(
|
req = ChatCompletionRequest(
|
||||||
messages=[],
|
messages=[],
|
||||||
model="m",
|
model="m",
|
||||||
tools=request_tools,
|
tools=strict_tools,
|
||||||
tool_choice="auto",
|
tool_choice="auto",
|
||||||
include_reasoning=include_reasoning,
|
include_reasoning=include_reasoning,
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -51,6 +51,24 @@ def sample_tools() -> list[ChatCompletionToolsParam]:
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def sample_tools_strict() -> list[ChatCompletionToolsParam]:
|
||||||
|
return [
|
||||||
|
ChatCompletionToolsParam(
|
||||||
|
type="function",
|
||||||
|
function={
|
||||||
|
"name": "get_weather",
|
||||||
|
"strict": True,
|
||||||
|
"parameters": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {"city": {"type": "string"}},
|
||||||
|
"required": ["city"],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def test_supported_structural_tag_models_include_vllm_builtins():
|
def test_supported_structural_tag_models_include_vllm_builtins():
|
||||||
assert SUPPORTED_STRUCTURAL_TAG_MODELS == (
|
assert SUPPORTED_STRUCTURAL_TAG_MODELS == (
|
||||||
XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS | VLLM_BUILTIN_STRUCTURAL_TAG_MODELS
|
XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS | VLLM_BUILTIN_STRUCTURAL_TAG_MODELS
|
||||||
@@ -61,11 +79,11 @@ def test_supported_structural_tag_models_include_vllm_builtins():
|
|||||||
@pytest.mark.parametrize("model", sorted(XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS))
|
@pytest.mark.parametrize("model", sorted(XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS))
|
||||||
def test_get_model_structural_tag_supports_all_xgrammar_builtins(
|
def test_get_model_structural_tag_supports_all_xgrammar_builtins(
|
||||||
model: str,
|
model: str,
|
||||||
sample_tools: list[ChatCompletionToolsParam],
|
sample_tools_strict: list[ChatCompletionToolsParam],
|
||||||
):
|
):
|
||||||
tag = get_model_structural_tag(
|
tag = get_model_structural_tag(
|
||||||
model=model,
|
model=model,
|
||||||
tools=sample_tools,
|
tools=sample_tools_strict,
|
||||||
tool_choice="auto",
|
tool_choice="auto",
|
||||||
reasoning=False,
|
reasoning=False,
|
||||||
)
|
)
|
||||||
@@ -219,7 +237,7 @@ def test_non_structural_tag_parser_uses_schema_constraints(
|
|||||||
|
|
||||||
def test_get_structural_tag_disables_reasoning(
|
def test_get_structural_tag_disables_reasoning(
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
sample_tools: list[ChatCompletionToolsParam],
|
sample_tools_strict: list[ChatCompletionToolsParam],
|
||||||
):
|
):
|
||||||
captured: list[bool] = []
|
captured: list[bool] = []
|
||||||
|
|
||||||
@@ -235,10 +253,10 @@ def test_get_structural_tag_disables_reasoning(
|
|||||||
request = ChatCompletionRequest(
|
request = ChatCompletionRequest(
|
||||||
messages=[],
|
messages=[],
|
||||||
model="m",
|
model="m",
|
||||||
tools=sample_tools,
|
tools=sample_tools_strict,
|
||||||
tool_choice="auto",
|
tool_choice="auto",
|
||||||
)
|
)
|
||||||
parser = Qwen3EngineToolParser(MagicMock(), tools=sample_tools)
|
parser = Qwen3EngineToolParser(MagicMock(), tools=sample_tools_strict)
|
||||||
|
|
||||||
parser.get_structural_tag(request)
|
parser.get_structural_tag(request)
|
||||||
|
|
||||||
@@ -247,7 +265,7 @@ def test_get_structural_tag_disables_reasoning(
|
|||||||
|
|
||||||
def test_unified_parser_get_structural_tag_disables_reasoning(
|
def test_unified_parser_get_structural_tag_disables_reasoning(
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
sample_tools: list[ChatCompletionToolsParam],
|
sample_tools_strict: list[ChatCompletionToolsParam],
|
||||||
):
|
):
|
||||||
captured: list[bool] = []
|
captured: list[bool] = []
|
||||||
|
|
||||||
@@ -266,10 +284,10 @@ def test_unified_parser_get_structural_tag_disables_reasoning(
|
|||||||
request = ChatCompletionRequest(
|
request = ChatCompletionRequest(
|
||||||
messages=[],
|
messages=[],
|
||||||
model="m",
|
model="m",
|
||||||
tools=sample_tools,
|
tools=sample_tools_strict,
|
||||||
tool_choice="auto",
|
tool_choice="auto",
|
||||||
)
|
)
|
||||||
parser = TestParser(MagicMock(), tools=sample_tools)
|
parser = TestParser(MagicMock(), tools=sample_tools_strict)
|
||||||
parser.reasoning_parser = MagicMock(adjust_request=lambda request: request)
|
parser.reasoning_parser = MagicMock(adjust_request=lambda request: request)
|
||||||
|
|
||||||
parser.adjust_request(request)
|
parser.adjust_request(request)
|
||||||
@@ -279,7 +297,7 @@ def test_unified_parser_get_structural_tag_disables_reasoning(
|
|||||||
|
|
||||||
def test_xgrammar_function_parameters_are_preserved(
|
def test_xgrammar_function_parameters_are_preserved(
|
||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
sample_tools: list[ChatCompletionToolsParam],
|
sample_tools_strict: list[ChatCompletionToolsParam],
|
||||||
):
|
):
|
||||||
captured: list[list[dict]] = []
|
captured: list[list[dict]] = []
|
||||||
|
|
||||||
@@ -294,15 +312,31 @@ def test_xgrammar_function_parameters_are_preserved(
|
|||||||
|
|
||||||
get_model_structural_tag(
|
get_model_structural_tag(
|
||||||
model="llama",
|
model="llama",
|
||||||
tools=sample_tools,
|
tools=sample_tools_strict,
|
||||||
tool_choice="auto",
|
tool_choice="auto",
|
||||||
reasoning=False,
|
reasoning=False,
|
||||||
)
|
)
|
||||||
|
|
||||||
assert (
|
assert (
|
||||||
captured[0][0]["function"]["parameters"] == sample_tools[0].function.parameters
|
captured[0][0]["function"]["parameters"]
|
||||||
|
== sample_tools_strict[0].function.parameters
|
||||||
)
|
)
|
||||||
assert sample_tools[0].function.parameters is not None
|
assert sample_tools_strict[0].function.parameters is not None
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("model", sorted(XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS))
|
||||||
|
def test_auto_tool_choice_skips_structural_tag_without_strict(
|
||||||
|
model: str,
|
||||||
|
sample_tools: list[ChatCompletionToolsParam],
|
||||||
|
):
|
||||||
|
tag = get_model_structural_tag(
|
||||||
|
model=model,
|
||||||
|
tools=sample_tools,
|
||||||
|
tool_choice="auto",
|
||||||
|
reasoning=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert tag is None
|
||||||
|
|
||||||
|
|
||||||
def test_get_function_parameters_relaxes_function_strict_false():
|
def test_get_function_parameters_relaxes_function_strict_false():
|
||||||
|
|||||||
@@ -75,6 +75,7 @@ class AnthropicTool(BaseModel):
|
|||||||
name: str
|
name: str
|
||||||
description: str | None = None
|
description: str | None = None
|
||||||
input_schema: dict[str, Any]
|
input_schema: dict[str, Any]
|
||||||
|
strict: bool | None = None
|
||||||
defer_loading: bool | None = None
|
defer_loading: bool | None = None
|
||||||
|
|
||||||
@field_validator("input_schema")
|
@field_validator("input_schema")
|
||||||
|
|||||||
@@ -462,6 +462,7 @@ class AnthropicServingMessages(OpenAIServingChat):
|
|||||||
"name": tool.name,
|
"name": tool.name,
|
||||||
"description": tool.description,
|
"description": tool.description,
|
||||||
"parameters": tool.input_schema,
|
"parameters": tool.input_schema,
|
||||||
|
"strict": tool.strict,
|
||||||
"defer_loading": tool.defer_loading,
|
"defer_loading": tool.defer_loading,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -247,11 +247,14 @@ class FunctionDefinition(OpenAIBaseModel):
|
|||||||
name: str
|
name: str
|
||||||
description: str | None = None
|
description: str | None = None
|
||||||
parameters: dict[str, Any] | None = None
|
parameters: dict[str, Any] | None = None
|
||||||
|
strict: bool | None = None
|
||||||
defer_loading: bool | None = None
|
defer_loading: bool | None = None
|
||||||
|
|
||||||
@model_serializer(mode="wrap")
|
@model_serializer(mode="wrap")
|
||||||
def _serialize(self, handler):
|
def _serialize(self, handler):
|
||||||
data = handler(self)
|
data = handler(self)
|
||||||
|
if self.strict is None:
|
||||||
|
data.pop("strict", None)
|
||||||
if self.defer_loading is None:
|
if self.defer_loading is None:
|
||||||
data.pop("defer_loading", None)
|
data.pop("defer_loading", None)
|
||||||
return data
|
return data
|
||||||
|
|||||||
@@ -84,6 +84,17 @@ def register_vllm_structural_tag(model: str):
|
|||||||
return decorator
|
return decorator
|
||||||
|
|
||||||
|
|
||||||
|
def _any_tool_strict(
|
||||||
|
tools: Sequence[ChatCompletionToolsParam | ResponsesTool],
|
||||||
|
) -> bool:
|
||||||
|
for tool in tools:
|
||||||
|
if isinstance(tool, FunctionTool) and tool.strict is True:
|
||||||
|
return True
|
||||||
|
if isinstance(tool, ChatCompletionToolsParam) and tool.function.strict is True:
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
def get_model_structural_tag(
|
def get_model_structural_tag(
|
||||||
model: str,
|
model: str,
|
||||||
tools: Sequence[ChatCompletionToolsParam | ResponsesTool] | None,
|
tools: Sequence[ChatCompletionToolsParam | ResponsesTool] | None,
|
||||||
@@ -95,6 +106,9 @@ def get_model_structural_tag(
|
|||||||
if not tools or tool_choice == "none":
|
if not tools or tool_choice == "none":
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
if tool_choice == "auto" and not _any_tool_strict(tools):
|
||||||
|
return None
|
||||||
|
|
||||||
dumped_tools = [_dump_tool_for_xgrammar(tool) for tool in tools]
|
dumped_tools = [_dump_tool_for_xgrammar(tool) for tool in tools]
|
||||||
dumped_tool_choice = _dump_tool_choice_for_xgrammar(tool_choice)
|
dumped_tool_choice = _dump_tool_choice_for_xgrammar(tool_choice)
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user