iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[Frontend] Add --tool-strict-level for server-side control for structural tag activation (#56268)

Browse Source 
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: lazymio <mio@lazym.io>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
This commit is contained in:
lazymio
2026-09-18 04:32:39 +00:00
committed by GitHub
co-authored by Claude Opus 5 Bugen Zhao
parent d12c276853
commit dee37d8911
35 changed files with 742 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/features/tool_calling.md
+30 -10
View File
@@ -109,18 +109,22 @@ vLLM supports the `tool_choice='none'` option in the chat completion API. When t
## Constrained Decoding Behavior
Whether vLLM enforces the tool parameter schema during generation depends on the `tool_choice` mode and the per-tool `strict` field:
Structural-tag parsers resolve call obligation, grammar activation, and argument-schema enforcement separately. With structural-tag enforcement enabled:
| `tool_choice` value | Schema-constrained decoding | Behavior |
| `tool_choice` value | Call obligation | Structural-tag activation |
| --- | --- | --- |
| Named function | Yes (via structured outputs backend) | Arguments are guaranteed to be valid JSON conforming to the function's parameter schema. |
| `"required"` | Yes (via structured outputs backend) | Same as named function. The model must produce at least one tool call. |
| `"auto"` | Only when `strict: true` is set on at least one tool | Structural-tag parsers constrain tool-call arguments when a tool opts in with `strict: true`. Without it, the model generates freely and tool calls are extracted from raw text. |
| `"none"` | N/A | No tool calls are produced. |
| Named function | Call the selected function | Always |
| `"required"` | Produce at least one tool call | Always |
| `"auto"` | Tool calls are optional | When at least one tool sets `strict: true`, or `--tool-strict-level` is `function` or `parameter` |
| `"none"` | Tool calling is disabled | Disabled |
When a structural tag applies, each tool's declared parameter schema is enforced only when that tool sets `strict: true` or the server uses `--tool-strict-level parameter`. Tools with omitted or false `strict` receive broad argument-syntax constraints at levels `auto` and `function`, including for required and named calls.
For parsers using schema-derived JSON constraints, required and named calls continue to enforce the declared parameter schemas.
### Strict Mode
For `tool_choice="required"` or named function calling, structural-tag constraints are always applied regardless of the `strict` field. For `tool_choice="auto"`, setting `strict: true` on at least one tool opts in to structural-tag constraints; without it, the model generates freely and tool calls are extracted from raw text. The `strict` field is supported across all three API surfaces: Chat Completion, Responses, and Anthropic Messages.
Structural-tag constraints have two layers. The tool-call *envelope* (markup and function name) is constrained whenever a structural tag applies: always for `tool_choice="required"` and named function calling, and for `tool_choice="auto"` once at least one tool sets `strict: true` or the server raises the floor. The *argument schema* of an individual tool is pinned only when that tool sets `strict: true` (or under `--tool-strict-level parameter`); a tool that omits `strict` keeps its arguments unconstrained, even when another tool in the same request is strict. The `strict` field is supported across all three API surfaces: Chat Completion, Responses, and Anthropic Messages.
For best compatibility with strict schema enforcement, define tool parameter schemas in the OpenAI strict-schema style:
@@ -134,6 +138,22 @@ vLLM also provides a global toggle via the `VLLM_ENFORCE_STRICT_TOOL_CALLING` en
VLLM_ENFORCE_STRICT_TOOL_CALLING=false vllm serve ...
```
### Server-Side Strictness Floor
Most OpenAI-compatible clients and agent frameworks never set `strict` on their tools, so with `tool_choice="auto"` the model generates tool calls without any grammar and malformed markup can leak into the response. The `--tool-strict-level` option lets the server operator raise the floor for every request that carries tools, independently of what the client declares:
| Value | Behavior |
| --- | --- |
| `auto` (default) | Follow the request's tool choice and per-tool strictness. Required/named choices activate structural tags; `tool_choice="auto"` activates them when at least one tool sets `strict: true`. |
| `function` | Constrain the tool-call envelope (markup and the function name) for every request with tools, leaving argument contents free unless the client marked the tool `strict: true`. |
| `parameter` | Additionally pin argument schemas for every tool, as if every tool had `strict: true`. |
```bash
vllm serve ... --tool-strict-level function
```
The floor never relaxes a constraint the request would already receive: tools the client marked `strict: true` keep their schemas at every level. With `tool_choice="auto"`, the grammar does not force a tool call; a plain text response stays valid. `VLLM_ENFORCE_STRICT_TOOL_CALLING=false` disables structural tags entirely and takes precedence over this option.
## Automatic Function Calling
To enable this feature, you should set the following flags:
@@ -152,7 +172,7 @@ from HuggingFace; and you can find an example of this in a `tokenizer_config.jso
If your favorite tool-calling model is not supported, please feel free to contribute a parser & tool use chat template!
!!! note
With `tool_choice="auto"`, schema-level constraint requires both `VLLM_ENFORCE_STRICT_TOOL_CALLING=true` (the default) and at least one tool with `strict: true`. When these conditions are met and the selected parser supports structural tags, vLLM constrains tool-call arguments. Otherwise, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema.
With `tool_choice="auto"`, structural-tag constraints require both `VLLM_ENFORCE_STRICT_TOOL_CALLING=true` (the default) and at least one tool with `strict: true`, or a server-side floor set via `--tool-strict-level`. When these conditions are met and the selected parser supports structural tags, vLLM constrains the tool-call envelope and pins the argument schema of each tool that sets `strict: true` (or of every tool under `--tool-strict-level parameter`). Otherwise, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema.
### Hermes Models (`hermes`)
@@ -162,8 +182,8 @@ All Nous Research Hermes-series models newer than Hermes 2 Pro should be support
* `NousResearch/Hermes-2-Theta-*`
* `NousResearch/Hermes-3-*`
_Note that the Hermes 2 **Theta** models are known to have degraded tool call quality and capabilities due to the merge
step in their creation_.
*Note that the Hermes 2 **Theta** models are known to have degraded tool call quality and capabilities due to the merge
step in their creation*.
Flags: `--tool-call-parser hermes`
rust/src/chat/src/backend/hf.rs
+2
View File
@@ -134,6 +134,7 @@ impl ChatBackend for HfChatBackend {
self.tokenizer.clone(),
options.tool_call_parser,
options.reasoning_parser,
options.tool_strict_level,
)?))
}
}
@@ -343,6 +344,7 @@ mod tests {
let error = match backend.new_chat_output_processor(
&mut request,
NewChatOutputProcessorOptions {
tool_strict_level: crate::ToolStrictLevel::Auto,
tool_call_parser: &ParserSelection::Explicit("json".to_string()),
reasoning_parser: &ParserSelection::Auto,
},
rust/src/chat/src/backend/mod.rs
+2 -1
View File
@@ -13,7 +13,7 @@ use crate::multimodal::{MmLimitPerPrompt, MultimodalModelInfo};
use crate::output::DynChatOutputProcessor;
use crate::renderer::DynChatRenderer;
use crate::request::ChatRequest;
use crate::{ChatTemplateContentFormatOption, ParserSelection, RendererSelection};
use crate::{ChatTemplateContentFormatOption, ParserSelection, RendererSelection, ToolStrictLevel};
pub mod hf;
@@ -21,6 +21,7 @@ pub mod hf;
pub struct NewChatOutputProcessorOptions<'a> {
pub tool_call_parser: &'a ParserSelection,
pub reasoning_parser: &'a ParserSelection,
pub tool_strict_level: ToolStrictLevel,
}
/// Minimal prompt-processing backend needed by `vllm-chat`.
rust/src/chat/src/lib.rs
+18 -1
View File
@@ -30,7 +30,7 @@ pub use parser::reasoning::{
ReasoningDelta, ReasoningError, ReasoningParser, ReasoningParserFactory,
};
pub use parser::tool::{ToolParser, ToolParserError, ToolParserFactory};
pub use parser::{ParserSelection, validate_parser_overrides};
pub use parser::{ParserSelection, ToolStrictLevel, validate_parser_overrides};
pub use reasoning::EffortValue;
pub use renderer::hf::ChatTemplateContentFormatOption;
pub use renderer::{
@@ -74,6 +74,8 @@ pub struct ChatRequestProcessor {
tool_call_parser: ParserSelection,
/// Reasoning parser selection used when preparing generation requests.
reasoning_parser: ParserSelection,
/// Server-side floor for tool-call structural tags.
tool_strict_level: ToolStrictLevel,
}
impl ChatRequestProcessor {
@@ -85,6 +87,7 @@ impl ChatRequestProcessor {
model_dtype: Some(model_dtype),
tool_call_parser: ParserSelection::Auto,
reasoning_parser: ParserSelection::Auto,
tool_strict_level: ToolStrictLevel::Auto,
}
}
@@ -95,6 +98,7 @@ impl ChatRequestProcessor {
model_dtype: None,
tool_call_parser: ParserSelection::Auto,
reasoning_parser: ParserSelection::Auto,
tool_strict_level: ToolStrictLevel::Auto,
}
}
@@ -109,6 +113,12 @@ impl ChatRequestProcessor {
self
}
/// Configure the server-side floor for tool-call structural tags.
pub fn with_tool_strict_level(mut self, tool_strict_level: ToolStrictLevel) -> Self {
self.tool_strict_level = tool_strict_level;
self
}
async fn finalize_rendered_prompt(
&self,
request: &ChatRequest,
@@ -207,6 +217,7 @@ impl ChatRequestProcessor {
NewChatOutputProcessorOptions {
tool_call_parser: &self.tool_call_parser,
reasoning_parser: &self.reasoning_parser,
tool_strict_level: self.tool_strict_level,
},
)?;
let text_request = self.prepare_text_request(request).await?;
@@ -255,6 +266,12 @@ impl ChatLlm {
self
}
/// Set the server-side floor for tool-call structural tags.
pub fn with_tool_strict_level(mut self, tool_strict_level: ToolStrictLevel) -> Self {
self.processor.tool_strict_level = tool_strict_level;
self
}
/// Override the effective model dtype used for multimodal tensor encoding.
pub fn with_model_dtype(mut self, model_dtype: ModelDtype) -> Self {
self.processor.model_dtype = Some(model_dtype);
rust/src/chat/src/output/default/mod.rs
+12 -3
View File
@@ -18,10 +18,10 @@ use self::unified::unified_event_stream;
use super::structured::structured_chat_event_stream;
use crate::error::Result;
use crate::output::{ChatOutputProcessor, DynChatEventStream, DynDecodedTextEventStream};
use crate::parser::ParserSelection;
use crate::parser::reasoning::{ReasoningParser, ReasoningParserFactory};
use crate::parser::tool::{ToolParser, ToolParserFactory};
use crate::parser::unified::UnifiedParserFactory;
use crate::parser::{ParserSelection, ToolStrictLevel};
use crate::request::{ChatRequest, ChatTool};
use crate::{Error, Result as ChatResult};
@@ -49,6 +49,7 @@ impl DefaultChatOutputProcessor {
tokenizer: DynTokenizer,
tool_call_parser: &ParserSelection,
reasoning_parser: &ParserSelection,
tool_strict_level: ToolStrictLevel,
) -> ChatResult<Self> {
let parser = if let Some(parser) = Self::resolve_optional_unified_parser(
request.tools(),
@@ -73,7 +74,11 @@ impl DefaultChatOutputProcessor {
Box::new(CombinedParser::new(reasoning_parser, tool_parser)) as Box<dyn UnifiedParser>
};
apply_structural_tag_constraint(request, parser.structural_tag_builder())?;
apply_structural_tag_constraint(
request,
parser.structural_tag_builder(),
tool_strict_level,
)?;
if parser.preserve_special_tokens() {
request.decode_options.skip_special_tokens = false;
@@ -191,7 +196,7 @@ mod tests {
use vllm_tokenizer::test_utils::TestTokenizer;
use super::DefaultChatOutputProcessor;
use crate::parser::ParserSelection;
use crate::parser::{ParserSelection, ToolStrictLevel};
use crate::request::ChatRequest;
fn tokenizer() -> Arc<TestTokenizer> {
@@ -213,6 +218,7 @@ mod tests {
tokenizer(),
&selection,
&selection,
ToolStrictLevel::Auto,
)
.unwrap();
}
@@ -227,6 +233,7 @@ mod tests {
tokenizer(),
&ParserSelection::Auto,
&ParserSelection::Auto,
ToolStrictLevel::Auto,
)
.unwrap();
}
@@ -244,6 +251,7 @@ mod tests {
tokenizer(),
tool,
reasoning,
ToolStrictLevel::Auto,
)
.unwrap();
}
@@ -258,6 +266,7 @@ mod tests {
tokenizer(),
&ParserSelection::Auto,
&ParserSelection::Explicit("gemma4".to_string()),
ToolStrictLevel::Auto,
) {
Ok(_) => panic!("expected mixed Gemma4 parser selection to fail"),
Err(error) => error,
rust/src/chat/src/output/default/structural_tag.rs
+191 -29
View File
@@ -14,19 +14,21 @@ use xgrammar_structural_tag::{
build_structural_tag,
};
use crate::parser::ToolStrictLevel;
use crate::request::{ChatRequest, ChatToolChoice};
use crate::{Error, Result as ChatResult};
/// Apply structural tag constraints to the request based on the tool parser's structural tag
/// support and the request's tool choice.
/// support, the request's tool choice, and the server-side strictness floor.
pub(super) fn apply_structural_tag_constraint(
request: &mut ChatRequest,
builder: Option<&dyn StructuralTagBuilder>,
strict_level: ToolStrictLevel,
) -> ChatResult<()> {
let Some(builder) = builder else {
return Ok(());
};
let Some(tool_choice) = structural_tag_tool_choice(request) else {
let Some(tool_choice) = structural_tag_tool_choice(request, strict_level) else {
return Ok(());
};
@@ -34,11 +36,14 @@ pub(super) fn apply_structural_tag_constraint(
.tools()
.iter()
.map(|tool| {
// A tool without an explicit `strict` is non-strict: its call envelope is
// constrained, but its arguments stay free unless the server level is `parameter`.
let strict = strict_level >= ToolStrictLevel::Parameter || tool.strict == Some(true);
ToolParam::Function(FunctionToolParam::new(FunctionDefinition {
name: tool.name.clone(),
description: tool.description.clone(),
parameters: Some(tool.parameters.clone()),
strict: tool.strict,
strict: Some(strict),
}))
})
.collect::<Vec<_>>();
@@ -66,17 +71,21 @@ pub(super) fn apply_structural_tag_constraint(
/// Resolve the tool choice used for [`xgrammar_structural_tag`] based on the request.
///
/// Returns `None` if no structural tag constraints should be applied.
fn structural_tag_tool_choice(request: &ChatRequest) -> Option<StructuralTagToolChoice> {
fn structural_tag_tool_choice(
request: &ChatRequest,
strict_level: ToolStrictLevel,
) -> Option<StructuralTagToolChoice> {
if request.tools().is_empty() {
return None;
}
match request.tool_choice() {
// For `Auto`, only apply the structural tag if there's at least one strict tool.
ChatToolChoice::Auto if request.tools().iter().any(|tool| tool.strict == Some(true)) => {
Some(StructuralTagToolChoice::auto())
}
ChatToolChoice::Auto | ChatToolChoice::None => None,
ChatToolChoice::None => None,
// For `Auto`, apply the structural tag only when the server raises the floor or at
// least one tool opts in with `strict: true`.
ChatToolChoice::Auto => (strict_level >= ToolStrictLevel::Function
|| request.tools().iter().any(|tool| tool.strict == Some(true)))
.then(StructuralTagToolChoice::auto),
ChatToolChoice::Required => Some(StructuralTagToolChoice::required()),
ChatToolChoice::Function { name } => Some(StructuralTagToolChoice::function(name.clone())),
@@ -148,8 +157,12 @@ mod tests {
let mut request = request(ChatToolChoice::Auto, vec![chat_tool("search", Some(true))]);
let parser = qwen3_coder_parser(request.tools());
apply_structural_tag_constraint(&mut request, parser.structural_tag_builder())
.expect("structural tag should build");
apply_structural_tag_constraint(
&mut request,
parser.structural_tag_builder(),
ToolStrictLevel::Auto,
)
.expect("structural tag should build");
let tag = structural_tag_value(&request);
assert_eq!(tag["type"], "structural_tag");
@@ -172,8 +185,12 @@ mod tests {
};
let parser = qwen3_coder_parser(request.tools());
apply_structural_tag_constraint(&mut request, parser.structural_tag_builder())
.expect("structural tag should build");
apply_structural_tag_constraint(
&mut request,
parser.structural_tag_builder(),
ToolStrictLevel::Auto,
)
.expect("structural tag should build");
let tag = structural_tag_value(&request);
assert!(tag.to_string().contains("lookup"));
@@ -199,8 +216,12 @@ mod tests {
};
let parser = qwen3_coder_parser(request.tools());
apply_structural_tag_constraint(&mut request, parser.structural_tag_builder())
.expect("structural tag should build");
apply_structural_tag_constraint(
&mut request,
parser.structural_tag_builder(),
ToolStrictLevel::Auto,
)
.expect("structural tag should build");
let tag = structural_tag_value(&request).to_string();
assert!(tag.contains("search"));
@@ -212,8 +233,12 @@ mod tests {
let mut request = request(ChatToolChoice::Auto, vec![chat_tool("search", None)]);
let parser = qwen3_coder_parser(request.tools());
apply_structural_tag_constraint(&mut request, parser.structural_tag_builder())
.expect("structural tag decision should succeed");
apply_structural_tag_constraint(
&mut request,
parser.structural_tag_builder(),
ToolStrictLevel::Auto,
)
.expect("structural tag decision should succeed");
assert!(request.sampling_params.structured_outputs.is_none());
}
@@ -227,8 +252,12 @@ mod tests {
});
let parser = qwen3_coder_parser(request.tools());
apply_structural_tag_constraint(&mut request, parser.structural_tag_builder())
.expect("structural tag should build");
apply_structural_tag_constraint(
&mut request,
parser.structural_tag_builder(),
ToolStrictLevel::Auto,
)
.expect("structural tag should build");
let params = structured_outputs(&request);
assert!(params.constraint.is_structural_tag());
@@ -242,8 +271,12 @@ mod tests {
let mut request = request(ChatToolChoice::Required, vec![chat_tool("search", None)]);
let parser = qwen3_coder_parser(request.tools());
apply_structural_tag_constraint(&mut request, parser.structural_tag_builder())
.expect("structural tag should build");
apply_structural_tag_constraint(
&mut request,
parser.structural_tag_builder(),
ToolStrictLevel::Auto,
)
.expect("structural tag should build");
let tag = structural_tag_value(&request);
assert_eq!(tag["type"], "structural_tag");
@@ -259,8 +292,12 @@ mod tests {
});
let parser = qwen3_coder_parser(request.tools());
apply_structural_tag_constraint(&mut request, parser.structural_tag_builder())
.expect("structural tag should build");
apply_structural_tag_constraint(
&mut request,
parser.structural_tag_builder(),
ToolStrictLevel::Auto,
)
.expect("structural tag should build");
let params = structured_outputs(&request);
assert!(params.constraint.is_structural_tag());
@@ -279,8 +316,12 @@ mod tests {
);
let parser = qwen3_coder_parser(request.tools());
apply_structural_tag_constraint(&mut request, parser.structural_tag_builder())
.expect("structural tag should build");
apply_structural_tag_constraint(
&mut request,
parser.structural_tag_builder(),
ToolStrictLevel::Auto,
)
.expect("structural tag should build");
let tag = structural_tag_value(&request).to_string();
assert!(tag.contains("lookup"));
@@ -292,8 +333,12 @@ mod tests {
let mut request = request(ChatToolChoice::None, vec![chat_tool("search", Some(true))]);
let parser = qwen3_coder_parser(request.tools());
apply_structural_tag_constraint(&mut request, parser.structural_tag_builder())
.expect("structural tag decision should succeed");
apply_structural_tag_constraint(
&mut request,
parser.structural_tag_builder(),
ToolStrictLevel::Auto,
)
.expect("structural tag decision should succeed");
assert!(request.sampling_params.structured_outputs.is_none());
}
@@ -307,10 +352,127 @@ mod tests {
});
let parser = qwen3_coder_parser(request.tools());
apply_structural_tag_constraint(&mut request, parser.structural_tag_builder())
.expect("structural tag decision should succeed");
apply_structural_tag_constraint(
&mut request,
parser.structural_tag_builder(),
ToolStrictLevel::Auto,
)
.expect("structural tag decision should succeed");
let params = structured_outputs(&request);
assert!(params.constraint.is_json_object());
}
fn build(tool_choice: ChatToolChoice, tools: Vec<Tool>, level: ToolStrictLevel) -> ChatRequest {
let mut request = request(tool_choice, tools);
let parser = qwen3_coder_parser(request.tools());
apply_structural_tag_constraint(&mut request, parser.structural_tag_builder(), level)
.expect("structural tag decision should succeed");
request
}
fn tag_string(request: &ChatRequest) -> String {
structural_tag_value(request).to_string()
}
#[test]
fn function_level_constrains_auto_without_strict_tools() {
let request = build(
ChatToolChoice::Auto,
vec![chat_tool("search", None)],
ToolStrictLevel::Function,
);
let tag = structural_tag_value(&request);
assert_eq!(tag["type"], "structural_tag");
assert!(tag.to_string().contains("search"));
}
#[test]
fn absent_strict_is_non_strict() {
let unset = build(
ChatToolChoice::Required,
vec![chat_tool("search", None)],
ToolStrictLevel::Auto,
);
let non_strict = build(
ChatToolChoice::Required,
vec![chat_tool("search", Some(false))],
ToolStrictLevel::Auto,
);
let strict = build(
ChatToolChoice::Required,
vec![chat_tool("search", Some(true))],
ToolStrictLevel::Auto,
);
assert_eq!(tag_string(&unset), tag_string(&non_strict));
assert_ne!(tag_string(&unset), tag_string(&strict));
}
#[test]
fn strict_tool_does_not_pin_its_neighbour() {
let mixed = build(
ChatToolChoice::Auto,
vec![chat_tool("weather", Some(true)), chat_tool("search", None)],
ToolStrictLevel::Auto,
);
let explicit = build(
ChatToolChoice::Auto,
vec![
chat_tool("weather", Some(true)),
chat_tool("search", Some(false)),
],
ToolStrictLevel::Auto,
);
assert_eq!(tag_string(&mixed), tag_string(&explicit));
}
#[test]
fn parameter_level_pins_argument_schemas() {
let pinned = build(
ChatToolChoice::Auto,
vec![chat_tool("search", None)],
ToolStrictLevel::Parameter,
);
let strict = build(
ChatToolChoice::Auto,
vec![chat_tool("search", Some(true))],
ToolStrictLevel::Auto,
);
let envelope_only = build(
ChatToolChoice::Auto,
vec![chat_tool("search", None)],
ToolStrictLevel::Function,
);
assert_eq!(tag_string(&pinned), tag_string(&strict));
assert_ne!(tag_string(&pinned), tag_string(&envelope_only));
}
#[test]
fn tool_strict_level_parses_case_insensitively() {
assert_eq!("AUTO".parse::<ToolStrictLevel>(), Ok(ToolStrictLevel::Auto));
assert_eq!(
"PARAMETER".parse::<ToolStrictLevel>(),
Ok(ToolStrictLevel::Parameter)
);
assert_eq!(
"function".parse::<ToolStrictLevel>(),
Ok(ToolStrictLevel::Function)
);
assert_eq!(ToolStrictLevel::default(), ToolStrictLevel::Auto);
assert_eq!(
serde_json::to_value(ToolStrictLevel::Auto).unwrap(),
json!("auto")
);
assert_eq!(
serde_json::from_value::<ToolStrictLevel>(json!("auto")).unwrap(),
ToolStrictLevel::Auto
);
assert!("off".parse::<ToolStrictLevel>().is_err());
assert!("strict".parse::<ToolStrictLevel>().is_err());
}
}
rust/src/chat/src/parser/mod.rs
+60
View File
@@ -57,6 +57,66 @@ impl ParserSelection {
}
}
/// Server-side floor for tool-call structural tags (`--tool-strict-level`).
#[derive(
Debug,
Clone,
Copy,
PartialEq,
Eq,
PartialOrd,
Ord,
Default,
DeserializeFromStr,
SerializeDisplay,
)]
pub enum ToolStrictLevel {
/// Derive tool constraints from the request's tool choice and per-tool strictness.
#[default]
Auto,
/// Constrain the tool-call envelope for every request with tools.
Function,
/// Additionally pin argument schemas, as if every tool were `strict: true`.
Parameter,
}
impl ToolStrictLevel {
pub const AUTO_LITERAL: &str = "auto";
pub const FUNCTION_LITERAL: &str = "function";
pub const PARAMETER_LITERAL: &str = "parameter";
}
impl FromStr for ToolStrictLevel {
type Err = String;
fn from_str(value: &str) -> Result<Self, Self::Err> {
if value.eq_ignore_ascii_case(Self::AUTO_LITERAL) {
Ok(Self::Auto)
} else if value.eq_ignore_ascii_case(Self::FUNCTION_LITERAL) {
Ok(Self::Function)
} else if value.eq_ignore_ascii_case(Self::PARAMETER_LITERAL) {
Ok(Self::Parameter)
} else {
Err(format!(
"unknown tool strict level {value:?}; expected one of {}, {}, {}",
Self::AUTO_LITERAL,
Self::FUNCTION_LITERAL,
Self::PARAMETER_LITERAL
))
}
}
}
impl fmt::Display for ToolStrictLevel {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str(match self {
Self::Auto => Self::AUTO_LITERAL,
Self::Function => Self::FUNCTION_LITERAL,
Self::Parameter => Self::PARAMETER_LITERAL,
})
}
}
/// Validate explicit parser override names without starting request processing.
pub fn validate_parser_overrides(
tool_call_parser: &ParserSelection,
rust/src/chat/tests/chat.rs
+1
View File
@@ -227,6 +227,7 @@ impl ChatBackend for FakeChatBackend {
self.tokenizer(),
options.tool_call_parser,
options.reasoning_parser,
options.tool_strict_level,
)?))
}
}
rust/src/chat/tests/roundtrip.rs
+1
View File
@@ -717,6 +717,7 @@ async fn parse_completion(
let processor = backends.chat_backend.new_chat_output_processor(
&mut request,
NewChatOutputProcessorOptions {
tool_strict_level: vllm_chat::ToolStrictLevel::Auto,
tool_call_parser: &case.tool_call_parser,
reasoning_parser: &case.reasoning_parser,
},
rust/src/cmd/src/cli.rs
+16
View File
@@ -24,6 +24,7 @@ use serde_with::{DefaultOnNull, OneOrMany, serde_as};
use thiserror_ext::AsReport as _;
use uuid::Uuid;
use vllm_chat::GenerationConfigMode;
use vllm_chat::ToolStrictLevel;
use vllm_chat::multimodal::MmLimitPerPrompt;
use vllm_engine_core_client::TransportMode;
use vllm_managed_engine::ManagedEngineConfig;
@@ -134,6 +135,10 @@ pub struct RenderArgs {
/// `none` to disable parsing.
#[arg(long, default_value_t)]
reasoning_parser: ParserSelection,
/// Server-side floor for structural-tag based tool calling: `auto`,
/// `function`, or `parameter`.
#[arg(long, default_value_t)]
tool_strict_level: ToolStrictLevel,
/// Select the native chat renderer implementation.
#[arg(long = "tokenizer-mode", default_value_t)]
renderer: RendererSelection,
@@ -171,6 +176,7 @@ impl RenderArgs {
port: self.port,
tool_call_parser: self.tool_call_parser,
reasoning_parser: self.reasoning_parser,
tool_strict_level: self.tool_strict_level,
renderer: self.renderer,
chat_template: self.chat_template,
default_chat_template_kwargs: self.default_chat_template_kwargs.unwrap_or_default(),
@@ -241,6 +247,14 @@ pub struct SharedRuntimeArgs {
#[arg(long, default_value_t)]
#[serde(default = "default_py_bootstrap_parser_selection")]
pub reasoning_parser: ParserSelection,
/// Server-side floor for structural-tag based tool calling, applied on
/// top of the per-tool `strict` field: `auto` follows the request's
/// tool choice and per-tool strictness,
/// `function` constrains the tool-call envelope for every request with
/// tools, `parameter` additionally pins argument schemas.
#[arg(long, default_value_t)]
#[serde(default)]
pub tool_strict_level: ToolStrictLevel,
/// Select the chat renderer implementation.
#[arg(long = "tokenizer-mode", default_value_t)]
#[serde(default, rename = "tokenizer_mode")]
@@ -517,6 +531,7 @@ impl SharedRuntimeArgs {
listener_mode: HttpListenerMode::InheritedFd { fd: listen_fd },
tool_call_parser: self.tool_call_parser,
reasoning_parser: self.reasoning_parser,
tool_strict_level: self.tool_strict_level,
renderer: self.renderer,
language_model_only: self.language_model_only,
chat_template: self.chat_template,
@@ -574,6 +589,7 @@ impl SharedRuntimeArgs {
listener_mode,
tool_call_parser: self.tool_call_parser,
reasoning_parser: self.reasoning_parser,
tool_strict_level: self.tool_strict_level,
renderer: self.renderer,
language_model_only: self.language_model_only,
chat_template: self.chat_template,
rust/src/cmd/src/cli/tests.rs
+6
View File
@@ -184,6 +184,7 @@ fn serve_args_forward_python_flags_with_separator() {
engine_ready_timeout_secs: 600,
tool_call_parser: Auto,
reasoning_parser: Auto,
tool_strict_level: Auto,
renderer: Auto,
language_model_only: false,
max_logprobs: None,
@@ -1023,6 +1024,7 @@ fn frontend_args_accept_json() {
engine_ready_timeout_secs: 600,
tool_call_parser: None,
reasoning_parser: None,
tool_strict_level: Auto,
renderer: Auto,
language_model_only: false,
max_logprobs: None,
@@ -1693,6 +1695,7 @@ fn serve_args_accept_handshake_aliases() {
engine_ready_timeout_secs: 600,
tool_call_parser: Auto,
reasoning_parser: Auto,
tool_strict_level: Auto,
renderer: Auto,
language_model_only: false,
max_logprobs: None,
@@ -1850,6 +1853,7 @@ fn serve_frontend_config_uses_dp_address_as_advertised_host() {
},
tool_call_parser: Auto,
reasoning_parser: Auto,
tool_strict_level: Auto,
renderer: Auto,
language_model_only: false,
chat_template: None,
@@ -1942,6 +1946,7 @@ fn serve_frontend_config_keeps_tcp_transport_for_non_local_only_topology() {
},
tool_call_parser: Auto,
reasoning_parser: Auto,
tool_strict_level: Auto,
renderer: Auto,
language_model_only: false,
chat_template: None,
@@ -2055,6 +2060,7 @@ fn frontend_config_uses_external_coordinator_when_coordinator_address_is_present
},
tool_call_parser: None,
reasoning_parser: None,
tool_strict_level: Auto,
renderer: Auto,
language_model_only: false,
chat_template: None,
rust/src/server/examples/external_engine_openai_qwen.rs
+2
View File
@@ -16,6 +16,7 @@ use clap::Parser;
use futures::StreamExt as _;
use tokio_util::sync::CancellationToken;
use tracing_subscriber::EnvFilter;
use vllm_chat::ToolStrictLevel;
use vllm_engine_core_client::TransportMode;
use vllm_server::{
ApiServerOptions, ChatTemplateContentFormatOption, Config, CoordinatorMode, CorsConfig,
@@ -70,6 +71,7 @@ async fn main() -> Result<()> {
},
tool_call_parser: ParserSelection::Auto,
reasoning_parser: ParserSelection::Auto,
tool_strict_level: ToolStrictLevel::Auto,
renderer: RendererSelection::Auto,
language_model_only: false,
chat_template: None,
rust/src/server/src/config.rs
+3
View File
@@ -14,6 +14,7 @@ use serde_json::Value;
use vllm_chat::multimodal::MmLimitPerPrompt;
use vllm_chat::{
ChatTemplateContentFormatOption, GenerationConfigMode, ParserSelection, RendererSelection,
ToolStrictLevel,
};
use vllm_engine_core_client::{CoordinatorMode as EngineCoreCoordinatorMode, TransportMode};
use vllm_text::backend::hf::HfOverrides;
@@ -229,6 +230,8 @@ pub struct Config {
pub tool_call_parser: ParserSelection,
/// Reasoning parser selection.
pub reasoning_parser: ParserSelection,
/// Server-side floor for tool-call structural tags.
pub tool_strict_level: ToolStrictLevel,
/// Chat renderer selection.
pub renderer: RendererSelection,
/// Disable frontend-side multimodal preprocessing and render the model as
rust/src/server/src/grpc/tests.rs
+2
View File
@@ -244,6 +244,7 @@ impl ChatBackend for FakeTextBackend {
self.tokenizer(),
options.tool_call_parser,
options.reasoning_parser,
options.tool_strict_level,
)?))
}
}
@@ -292,6 +293,7 @@ impl ChatBackend for FakeMultimodalBackend {
self.tokenizer(),
options.tool_call_parser,
options.reasoning_parser,
options.tool_strict_level,
)?))
}
}
rust/src/server/src/lib.rs
+2 -1
View File
@@ -139,7 +139,8 @@ async fn build_state(config: &Config) -> Result<Arc<AppState>> {
let chat = ChatLlm::new(text, chat_backend)
.with_tool_call_parser(config.tool_call_parser.clone())
.with_reasoning_parser(config.reasoning_parser.clone());
.with_reasoning_parser(config.reasoning_parser.clone())
.with_tool_strict_level(config.tool_strict_level);
let state = Arc::new(
AppState::new(served_model_names, chat)
rust/src/server/src/render.rs
+9 -5
View File
@@ -10,7 +10,7 @@ use tokio_util::sync::CancellationToken;
use tracing::info;
use vllm_chat::{
ChatRequestProcessor, ChatTemplateContentFormatOption, LoadModelBackendsOptions,
ParserSelection, RendererSelection, load_model_backends,
ParserSelection, RendererSelection, ToolStrictLevel, load_model_backends,
};
use vllm_text::TextRequestProcessor;
use vllm_text::backend::hf::HfOverrides;
@@ -32,6 +32,7 @@ pub struct RenderConfig {
pub port: u16,
pub tool_call_parser: ParserSelection,
pub reasoning_parser: ParserSelection,
pub tool_strict_level: ToolStrictLevel,
pub renderer: RendererSelection,
pub chat_template: Option<String>,
pub default_chat_template_kwargs: HashMap<String, Value>,
@@ -88,10 +89,12 @@ async fn build_state(config: &RenderConfig) -> Result<Arc<RenderState>> {
let max_model_len = config.max_model_len.unwrap_or(u32::MAX);
let text = TextRequestProcessor::new(loaded.text_backend, max_model_len)
.with_max_logprobs(config.max_logprobs);
let chat = ChatRequestProcessor::render_only(loaded.chat_backend).with_parser_selections(
config.tool_call_parser.clone(),
config.reasoning_parser.clone(),
);
let chat = ChatRequestProcessor::render_only(loaded.chat_backend)
.with_parser_selections(
config.tool_call_parser.clone(),
config.reasoning_parser.clone(),
)
.with_tool_strict_level(config.tool_strict_level);
Ok(Arc::new(RenderState {
model: config.model.clone(),
served_model_names,
@@ -154,6 +157,7 @@ mod tests {
shutdown.cancel();
let error = serve_render(
RenderConfig {
tool_strict_level: ToolStrictLevel::Auto,
model: "test-model".to_string(),
revision: None,
hf_overrides: Default::default(),
rust/src/server/src/routes/http_client_tests.rs
+1
View File
@@ -168,6 +168,7 @@ impl ChatBackend for FakeChatBackend {
self.tokenizer(),
options.tool_call_parser,
options.reasoning_parser,
options.tool_strict_level,
)?))
}
}
rust/src/server/src/routes/tests.rs
+1
View File
@@ -517,6 +517,7 @@ impl ChatBackend for FakeChatBackend {
self.tokenizer(),
options.tool_call_parser,
options.reasoning_parser,
options.tool_strict_level,
)?))
}
}
tests/tool_parsers/test_structural_tag_registry.py
+250 -4
View File
@@ -1,6 +1,7 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import json
from types import SimpleNamespace
from unittest.mock import MagicMock
@@ -33,9 +34,11 @@ from vllm.tool_parsers.structural_tag_registry import (
SUPPORTED_STRUCTURAL_TAG_MODELS,
VLLM_BUILTIN_STRUCTURAL_TAG_MODELS,
XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS,
ToolChoice,
get_function_parameters,
get_model_structural_tag,
)
from vllm.tool_parsers.tool_strict_level import ToolStrictLevel
@pytest.fixture
@@ -231,11 +234,11 @@ def test_get_model_structural_tag_supports_all_xgrammar_builtins(
def test_get_model_structural_tag_supports_vllm_hermes(
sample_tools: list[ChatCompletionToolsParam],
sample_tools_strict: list[ChatCompletionToolsParam],
):
tag = get_model_structural_tag(
model="hermes",
tools=sample_tools,
tools=sample_tools_strict,
tool_choice="required",
reasoning=False,
)
@@ -273,6 +276,22 @@ def test_get_model_structural_tag_supports_vllm_hermes(
assert content["json_schema"] == expected_schema
def test_hermes_required_without_strict_leaves_arguments_free(
sample_tools: list[ChatCompletionToolsParam],
):
tag = get_model_structural_tag(
model="hermes",
tools=sample_tools,
tool_choice="required",
reasoning=False,
)
assert isinstance(tag, StructuralTag)
tags = tag.model_dump()["format"]["tags"]
assert tags
assert all(tag_dump["content"]["json_schema"] is True for tag_dump in tags)
def test_hermes_required_tool_calls_use_empty_separator():
tools = [
ChatCompletionToolsParam(
@@ -361,10 +380,19 @@ def _k3_tools(*calls: str) -> str:
return f"{_K3_TOOLS_OPEN}{''.join(calls)}{_K3_TOOLS_CLOSE}"
def _k3_grammar(tool_choice, tools=None):
def _as_strict(tools: list[ChatCompletionToolsParam]) -> list[ChatCompletionToolsParam]:
for tool in tools:
tool.function.strict = True
return tools
def _k3_grammar(tool_choice, tools=None, *, strict: bool = True):
"""Build the K3 grammar; tools are strict by default so the typed
argument channel is exercised (an unset ``strict`` leaves arguments free)."""
tools = tools if tools is not None else _k3_tools_by_name()
tag = get_model_structural_tag(
model="kimi_k3",
tools=tools if tools is not None else _k3_tools_by_name(),
tools=_as_strict(tools) if strict else tools,
tool_choice=tool_choice,
reasoning=False,
)
@@ -467,6 +495,18 @@ def test_kimi_k3_required_rejects_invalid(body: str):
assert not _is_grammar_accept_string(_k3_grammar("required"), body)
def test_kimi_k3_required_without_strict_leaves_arguments_free():
# An unset ``strict`` pins the call envelope only: a value the schema
# would reject is accepted, and the declared-required argument may be
# omitted.
grammar = _k3_grammar("required", strict=False)
body = _k3_response() + _k3_tools(
_k3_call("get_weather", _k3_arg("days", "number", "abc"))
)
assert _is_grammar_accept_string(grammar, body)
assert not _is_grammar_accept_string(_k3_grammar("required"), body)
def test_kimi_k3_schema_without_required_accepts_empty_call():
tools = [
ChatCompletionToolsParam(
@@ -829,3 +869,209 @@ def test_kimi_k3_forced_tool_choice_builds_single_mandatory_call():
response_only = _k3_response("no call here")
assert _is_grammar_accept_string(grammar, ok)
assert not _is_grammar_accept_string(grammar, response_only)
def _pins_argument_schema(tag: StructuralTag) -> bool:
return '"json_schema": {' in json.dumps(tag.model_dump(), ensure_ascii=False)
def _dumped(tag: StructuralTag) -> str:
return json.dumps(tag.model_dump(), ensure_ascii=False)
def test_tool_strict_level_auto_is_the_default(
sample_tools: list[ChatCompletionToolsParam],
):
"""Auto + no strict tool gets no tag unless the operator raises the floor."""
assert (
get_model_structural_tag(
model="deepseek_v4",
tools=sample_tools,
tool_choice="auto",
reasoning=False,
)
is None
)
@pytest.mark.parametrize("model", sorted(SUPPORTED_STRUCTURAL_TAG_MODELS))
def test_tool_strict_level_function_lifts_auto_gate(
model: str,
sample_tools: list[ChatCompletionToolsParam],
):
tag = get_model_structural_tag(
model=model,
tools=sample_tools,
tool_choice="auto",
reasoning=False,
strict_level=ToolStrictLevel.FUNCTION,
)
assert tag is not None
@pytest.mark.parametrize("model", sorted(XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS))
def test_tool_strict_level_function_pins_envelope_only(
model: str,
sample_tools: list[ChatCompletionToolsParam],
):
"""The request's own tools stay untouched; unset ``strict`` stays free."""
tag = get_model_structural_tag(
model=model,
tools=sample_tools,
tool_choice="auto",
reasoning=False,
strict_level=ToolStrictLevel.FUNCTION,
)
assert tag is not None
assert not _pins_argument_schema(tag)
assert all(tool.function.strict is None for tool in sample_tools)
@pytest.mark.parametrize("model", sorted(XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS))
def test_tool_strict_level_parameter_pins_argument_schemas(
model: str,
sample_tools: list[ChatCompletionToolsParam],
):
tag = get_model_structural_tag(
model=model,
tools=sample_tools,
tool_choice="auto",
reasoning=False,
strict_level=ToolStrictLevel.PARAMETER,
)
assert tag is not None
assert _pins_argument_schema(tag)
def test_tool_strict_level_function_keeps_client_strict_tools_strict(
sample_tools_strict: list[ChatCompletionToolsParam],
):
tag = get_model_structural_tag(
model="deepseek_v4",
tools=sample_tools_strict,
tool_choice="auto",
reasoning=False,
strict_level=ToolStrictLevel.FUNCTION,
)
assert tag is not None
assert _pins_argument_schema(tag)
@pytest.mark.parametrize("model", sorted(XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS))
def test_absent_strict_is_not_pinned_next_to_a_strict_tool(model: str):
"""A strict tool must not drag a neighbour with unset ``strict`` into
schema enforcement."""
weather = ChatCompletionToolsParam(
type="function",
function={
"name": "get_weather",
"strict": True,
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
)
search = ChatCompletionToolsParam(
type="function",
function={
"name": "search",
"parameters": {
"type": "object",
"properties": {"query": {"type": "string"}},
"required": ["query"],
},
},
)
tag = get_model_structural_tag(
model=model,
tools=[weather, search],
tool_choice="auto",
reasoning=False,
)
assert tag is not None
dumped = _dumped(tag)
assert '"city"' in dumped
assert '"query"' not in dumped
@pytest.mark.parametrize(
"tool_choice",
[
"required",
ChatCompletionNamedToolChoiceParam(
function=ChatCompletionNamedFunction(name="get_weather")
),
],
)
@pytest.mark.parametrize("model", sorted(XGRAMMAR_BUILTIN_STRUCTURAL_TAG_MODELS))
def test_forced_tool_choice_pins_schema_only_for_strict_tools(
model: str,
tool_choice: ToolChoice,
sample_tools: list[ChatCompletionToolsParam],
sample_tools_strict: list[ChatCompletionToolsParam],
):
"""Required / named always constrain the call; the argument schema
follows the per-tool ``strict`` (or the parameter level)."""
free = get_model_structural_tag(
model=model,
tools=sample_tools,
tool_choice=tool_choice,
reasoning=False,
strict_level=ToolStrictLevel.FUNCTION,
)
pinned = get_model_structural_tag(
model=model,
tools=sample_tools_strict,
tool_choice=tool_choice,
reasoning=False,
)
floored = get_model_structural_tag(
model=model,
tools=sample_tools,
tool_choice=tool_choice,
reasoning=False,
strict_level=ToolStrictLevel.PARAMETER,
)
assert free is not None and not _pins_argument_schema(free)
assert pinned is not None and _pins_argument_schema(pinned)
assert floored is not None and _pins_argument_schema(floored)
def test_tool_strict_level_parameter_overrides_client_strict_false():
tools = [
ChatCompletionToolsParam(
type="function",
function={
"name": "get_weather",
"parameters": {"type": "object", "properties": {}},
"strict": False,
},
)
]
tag = get_model_structural_tag(
model="deepseek_v4",
tools=tools,
tool_choice="required",
reasoning=False,
strict_level=ToolStrictLevel.PARAMETER,
)
assert tag is not None
assert _pins_argument_schema(tag)
def test_tool_strict_level_from_name():
assert ToolStrictLevel.from_name("AUTO") is ToolStrictLevel.AUTO
assert ToolStrictLevel.from_name("Parameter") is ToolStrictLevel.PARAMETER
with pytest.raises(ValueError, match="expected one of auto, function, parameter"):
ToolStrictLevel.from_name("strict")
with pytest.raises(ValueError, match="expected one of auto, function, parameter"):
ToolStrictLevel.from_name("off")
vllm/entrypoints/anthropic/serving.py
+2
View File
@@ -112,6 +112,7 @@ class AnthropicServingMessages(OpenAIServingChat):
reasoning_parser: str = "",
enable_auto_tools: bool = False,
tool_parser: str | None = None,
tool_strict_level: str = "auto",
enable_prompt_tokens_details: bool = False,
enable_force_include_usage: bool = False,
default_chat_template_kwargs: dict[str, Any] | None = None,
@@ -127,6 +128,7 @@ class AnthropicServingMessages(OpenAIServingChat):
return_tokens_as_token_ids=return_tokens_as_token_ids,
reasoning_parser=reasoning_parser,
enable_auto_tools=enable_auto_tools,
tool_strict_level=tool_strict_level,
tool_parser=tool_parser,
enable_prompt_tokens_details=enable_prompt_tokens_details,
enable_force_include_usage=enable_force_include_usage,
vllm/entrypoints/cohere/serving.py
+2
View File
@@ -249,6 +249,7 @@ class CohereServingChatV2(OpenAIServingChat):
reasoning_parser: str = "",
enable_auto_tools: bool = False,
tool_parser: str | None = None,
tool_strict_level: str = "auto",
enable_prompt_tokens_details: bool = False,
enable_force_include_usage: bool = False,
default_chat_template_kwargs: dict[str, Any] | None = None,
@@ -265,6 +266,7 @@ class CohereServingChatV2(OpenAIServingChat):
return_tokens_as_token_ids=return_tokens_as_token_ids,
reasoning_parser=reasoning_parser,
enable_auto_tools=enable_auto_tools,
tool_strict_level=tool_strict_level,
tool_parser=tool_parser,
enable_prompt_tokens_details=enable_prompt_tokens_details,
enable_force_include_usage=enable_force_include_usage,
vllm/entrypoints/generate/api_router.py
+4
View File
@@ -112,6 +112,7 @@ async def init_generate_state(
return_tokens_as_token_ids=args.return_tokens_as_token_ids,
enable_auto_tools=args.enable_auto_tool_choice,
tool_parser=args.tool_call_parser,
tool_strict_level=args.tool_strict_level,
tool_server=state.tool_server,
reasoning_parser=args.structured_outputs_config.reasoning_parser,
enable_prompt_tokens_details=args.enable_prompt_tokens_details,
@@ -137,6 +138,7 @@ async def init_generate_state(
enable_auto_tools=args.enable_auto_tool_choice,
exclude_tools_when_tool_choice_none=args.exclude_tools_when_tool_choice_none,
tool_parser=args.tool_call_parser,
tool_strict_level=args.tool_strict_level,
reasoning_parser=args.structured_outputs_config.reasoning_parser,
enable_prompt_tokens_details=args.enable_prompt_tokens_details,
enable_force_include_usage=args.enable_force_include_usage,
@@ -178,6 +180,7 @@ async def init_generate_state(
return_tokens_as_token_ids=args.return_tokens_as_token_ids,
enable_auto_tools=args.enable_auto_tool_choice,
tool_parser=args.tool_call_parser,
tool_strict_level=args.tool_strict_level,
reasoning_parser=args.structured_outputs_config.reasoning_parser,
enable_prompt_tokens_details=args.enable_prompt_tokens_details,
enable_force_include_usage=args.enable_force_include_usage,
@@ -198,6 +201,7 @@ async def init_generate_state(
return_tokens_as_token_ids=args.return_tokens_as_token_ids,
enable_auto_tools=args.enable_auto_tool_choice,
tool_parser=args.tool_call_parser,
tool_strict_level=args.tool_strict_level,
reasoning_parser=args.structured_outputs_config.reasoning_parser,
enable_prompt_tokens_details=args.enable_prompt_tokens_details,
enable_force_include_usage=args.enable_force_include_usage,
vllm/entrypoints/launchers/api_server/app_state.py
+2
View File
@@ -95,6 +95,7 @@ async def init_app_state(
enable_auto_tools=args.enable_auto_tool_choice,
exclude_tools_when_tool_choice_none=args.exclude_tools_when_tool_choice_none,
tool_parser=args.tool_call_parser,
tool_strict_level=args.tool_strict_level,
reasoning_parser=args.structured_outputs_config.reasoning_parser,
default_chat_template_kwargs=default_chat_template_kwargs,
log_error_stack=args.log_error_stack,
@@ -111,6 +112,7 @@ async def init_app_state(
enable_auto_tools=args.enable_auto_tool_choice,
exclude_tools_when_tool_choice_none=args.exclude_tools_when_tool_choice_none,
tool_parser=args.tool_call_parser,
tool_strict_level=args.tool_strict_level,
reasoning_parser=args.structured_outputs_config.reasoning_parser,
default_chat_template_kwargs=default_chat_template_kwargs,
log_error_stack=args.log_error_stack,
vllm/entrypoints/launchers/cli_args.py
+8
View File
@@ -20,6 +20,7 @@ from vllm.entrypoints.chat_utils import (
)
from vllm.entrypoints.openai.models.protocol import LoRAModulePath
from vllm.tool_parsers import ToolParserManager
from vllm.tool_parsers.tool_strict_level import ToolStrictLevelName
from vllm.utils.argparse_utils import FlexibleArgumentParser
from .utils.constants import (
@@ -116,6 +117,13 @@ class BaseFrontendArgs:
"""Special the tool parser plugin write to parse the model-generated tool
into OpenAI API format, the name register in this plugin can be used in
`--tool-call-parser`."""
tool_strict_level: ToolStrictLevelName = "auto"
"""Server-side floor for structural-tag based tool calling, applied on top
of the per-tool `strict` field. `auto` follows the request's tool choice
and per-tool strictness; `function` constrains the
tool-call envelope (markup and function name) for every request with
tools; `parameter` additionally pins argument schemas, as if every tool
were `strict: true`."""
tool_server: str | None = None
"""Comma-separated list of host:port pairs (IPv4, IPv6, or hostname).
Examples: 127.0.0.1:8000, [::1]:8000, localhost:1234. Or `demo` for
vllm/entrypoints/launchers/render/app_state.py
+2
View File
@@ -60,6 +60,7 @@ async def init_render_app_state(
enable_auto_tools=args.enable_auto_tool_choice,
exclude_tools_when_tool_choice_none=args.exclude_tools_when_tool_choice_none,
tool_parser=args.tool_call_parser,
tool_strict_level=args.tool_strict_level,
reasoning_parser=args.reasoning_parser,
default_chat_template_kwargs=default_chat_template_kwargs,
log_error_stack=args.log_error_stack,
@@ -76,6 +77,7 @@ async def init_render_app_state(
enable_auto_tools=args.enable_auto_tool_choice,
exclude_tools_when_tool_choice_none=args.exclude_tools_when_tool_choice_none,
tool_parser=args.tool_call_parser,
tool_strict_level=args.tool_strict_level,
reasoning_parser=args.reasoning_parser,
default_chat_template_kwargs=default_chat_template_kwargs,
log_error_stack=args.log_error_stack,
vllm/entrypoints/openai/chat_completion/serving.py
+2
View File
@@ -132,6 +132,7 @@ class OpenAIServingChat(GenerateBaseServing):
enable_auto_tools: bool = False,
exclude_tools_when_tool_choice_none: bool = False,
tool_parser: str | None = None,
tool_strict_level: str = "auto",
enable_prompt_tokens_details: bool = False,
enable_force_include_usage: bool = False,
enable_log_outputs: bool = False,
@@ -161,6 +162,7 @@ class OpenAIServingChat(GenerateBaseServing):
tool_parser_name=tool_parser,
reasoning_parser_name=reasoning_parser,
enable_auto_tools=enable_auto_tools,
tool_strict_level=tool_strict_level,
model_name=self.model_config.model,
is_harmony=self.model_config.hf_config.model_type == "gpt_oss",
)
vllm/entrypoints/openai/responses/serving.py
+2
View File
@@ -109,6 +109,7 @@ class OpenAIServingResponses(GenerateBaseServing):
reasoning_parser: str = "",
enable_auto_tools: bool = False,
tool_parser: str | None = None,
tool_strict_level: str = "auto",
tool_server: ToolServer | None = None,
enable_prompt_tokens_details: bool = False,
enable_force_include_usage: bool = False,
@@ -135,6 +136,7 @@ class OpenAIServingResponses(GenerateBaseServing):
tool_parser_name=tool_parser,
reasoning_parser_name=reasoning_parser,
enable_auto_tools=enable_auto_tools,
tool_strict_level=tool_strict_level,
model_name=self.model_config.model,
is_harmony=self.model_config.hf_config.model_type == "gpt_oss",
)
vllm/parser/abstract_parser.py
+4
View File
@@ -36,6 +36,7 @@ from vllm.tool_parsers.streaming import (
extract_named_tool_call_streaming,
extract_required_tool_call_streaming,
)
from vllm.tool_parsers.tool_strict_level import ToolStrictLevel
logger = init_logger(__name__)
@@ -95,6 +96,8 @@ class Parser:
# Subclasses should override these if they use specific parser classes
reasoning_parser_cls: type[ReasoningParser] | None = None
tool_parser_cls: type[ToolParser] | None = None
# Server-side floor for tool-call structural tags (--tool-strict-level).
tool_strict_level: ToolStrictLevel = ToolStrictLevel.AUTO
def __init__(
self,
@@ -415,6 +418,7 @@ class DelegatingParser(Parser):
structure_tag = self._tool_parser.get_structural_tag(
request,
reasoning=False,
strict_level=self.tool_strict_level,
)
if structure_tag is None:
return request
vllm/parser/parser_manager.py
+10
View File
@@ -6,6 +6,7 @@ from __future__ import annotations
from typing import TYPE_CHECKING
from vllm.logger import init_logger
from vllm.tool_parsers.tool_strict_level import ToolStrictLevel
if TYPE_CHECKING:
from vllm.parser.abstract_parser import Parser
@@ -77,6 +78,7 @@ class ParserManager:
enable_auto_tools: bool = False,
model_name: str | None = None,
is_harmony: bool = False,
tool_strict_level: str = "auto",
) -> type[Parser] | None:
"""Get a Parser that handles both reasoning and tool parsing.
@@ -89,6 +91,8 @@ class ParserManager:
model_name: The model name for parser-specific warnings.
is_harmony: Whether the selected model uses the Harmony format.
If True, HarmonyParser is always returned.
tool_strict_level: Server-side floor for tool-call structural
tags (``--tool-strict-level``).
Returns:
A Parser class, or None if neither parser is specified.
@@ -105,11 +109,14 @@ class ParserManager:
if reasoning_parser_cls is None and tool_parser_cls is None:
return None
strict_level = ToolStrictLevel.from_name(tool_strict_level)
if is_harmony:
from vllm.parser.harmony import HarmonyParser
HarmonyParser.reasoning_parser_cls = reasoning_parser_cls
HarmonyParser.tool_parser_cls = tool_parser_cls
HarmonyParser.tool_strict_level = strict_level
return HarmonyParser
if reasoning_parser_name == "kimi_k3" or tool_parser_name == "kimi_k3":
@@ -121,6 +128,7 @@ class ParserManager:
class _KimiK3Parser(KimiK3Parser):
reasoning_parser_cls = r_cls
tool_parser_cls = t_cls
tool_strict_level = strict_level
return _KimiK3Parser
@@ -136,6 +144,7 @@ class ParserManager:
class _CohereCommandParser(CohereCommandParser):
reasoning_parser_cls = r_cls
tool_parser_cls = t_cls
tool_strict_level = strict_level
return _CohereCommandParser
@@ -147,5 +156,6 @@ class ParserManager:
class _Parser(DelegatingParser):
reasoning_parser_cls = r_cls
tool_parser_cls = t_cls
tool_strict_level = strict_level
return _Parser
vllm/renderers/online_derenderer.py
+2
View File
@@ -54,6 +54,7 @@ class OnlineDerenderer:
exclude_tools_when_tool_choice_none: bool = False,
tool_parser: str | None = None,
reasoning_parser: str | None = None,
tool_strict_level: str = "auto",
default_chat_template_kwargs: dict[str, Any] | None = None,
log_error_stack: bool = False,
) -> None:
@@ -68,6 +69,7 @@ class OnlineDerenderer:
tool_parser_name=tool_parser,
reasoning_parser_name=reasoning_parser,
enable_auto_tools=enable_auto_tools,
tool_strict_level=tool_strict_level,
model_name=model_config.model,
is_harmony=self.use_harmony,
)
vllm/renderers/online_renderer.py
+2
View File
@@ -127,6 +127,7 @@ class OnlineRenderer:
exclude_tools_when_tool_choice_none: bool = False,
tool_parser: str | None = None,
reasoning_parser: str | None = None,
tool_strict_level: str = "auto",
default_chat_template_kwargs: dict[str, Any] | None = None,
log_error_stack: bool = False,
) -> None:
@@ -141,6 +142,7 @@ class OnlineRenderer:
tool_parser_name=tool_parser,
reasoning_parser_name=reasoning_parser,
enable_auto_tools=enable_auto_tools,
tool_strict_level=tool_strict_level,
model_name=model_config.model,
is_harmony=self.use_harmony,
)
vllm/tool_parsers/abstract_tool_parser.py
+3
View File
@@ -30,6 +30,7 @@ from vllm.sampling_params import (
StructuredOutputsParams,
)
from vllm.tokenizers import TokenizerLike
from vllm.tool_parsers.tool_strict_level import ToolStrictLevel
from vllm.tool_parsers.utils import Tool, get_json_schema_from_tools
from vllm.utils.collection_utils import is_list_of
from vllm.utils.import_utils import import_plugin
@@ -168,6 +169,7 @@ class ToolParser:
request: ChatCompletionRequest | ResponsesRequest,
*,
reasoning: bool = False,
strict_level: ToolStrictLevel = ToolStrictLevel.AUTO,
):
if self.structural_tag_model is None:
return None
@@ -180,6 +182,7 @@ class ToolParser:
tools=request.tools,
tool_choice=request.tool_choice,
reasoning=reasoning,
strict_level=strict_level,
)
def extract_tool_calls(
vllm/tool_parsers/hy_v4_tool_parser.py
+4
View File
@@ -25,6 +25,7 @@ from vllm.entrypoints.openai.responses.protocol import ResponsesRequest
from vllm.logger import init_logger
from vllm.tokenizers import TokenizerLike
from vllm.tool_parsers.abstract_tool_parser import ToolParser
from vllm.tool_parsers.tool_strict_level import ToolStrictLevel
from vllm.tool_parsers.utils import partial_tag_overlap
if TYPE_CHECKING:
@@ -994,6 +995,7 @@ class HYV4ToolParser(ToolParser):
request: ChatCompletionRequest | ResponsesRequest,
*,
reasoning: bool = False,
strict_level: ToolStrictLevel = ToolStrictLevel.AUTO,
) -> StructuralTag | None:
"""Build a structural tag matching HYV4's tool tokens.
@@ -1009,6 +1011,7 @@ class HYV4ToolParser(ToolParser):
Args:
request: The request being adjusted.
reasoning: Whether the grammar also covers the reasoning phase.
strict_level: Server-side floor from ``--tool-strict-level``.
Returns:
The structural tag, or None when structural tagging does not apply.
@@ -1041,6 +1044,7 @@ class HYV4ToolParser(ToolParser):
tool_choice=request.tool_choice,
reasoning=reasoning,
token_suffix=self._extractor.token_suffix,
strict_level=strict_level,
)
except Exception:
logger.warning(
vllm/tool_parsers/structural_tag_registry.py
+52 -7
View File
@@ -35,6 +35,7 @@ from vllm.entrypoints.openai.chat_completion.protocol import (
ChatCompletionNamedToolChoiceParam,
ChatCompletionToolsParam,
)
from vllm.tool_parsers.tool_strict_level import ToolStrictLevel
ToolChoice: TypeAlias = (
Literal["none", "auto", "required"]
@@ -94,15 +95,57 @@ def register_vllm_structural_tag(
return decorator
def _tool_is_strict(tool: ChatCompletionToolsParam | ResponsesTool) -> bool:
if isinstance(tool, FunctionTool):
return tool.strict is True
if isinstance(tool, ChatCompletionToolsParam):
return tool.function.strict is True
return False
def _any_tool_strict(
tools: Sequence[ChatCompletionToolsParam | ResponsesTool],
) -> bool:
for tool in tools:
if isinstance(tool, FunctionTool) and tool.strict is True:
return True
if isinstance(tool, ChatCompletionToolsParam) and tool.function.strict is True:
return True
return False
return any(_tool_is_strict(tool) for tool in tools)
def _with_tool_strict(
tool: ChatCompletionToolsParam | ResponsesTool,
strict: bool,
) -> ChatCompletionToolsParam | ResponsesTool:
"""Return a copy of ``tool`` with ``strict`` set, leaving the request's alone."""
if isinstance(tool, FunctionTool):
return tool.model_copy(update={"strict": strict})
if isinstance(tool, ChatCompletionToolsParam):
return tool.model_copy(
update={"function": tool.function.model_copy(update={"strict": strict})}
)
return tool
def _resolve_tool_strictness(
tools: Sequence[ChatCompletionToolsParam | ResponsesTool],
tool_choice: ToolChoice,
strict_level: ToolStrictLevel,
) -> Sequence[ChatCompletionToolsParam | ResponsesTool] | None:
"""Decide whether a structural tag applies and pin each tool's ``strict``.
A tool without an explicit ``strict`` is treated as non-strict: its call
envelope is still constrained, but its arguments stay free unless the
server level is PARAMETER. ``None`` means no structural tag.
"""
if (
tool_choice == "auto"
and strict_level == ToolStrictLevel.AUTO
and not _any_tool_strict(tools)
):
return None
return [
_with_tool_strict(
tool, strict_level >= ToolStrictLevel.PARAMETER or _tool_is_strict(tool)
)
for tool in tools
]
def get_model_structural_tag(
@@ -111,12 +154,14 @@ def get_model_structural_tag(
tool_choice: ToolChoice,
reasoning: bool,
token_suffix: str = "",
strict_level: ToolStrictLevel = ToolStrictLevel.AUTO,
) -> StructuralTag | None:
"""Build a structural tag with xgrammar's builtin model templates."""
if not tools or tool_choice == "none":
return None
if tool_choice == "auto" and not _any_tool_strict(tools):
tools = _resolve_tool_strictness(tools, tool_choice, strict_level)
if tools is None:
return None
dumped_tools = [_dump_tool_for_xgrammar(tool) for tool in tools]
vllm/tool_parsers/tool_strict_level.py
+32
View File
@@ -0,0 +1,32 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import enum
from typing import Literal
ToolStrictLevelName = Literal["auto", "function", "parameter"]
class ToolStrictLevel(enum.IntEnum):
"""Server-side floor for tool-call structural tags (``--tool-strict-level``).
AUTO: derive call constraints from ``tool_choice`` and argument-schema
enforcement from each tool's explicit ``strict`` field.
FUNCTION: constrain the tool-call envelope for every request with tools.
PARAMETER: additionally pin argument schemas, as if every tool were
``strict: true``.
"""
AUTO = 0
FUNCTION = 1
PARAMETER = 2
@classmethod
def from_name(cls, name: str) -> "ToolStrictLevel":
try:
return cls[name.strip().upper()]
except KeyError:
expected = ", ".join(level.name.lower() for level in cls)
raise ValueError(
f"Unknown tool strict level {name!r}; expected one of {expected}."
) from None