[Bugfix][Structured Output][Spec Decode] Constrain bitmask and trim grammar advance at the reasoning boundary (#44297)
Signed-off-by: Allen.Yu <yuyue0225sc@163.com> Signed-off-by: yue.yu <yuyue0225sc@163.com> Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com>
This commit is contained in:
co-authored by
Benjamin Chislett
parent
26eb87204d
commit
e7c9df9449
@@ -23,6 +23,12 @@ class StructuredOutputRequest:
|
||||
params: StructuredOutputsParams
|
||||
_grammar: Future[StructuredOutputGrammar] | StructuredOutputGrammar | None = None
|
||||
reasoning_ended: bool | None = None
|
||||
# Absolute index into the request's all_token_ids of the last reasoning
|
||||
# token (the reasoning-end marker). Tokens at or before this index are
|
||||
# reasoning content and must never be fed to the grammar. Only set when
|
||||
# reasoning ends in a step whose tokens the scheduler advances immediately
|
||||
# (structural tags + speculative decoding, see #42452).
|
||||
reasoning_end_token_index: int | None = None
|
||||
reasoning_parser_kwargs: dict[str, Any] | None = None
|
||||
# Cached per request; do not share reasoning parsers across requests because
|
||||
# their behavior can depend on reasoning_parser_kwargs.
|
||||
|
||||
Reference in New Issue
Block a user