[Bugfix][Parser] Fix special tokens (EOS/BOS) leaking into reasoning content (#48748)

Signed-off-by: Ben Browning <56071+bbrowning@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
Ben Browning
2026-07-22 16:10:55 -04:00
committed by GitHub
co-authored by mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent b44311b6ef
commit 3de4b2bf3c
3 changed files with 72 additions and 15 deletions
@@ -304,14 +304,7 @@ class StreamingParserEngine:
transition = self.config.transitions.get(key)
if transition is None:
if (
self._has_drops
and terminal == DROP_TERMINAL
# Preserve drop tokens when skip_tool_parsing is active so
# the reasoning pass doesn't silently remove tokens that a
# later tool-call pass might need to see.
and not self.skip_tool_parsing
):
if self._has_drops and terminal == DROP_TERMINAL:
return []
return self._emit_for_state(value)