[Frontend] Remove the slow tokenizer mode (#58545)

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Harry Mellor
2026-09-24 08:48:28 -07:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 344fcc252a
commit d636456b1f
12 changed files with 11 additions and 39 deletions
+1 -1
View File
@@ -20,7 +20,7 @@ def fast_tokenizer():
"""gpt2 ships a Fast tokenizer; use it to test the offsets happy path."""
from transformers import AutoTokenizer
return AutoTokenizer.from_pretrained("openai-community/gpt2", use_fast=True)
return AutoTokenizer.from_pretrained("openai-community/gpt2")
def _make_base_renderer_with(tokenizer):