[Bugfix]: Fix TokenizerLike interface (#30009)

Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
This commit is contained in:
Rohan Potdar
2025-12-05 20:56:40 -08:00
committed by GitHub
parent e858bc4d14
commit 40a046cd82
8 changed files with 78 additions and 52 deletions
+3
View File
@@ -22,6 +22,9 @@ class TokenizerLike(Protocol):
) -> "TokenizerLike":
raise NotImplementedError
def num_special_tokens_to_add(self) -> int:
raise NotImplementedError
@property
def all_special_tokens(self) -> list[str]:
raise NotImplementedError