[BugFix][Frontend] Use LoRA tokenizer in OpenAI APIs (#6227)

Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
This commit is contained in:
Nick Hill
2024-07-18 15:13:30 +08:00
committed by GitHub
co-authored by Cyrus Leung
parent 8a74c68bd1
commit e2fbaee725
16 changed files with 267 additions and 186 deletions
+3
View File
@@ -88,6 +88,9 @@ def get_tokenizer(
"Cannot use the fast tokenizer in slow tokenizer mode.")
kwargs["use_fast"] = False
if "truncation_side" not in kwargs:
kwargs["truncation_side"] = "left"
try:
tokenizer = AutoTokenizer.from_pretrained(
tokenizer_name,