[Model] Support Unlimited OCR (#46564)
Signed-off-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Roger Wang <hey@rogerw.io>
This commit is contained in:
co-authored by
Isotr0py
Roger Wang
parent
a65f93fb2e
commit
c6741b2ad4
@@ -31,7 +31,11 @@ logger = init_logger(__name__)
|
||||
# temporary workaround and better long term solutions are:
|
||||
# - Add model type to MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS in transformers (better)
|
||||
# - Fix tokenizer_class on the hub for the affected models (best)
|
||||
_MODEL_TYPES_WITH_INCORRECT_TOKENIZER_CLASS: set[str] = {"step3_vl", "step3p7"}
|
||||
_MODEL_TYPES_WITH_INCORRECT_TOKENIZER_CLASS: set[str] = {
|
||||
"step3_vl",
|
||||
"step3p7",
|
||||
"unlimited-ocr",
|
||||
}
|
||||
|
||||
_VLLM_TOKENIZERS = {
|
||||
"deepseek_v32": ("deepseek_v32", "DeepseekV32Tokenizer"),
|
||||
|
||||
Reference in New Issue
Block a user