Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: Harry Mellor <hej.mellor@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
804e5377aa
commit
8c1557a79c
@@ -23,8 +23,7 @@ class ThreadSafeHFTokenizerMixin:
|
||||
|
||||
|
||||
def maybe_make_thread_pool(tokenizer: _T, copies: int = 1):
|
||||
"""
|
||||
If `tokenizer` is a `TokenizersBackend`, modify the tokenizer
|
||||
"""If `tokenizer` is a `TokenizersBackend`, modify the tokenizer
|
||||
in-place to make the public interface thread-safe by routing calls
|
||||
through a deep-copied tokenizer pool.
|
||||
|
||||
@@ -105,8 +104,7 @@ def maybe_make_thread_pool(tokenizer: _T, copies: int = 1):
|
||||
|
||||
|
||||
def get_cached_tokenizer(tokenizer: HfTokenizer) -> HfTokenizer:
|
||||
"""
|
||||
By default, transformers will recompute multiple tokenizer properties
|
||||
"""By default, transformers will recompute multiple tokenizer properties
|
||||
each time they are called, leading to a significant slowdown.
|
||||
This proxy caches these properties for faster access.
|
||||
"""
|
||||
|
||||
Reference in New Issue
Block a user