Antoni BaumandMeloYang bfdb1ba5c3 [Core] Improve detokenization performance for prefill (#3469) ...

Co-authored-by: MeloYang <meloyang05@gmail.com>

2024-03-22 13:44:12 -07:00

..

configs

[Misc] Bump up transformers to v4.39.0 & Remove StarCoder2Config (#3551)

2024-03-21 07:58:12 -07:00

tokenizer_group

[Core] Add generic typing to LRUCache (#3511)

2024-03-20 00:36:09 -07:00

tokenizers

Re-enable the 80 char line width limit (#3305)

2024-03-10 19:49:14 -07:00

__init__.py

[Tokenizer] Add an option to specify tokenizer (#284)

2023-06-28 09:46:58 -07:00

config.py

[Misc] Bump up transformers to v4.39.0 & Remove StarCoder2Config (#3551)

2024-03-21 07:58:12 -07:00

detokenizer.py

[Core] Improve detokenization performance for prefill (#3469)

2024-03-22 13:44:12 -07:00

tokenizer.py

[Core] Improve detokenization performance for prefill (#3469)

2024-03-22 13:44:12 -07:00