11 Commits
Author SHA1 Message Date
wang.yuqi c1692c85f1 [Misc] mv chat_utils.py to renderers. (#60522)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-10-10 06:50:34 +00:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
Flora FengandWentao Ye 8bcc916a98 [Bugfix] Fix DeepSeek V4/3.2 tokenizer vocab size overcount crashing guided decoding (#51727)
Signed-off-by: sfeng33 <4florafeng@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-08-10 23:00:08 +00:00
Harry Mellor dd94484577 Bump Transformers version to 5.10.4 (#41359)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-07-07 05:13:28 -07:00
Matthew Bonanni 9efc4db965 [Bugfix] Fix DeepSeek-V3.2 tokenizer stripping spaces (#37004)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
2026-03-13 22:55:36 +00:00
Cyrus Leung 7196348157 [Bugfix] Fix Qwen-VL tokenizer implementation (#36140)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-03-05 08:07:19 -08:00
Nick Hill a96197f564 [Perf] Simplify DeepseekV32 tokenizer, ensure fast detokenization used (#33855)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-02-08 07:16:34 +00:00
a32cb49b60 feat(frontend): early-fail tokenization guard for user requests (#31366)
Signed-off-by: limingliang <limingliang@stepfun.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: limingliang <limingliang@stepfun.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-02-05 19:38:02 -08:00
Cyrus Leung 6444824873 [Misc] Implement TokenizerLike.convert_tokens_to_ids (#31796)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
2026-01-06 12:08:22 +00:00
高鑫崧andChauncey b7b6a60aca Adapt the old parameter enable_thinking in chat_template_kwargs (#30852)
Signed-off-by: xinsong.gao <1418762819@qq.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2025-12-17 07:10:59 -08:00
Jee Jee Li a524d1ba0a [Bugfix] Fix deepseek_v32 tokenizer_mode (#30658)
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
2025-12-15 04:20:31 +00:00