3 people c7e9816c6a [Bugfix][Rust Frontend] Skip undefined token ids in decode and anchor them zero-width (#55240) ...

Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Feathbow <feathbow@gmail.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>

2026-09-09 09:55:04 +00:00

..

hf

[CI] Add SPDX license header to Rust/Protobuf sources (#48472)

2026-07-13 10:22:47 +01:00

incremental

[Rust Frontend] Use token-attributed text in reasoning and unified parsers (#54884)

2026-09-04 02:05:05 +08:00

byte_level_decode.rs

[CI] Add SPDX license header to Rust/Protobuf sources (#48472)

2026-07-13 10:22:47 +01:00

error.rs

[CI] Add SPDX license header to Rust/Protobuf sources (#48472)

2026-07-13 10:22:47 +01:00

hf.rs

[Bugfix][Rust Frontend] Skip undefined token ids in decode and anchor them zero-width (#55240)

2026-09-09 09:55:04 +00:00

incremental.rs

[Bugfix][Rust Frontend] Skip undefined token ids in decode and anchor them zero-width (#55240)

2026-09-09 09:55:04 +00:00

lib.rs

[Rust Frontend] Attribute decoded text to tokens (#52910)

2026-09-01 22:25:31 +00:00

tekken.rs

[Rust Frontend] Add ordinary-text tokenizer encoding (#49992)

2026-07-28 15:46:51 +08:00

test_utils.rs

[Rust Frontend] Add HY3 unified parser and local XGrammar structural-tag builder (#53054)

2026-08-20 19:13:54 -07:00

tiktoken.rs

[Rust Frontend] Attribute decoded text to tokens (#52910)

2026-09-01 22:25:31 +00:00