[Bugfix][Rust Frontend] Fix missing added tokens in hf/fastokens tokenizer (#44683)

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
This commit is contained in:
Isotr0py
2026-06-10 03:52:41 -07:00
committed by GitHub
co-authored by Bugen Zhao
parent a1ec011a83
commit 9ad08c4d15
4 changed files with 215 additions and 8 deletions
+1 -1
View File
@@ -225,7 +225,7 @@ impl MultimodalModelInfo {
///
/// The HF renderer uses this token while flattening image content in string
/// content format.
pub(crate) fn placeholder_token(&self) -> &str {
pub fn placeholder_token(&self) -> &str {
&self.spec.placeholder_token
}
}