[Frontend] Rework fastokens integration (#43168)

Signed-off-by: Nick Hill <nickhill123@gmail.com>
This commit is contained in:
Nick Hill
2026-05-21 15:36:58 -07:00
committed by GitHub
parent 0b59fc45dd
commit 0f66623b0d
7 changed files with 63 additions and 55 deletions
+16 -13
View File
@@ -270,31 +270,34 @@ Known supported models (with corresponding benchmarks):
## Input Processing
### fastokens Tokenizer Mode
### fastokens Backend
By default vLLM uses the standard Hugging Face `tokenizers` library to power
the fast tokenizer (`--tokenizer-mode hf`). For BPE tokenizers (Qwen, Llama,
DeepSeek, GPT-OSS, etc.) you can switch to the
[fastokens](https://github.com/crusoecloud/fastokens) Rust backend, a drop-in
replacement that's substantially faster on encode/decode and on streaming
detokenization:
the fast tokenizer. For BPE tokenizers (Qwen, Llama, DeepSeek, GPT-OSS, etc.)
you can switch to the [fastokens](https://github.com/crusoecloud/fastokens)
Rust backend, a drop-in replacement that's substantially faster on
encode/decode and on streaming detokenization. Enable it by setting
`VLLM_USE_FASTOKENS=1`:
```console
vllm serve Qwen/Qwen3-8B --tokenizer-mode fastokens
VLLM_USE_FASTOKENS=1 vllm serve Qwen/Qwen3-8B
```
Equivalent in the offline API:
```python
import os
os.environ["VLLM_USE_FASTOKENS"] = "1"
from vllm import LLM
llm = LLM(model="Qwen/Qwen3-8B", tokenizer_mode="fastokens")
llm = LLM(model="Qwen/Qwen3-8B")
```
The `fastokens` Python package must be installed; if it isn't, vLLM raises
a clear `ImportError` at tokenizer load. `fastokens` loads a Hugging Face
fast tokenizer with its inner Rust tokenizer replaced by the fastokens shim,
so it is mutually exclusive with non-HF modes such as `mistral` or
`deepseek_v32`.
The `fastokens` Python package (>= 0.2.0) must be installed; if it isn't,
vLLM raises a clear `ImportError` at tokenizer load. The override applies to
any `--tokenizer-mode` that ends up loading an HF fast tokenizer (`hf`,
`deepseek_v32`, `deepseek_v4`, `qwen_vl`, …). Modes that don't use the HF
fast tokenizer (`mistral`, `grok2`, `kimi_audio`) ignore the flag.
Tokenizer-bound workloads — long shared prefixes, bursty short prompts,
batch detokenization — see the largest wins. If your bottleneck is GPU