tokenizer: Add fastokens support (#41741)

Signed-off-by: AlonKejzman <alonkeizman@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
This commit is contained in:
AlonKejzman
2026-05-07 22:49:42 +08:00
committed by GitHub
co-authored by Roger Wang
parent 003159d98b
commit 2a16ece2d3
6 changed files with 93 additions and 5 deletions
+30
View File
@@ -270,6 +270,36 @@ Known supported models (with corresponding benchmarks):
## Input Processing
### fastokens Tokenizer Mode
By default vLLM uses the standard Hugging Face `tokenizers` library to power
the fast tokenizer (`--tokenizer-mode hf`). For BPE tokenizers (Qwen, Llama,
DeepSeek, GPT-OSS, etc.) you can switch to the
[fastokens](https://github.com/crusoecloud/fastokens) Rust backend, a drop-in
replacement that's substantially faster on encode/decode and on streaming
detokenization:
```console
vllm serve Qwen/Qwen3-8B --tokenizer-mode fastokens
```
Equivalent in the offline API:
```python
from vllm import LLM
llm = LLM(model="Qwen/Qwen3-8B", tokenizer_mode="fastokens")
```
The `fastokens` Python package must be installed; if it isn't, vLLM raises
a clear `ImportError` at tokenizer load. `fastokens` loads a Hugging Face
fast tokenizer with its inner Rust tokenizer replaced by the fastokens shim,
so it is mutually exclusive with non-HF modes such as `mistral` or
`deepseek_v32`.
Tokenizer-bound workloads — long shared prefixes, bursty short prompts,
batch detokenization — see the largest wins. If your bottleneck is GPU
prefill/decode, the tokenizer change is unlikely to be visible end-to-end.
### Parallel Processing
You can run input processing in parallel via [API server scale-out](../serving/data_parallel_deployment.md#internal-load-balancing).