[Frontend] Rework fastokens integration (#43168)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
This commit is contained in:
@@ -270,31 +270,34 @@ Known supported models (with corresponding benchmarks):
|
||||
|
||||
## Input Processing
|
||||
|
||||
### fastokens Tokenizer Mode
|
||||
### fastokens Backend
|
||||
|
||||
By default vLLM uses the standard Hugging Face `tokenizers` library to power
|
||||
the fast tokenizer (`--tokenizer-mode hf`). For BPE tokenizers (Qwen, Llama,
|
||||
DeepSeek, GPT-OSS, etc.) you can switch to the
|
||||
[fastokens](https://github.com/crusoecloud/fastokens) Rust backend, a drop-in
|
||||
replacement that's substantially faster on encode/decode and on streaming
|
||||
detokenization:
|
||||
the fast tokenizer. For BPE tokenizers (Qwen, Llama, DeepSeek, GPT-OSS, etc.)
|
||||
you can switch to the [fastokens](https://github.com/crusoecloud/fastokens)
|
||||
Rust backend, a drop-in replacement that's substantially faster on
|
||||
encode/decode and on streaming detokenization. Enable it by setting
|
||||
`VLLM_USE_FASTOKENS=1`:
|
||||
|
||||
```console
|
||||
vllm serve Qwen/Qwen3-8B --tokenizer-mode fastokens
|
||||
VLLM_USE_FASTOKENS=1 vllm serve Qwen/Qwen3-8B
|
||||
```
|
||||
|
||||
Equivalent in the offline API:
|
||||
|
||||
```python
|
||||
import os
|
||||
os.environ["VLLM_USE_FASTOKENS"] = "1"
|
||||
|
||||
from vllm import LLM
|
||||
llm = LLM(model="Qwen/Qwen3-8B", tokenizer_mode="fastokens")
|
||||
llm = LLM(model="Qwen/Qwen3-8B")
|
||||
```
|
||||
|
||||
The `fastokens` Python package must be installed; if it isn't, vLLM raises
|
||||
a clear `ImportError` at tokenizer load. `fastokens` loads a Hugging Face
|
||||
fast tokenizer with its inner Rust tokenizer replaced by the fastokens shim,
|
||||
so it is mutually exclusive with non-HF modes such as `mistral` or
|
||||
`deepseek_v32`.
|
||||
The `fastokens` Python package (>= 0.2.0) must be installed; if it isn't,
|
||||
vLLM raises a clear `ImportError` at tokenizer load. The override applies to
|
||||
any `--tokenizer-mode` that ends up loading an HF fast tokenizer (`hf`,
|
||||
`deepseek_v32`, `deepseek_v4`, `qwen_vl`, …). Modes that don't use the HF
|
||||
fast tokenizer (`mistral`, `grok2`, `kimi_audio`) ignore the flag.
|
||||
|
||||
Tokenizer-bound workloads — long shared prefixes, bursty short prompts,
|
||||
batch detokenization — see the largest wins. If your bottleneck is GPU
|
||||
|
||||
Reference in New Issue
Block a user