tokenizer: Add fastokens support (#41741)
Signed-off-by: AlonKejzman <alonkeizman@gmail.com> Co-authored-by: Roger Wang <hey@rogerw.io>
This commit is contained in:
co-authored by
Roger Wang
parent
003159d98b
commit
2a16ece2d3
@@ -270,6 +270,36 @@ Known supported models (with corresponding benchmarks):
|
||||
|
||||
## Input Processing
|
||||
|
||||
### fastokens Tokenizer Mode
|
||||
|
||||
By default vLLM uses the standard Hugging Face `tokenizers` library to power
|
||||
the fast tokenizer (`--tokenizer-mode hf`). For BPE tokenizers (Qwen, Llama,
|
||||
DeepSeek, GPT-OSS, etc.) you can switch to the
|
||||
[fastokens](https://github.com/crusoecloud/fastokens) Rust backend, a drop-in
|
||||
replacement that's substantially faster on encode/decode and on streaming
|
||||
detokenization:
|
||||
|
||||
```console
|
||||
vllm serve Qwen/Qwen3-8B --tokenizer-mode fastokens
|
||||
```
|
||||
|
||||
Equivalent in the offline API:
|
||||
|
||||
```python
|
||||
from vllm import LLM
|
||||
llm = LLM(model="Qwen/Qwen3-8B", tokenizer_mode="fastokens")
|
||||
```
|
||||
|
||||
The `fastokens` Python package must be installed; if it isn't, vLLM raises
|
||||
a clear `ImportError` at tokenizer load. `fastokens` loads a Hugging Face
|
||||
fast tokenizer with its inner Rust tokenizer replaced by the fastokens shim,
|
||||
so it is mutually exclusive with non-HF modes such as `mistral` or
|
||||
`deepseek_v32`.
|
||||
|
||||
Tokenizer-bound workloads — long shared prefixes, bursty short prompts,
|
||||
batch detokenization — see the largest wins. If your bottleneck is GPU
|
||||
prefill/decode, the tokenizer change is unlikely to be visible end-to-end.
|
||||
|
||||
### Parallel Processing
|
||||
|
||||
You can run input processing in parallel via [API server scale-out](../serving/data_parallel_deployment.md#internal-load-balancing).
|
||||
|
||||
Reference in New Issue
Block a user