tokenizer: Add fastokens support (#41741)
Signed-off-by: AlonKejzman <alonkeizman@gmail.com> Co-authored-by: Roger Wang <hey@rogerw.io>
This commit is contained in:
co-authored by
Roger Wang
parent
003159d98b
commit
2a16ece2d3
@@ -270,6 +270,36 @@ Known supported models (with corresponding benchmarks):
|
||||
|
||||
## Input Processing
|
||||
|
||||
### fastokens Tokenizer Mode
|
||||
|
||||
By default vLLM uses the standard Hugging Face `tokenizers` library to power
|
||||
the fast tokenizer (`--tokenizer-mode hf`). For BPE tokenizers (Qwen, Llama,
|
||||
DeepSeek, GPT-OSS, etc.) you can switch to the
|
||||
[fastokens](https://github.com/crusoecloud/fastokens) Rust backend, a drop-in
|
||||
replacement that's substantially faster on encode/decode and on streaming
|
||||
detokenization:
|
||||
|
||||
```console
|
||||
vllm serve Qwen/Qwen3-8B --tokenizer-mode fastokens
|
||||
```
|
||||
|
||||
Equivalent in the offline API:
|
||||
|
||||
```python
|
||||
from vllm import LLM
|
||||
llm = LLM(model="Qwen/Qwen3-8B", tokenizer_mode="fastokens")
|
||||
```
|
||||
|
||||
The `fastokens` Python package must be installed; if it isn't, vLLM raises
|
||||
a clear `ImportError` at tokenizer load. `fastokens` loads a Hugging Face
|
||||
fast tokenizer with its inner Rust tokenizer replaced by the fastokens shim,
|
||||
so it is mutually exclusive with non-HF modes such as `mistral` or
|
||||
`deepseek_v32`.
|
||||
|
||||
Tokenizer-bound workloads — long shared prefixes, bursty short prompts,
|
||||
batch detokenization — see the largest wins. If your bottleneck is GPU
|
||||
prefill/decode, the tokenizer change is unlikely to be visible end-to-end.
|
||||
|
||||
### Parallel Processing
|
||||
|
||||
You can run input processing in parallel via [API server scale-out](../serving/data_parallel_deployment.md#internal-load-balancing).
|
||||
|
||||
@@ -21,7 +21,7 @@ Let's say we want to serve the popular Qwen model by running `vllm serve Qwen/Qw
|
||||
|
||||
Beyond that, there are two more things vLLM depends on Hugging Face for.
|
||||
|
||||
1. **Tokenizer**: vLLM uses the tokenizer from Hugging Face to tokenize the input text. The tokenizer is loaded using [AutoTokenizer.from_pretrained](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoTokenizer.from_pretrained) with the `model` argument as the model name and the `--revision` argument as the revision. It is also possible to use a tokenizer from another model by specifying the `--tokenizer` argument in the `vllm serve` command. Other relevant arguments are `--tokenizer-revision` and `--tokenizer-mode`. Please check Hugging Face's documentation for the meaning of these arguments. This part of the logic can be found in the [get_tokenizer](https://github.com/vllm-project/vllm/blob/127c07480ecea15e4c2990820c457807ff78a057/vllm/transformers_utils/tokenizer.py#L87) function. After obtaining the tokenizer, notably, vLLM will cache some expensive attributes of the tokenizer in [vllm.tokenizers.hf.get_cached_tokenizer][].
|
||||
1. **Tokenizer**: vLLM uses the tokenizer from Hugging Face to tokenize the input text. The tokenizer is loaded using [AutoTokenizer.from_pretrained](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoTokenizer.from_pretrained) with the `model` argument as the model name and the `--revision` argument as the revision. It is also possible to use a tokenizer from another model by specifying the `--tokenizer` argument in the `vllm serve` command. Other relevant arguments are `--tokenizer-revision` and `--tokenizer-mode`. Setting `--tokenizer-mode fastokens` swaps in a drop-in Rust BPE backend for the HF fast tokenizer (see [fastokens Tokenizer Mode](../configuration/optimization.md#fastokens-tokenizer-mode)). Please check Hugging Face's documentation for the meaning of these arguments. This part of the logic can be found in the [get_tokenizer](https://github.com/vllm-project/vllm/blob/127c07480ecea15e4c2990820c457807ff78a057/vllm/transformers_utils/tokenizer.py#L87) function. After obtaining the tokenizer, notably, vLLM will cache some expensive attributes of the tokenizer in [vllm.tokenizers.hf.get_cached_tokenizer][].
|
||||
|
||||
2. **Model weight**: vLLM downloads the model weight from the Hugging Face model hub using the `model` argument as the model name and the `--revision` argument as the revision. vLLM provides the argument `--load-format` to control what files to download from the model hub. By default, it will try to load the weights in the safetensors format and fall back to the PyTorch bin format if the safetensors format is not available. We can also pass `--load-format dummy` to skip downloading the weights.
|
||||
- It is recommended to use the safetensors format, as it is efficient for loading in distributed inference and also safe from arbitrary code execution. See the [documentation](https://huggingface.co/docs/safetensors/en/index) for more information on the safetensors format. This part of the logic can be found [here](https://github.com/vllm-project/vllm/blob/10b67d865d92e376956345becafc249d4c3c0ab7/vllm/model_executor/model_loader/loader.py#L385). Please note that:
|
||||
|
||||
Reference in New Issue
Block a user