[Frontend] Rework fastokens integration (#43168)

Signed-off-by: Nick Hill <nickhill123@gmail.com>
This commit is contained in:
Nick Hill
2026-05-21 15:36:58 -07:00
committed by GitHub
parent 0b59fc45dd
commit 0f66623b0d
7 changed files with 63 additions and 55 deletions
+16 -13
View File
@@ -270,31 +270,34 @@ Known supported models (with corresponding benchmarks):
## Input Processing
### fastokens Tokenizer Mode
### fastokens Backend
By default vLLM uses the standard Hugging Face `tokenizers` library to power
the fast tokenizer (`--tokenizer-mode hf`). For BPE tokenizers (Qwen, Llama,
DeepSeek, GPT-OSS, etc.) you can switch to the
[fastokens](https://github.com/crusoecloud/fastokens) Rust backend, a drop-in
replacement that's substantially faster on encode/decode and on streaming
detokenization:
the fast tokenizer. For BPE tokenizers (Qwen, Llama, DeepSeek, GPT-OSS, etc.)
you can switch to the [fastokens](https://github.com/crusoecloud/fastokens)
Rust backend, a drop-in replacement that's substantially faster on
encode/decode and on streaming detokenization. Enable it by setting
`VLLM_USE_FASTOKENS=1`:
```console
vllm serve Qwen/Qwen3-8B --tokenizer-mode fastokens
VLLM_USE_FASTOKENS=1 vllm serve Qwen/Qwen3-8B
```
Equivalent in the offline API:
```python
import os
os.environ["VLLM_USE_FASTOKENS"] = "1"
from vllm import LLM
llm = LLM(model="Qwen/Qwen3-8B", tokenizer_mode="fastokens")
llm = LLM(model="Qwen/Qwen3-8B")
```
The `fastokens` Python package must be installed; if it isn't, vLLM raises
a clear `ImportError` at tokenizer load. `fastokens` loads a Hugging Face
fast tokenizer with its inner Rust tokenizer replaced by the fastokens shim,
so it is mutually exclusive with non-HF modes such as `mistral` or
`deepseek_v32`.
The `fastokens` Python package (>= 0.2.0) must be installed; if it isn't,
vLLM raises a clear `ImportError` at tokenizer load. The override applies to
any `--tokenizer-mode` that ends up loading an HF fast tokenizer (`hf`,
`deepseek_v32`, `deepseek_v4`, `qwen_vl`, …). Modes that don't use the HF
fast tokenizer (`mistral`, `grok2`, `kimi_audio`) ignore the flag.
Tokenizer-bound workloads — long shared prefixes, bursty short prompts,
batch detokenization — see the largest wins. If your bottleneck is GPU
+1 -1
View File
@@ -21,7 +21,7 @@ Let's say we want to serve the popular Qwen model by running `vllm serve Qwen/Qw
Beyond that, there are two more things vLLM depends on Hugging Face for.
1. **Tokenizer**: vLLM uses the tokenizer from Hugging Face to tokenize the input text. The tokenizer is loaded using [AutoTokenizer.from_pretrained](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoTokenizer.from_pretrained) with the `model` argument as the model name and the `--revision` argument as the revision. It is also possible to use a tokenizer from another model by specifying the `--tokenizer` argument in the `vllm serve` command. Other relevant arguments are `--tokenizer-revision` and `--tokenizer-mode`. Setting `--tokenizer-mode fastokens` swaps in a drop-in Rust BPE backend for the HF fast tokenizer (see [fastokens Tokenizer Mode](../configuration/optimization.md#fastokens-tokenizer-mode)). Please check Hugging Face's documentation for the meaning of these arguments. This part of the logic can be found in the [get_tokenizer](https://github.com/vllm-project/vllm/blob/127c07480ecea15e4c2990820c457807ff78a057/vllm/transformers_utils/tokenizer.py#L87) function. After obtaining the tokenizer, notably, vLLM will cache some expensive attributes of the tokenizer in [vllm.tokenizers.hf.get_cached_tokenizer][].
1. **Tokenizer**: vLLM uses the tokenizer from Hugging Face to tokenize the input text. The tokenizer is loaded using [AutoTokenizer.from_pretrained](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoTokenizer.from_pretrained) with the `model` argument as the model name and the `--revision` argument as the revision. It is also possible to use a tokenizer from another model by specifying the `--tokenizer` argument in the `vllm serve` command. Other relevant arguments are `--tokenizer-revision` and `--tokenizer-mode`. Setting `VLLM_USE_FASTOKENS=1` swaps in a drop-in Rust BPE backend for any HF fast tokenizer loaded by vLLM (see [fastokens Backend](../configuration/optimization.md#fastokens-backend)). Please check Hugging Face's documentation for the meaning of these arguments. This part of the logic can be found in the [get_tokenizer](https://github.com/vllm-project/vllm/blob/127c07480ecea15e4c2990820c457807ff78a057/vllm/transformers_utils/tokenizer.py#L87) function. After obtaining the tokenizer, notably, vLLM will cache some expensive attributes of the tokenizer in [vllm.tokenizers.hf.get_cached_tokenizer][].
2. **Model weight**: vLLM downloads the model weight from the Hugging Face model hub using the `model` argument as the model name and the `--revision` argument as the revision. vLLM provides the argument `--load-format` to control what files to download from the model hub. By default, it will try to load the weights in the safetensors format and fall back to the PyTorch bin format if the safetensors format is not available. We can also pass `--load-format dummy` to skip downloading the weights.
- It is recommended to use the safetensors format, as it is efficient for loading in distributed inference and also safe from arbitrary code execution. See the [documentation](https://huggingface.co/docs/safetensors/en/index) for more information on the safetensors format. This part of the logic can be found [here](https://github.com/vllm-project/vllm/blob/10b67d865d92e376956345becafc249d4c3c0ab7/vllm/model_executor/model_loader/loader.py#L385). Please note that:
+8 -7
View File
@@ -93,9 +93,7 @@ def is_cumem_allocator_available() -> bool:
RunnerOption = Literal["auto", RunnerType]
ConvertType = Literal["none", "embed", "classify"]
ConvertOption = Literal["auto", ConvertType]
TokenizerMode = Literal[
"auto", "hf", "slow", "mistral", "deepseek_v32", "deepseek_v4", "fastokens"
]
TokenizerMode = Literal["auto", "hf", "slow", "mistral", "deepseek_v32", "deepseek_v4"]
ModelDType = Literal["auto", "half", "float16", "bfloat16", "float", "float32"]
LogprobsMode = Literal[
"raw_logits", "raw_logprobs", "processed_logits", "processed_logprobs"
@@ -148,10 +146,13 @@ class ModelConfig:
- "deepseek_v32" will always use the tokenizer from `deepseek_v32`.
- "deepseek_v4" will always use the tokenizer from `deepseek_v4`.
- "qwen_vl" will always use the tokenizer from `qwen_vl`.
- "fastokens" loads a Hugging Face fast tokenizer powered by the
[fastokens](https://github.com/crusoecloud/fastokens) Rust BPE backend
(requires the `fastokens` package to be installed).
- Other custom values can be supported via plugins."""
- Other custom values can be supported via plugins.
To swap the Rust BPE backend that powers HF fast tokenizers for the
[fastokens](https://github.com/crusoecloud/fastokens) implementation, set
`VLLM_USE_FASTOKENS=1` instead — that override applies to any mode that
loads an HF fast tokenizer (`hf`, `deepseek_v32`, `deepseek_v4`,
`qwen_vl`, …)."""
trust_remote_code: bool = False
"""Trust remote code (e.g., from HuggingFace) when downloading the model
and tokenizer."""
+7
View File
@@ -17,6 +17,7 @@ if TYPE_CHECKING:
VLLM_PORT: int | None = None
VLLM_RPC_BASE_PATH: str = tempfile.gettempdir()
VLLM_USE_MODELSCOPE: bool = False
VLLM_USE_FASTOKENS: bool = False
VLLM_RINGBUFFER_WARNING_INTERVAL: int = 60
VLLM_NCCL_SO_PATH: str | None = None
LD_LIBRARY_PATH: str | None = None
@@ -676,6 +677,12 @@ environment_variables: dict[str, Callable[[], Any]] = {
"VLLM_USE_MODELSCOPE": lambda: (
os.environ.get("VLLM_USE_MODELSCOPE", "False").lower() == "true"
),
# If true, replace the Rust BPE backend that powers HF fast tokenizers
# with the `fastokens` (https://github.com/crusoecloud/fastokens) shim.
# Applies to any tokenizer mode that loads an HF fast tokenizer
# (`hf`, `deepseek_v32`, `deepseek_v4`, `qwen_vl`, …). The `fastokens`
# Python package must be installed.
"VLLM_USE_FASTOKENS": lambda: bool(int(os.getenv("VLLM_USE_FASTOKENS", "0"))),
# Interval in seconds to log a warning message when the ring buffer is full
"VLLM_RINGBUFFER_WARNING_INTERVAL": lambda: int(
os.environ.get("VLLM_RINGBUFFER_WARNING_INTERVAL", "60")
-1
View File
@@ -22,7 +22,6 @@ logger = init_logger(__name__)
_VLLM_RENDERERS = {
"deepseek_v32": ("deepseek_v32", "DeepseekV32Renderer"),
"deepseek_v4": ("deepseek_v4", "DeepseekV4Renderer"),
"fastokens": ("hf", "HfRenderer"),
"grok2": ("grok2", "Grok2Renderer"),
"hf": ("hf", "HfRenderer"),
"kimi_audio": ("hf", "HfRenderer"),
+24 -32
View File
@@ -1,47 +1,39 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""``fastokens`` tokenizer mode.
"""``fastokens`` backend patch.
Loads a Hugging Face fast tokenizer whose internal Rust tokenizer is replaced
by the fastokens shim. fastokens also rebinds
``tokenizers.decoders.DecodeStream`` so the streaming detokenizer accepts the
shim. Both patches are installed for the lifetime of the process —
``patch_transformers()`` is idempotent.
When ``VLLM_USE_FASTOKENS=1`` is set, ``fastokens.patch_transformers()`` swaps
the inner Rust tokenizer of every HF fast tokenizer loaded afterwards with the
fastokens shim and rebinds ``tokenizers.decoders.DecodeStream`` so the
streaming detokenizer accepts the shim. The patch is process-global and
idempotent, so it applies to any tokenizer mode that ends up loading an HF
fast tokenizer (`hf`, `deepseek_v32`, `deepseek_v4`, `qwen_vl`, …).
"""
from pathlib import Path
from importlib.metadata import PackageNotFoundError, version
from .hf import CachedHfTokenizer, HfTokenizer
from .protocol import TokenizerLike
from packaging.version import Version
_MIN_FASTOKENS_VERSION = "0.2.0"
def _apply_fastokens_patch() -> None:
def apply_fastokens_patch() -> None:
try:
import fastokens
except ImportError as e:
raise ImportError(
"The 'fastokens' package is required for tokenizer_mode='fastokens'."
f"The 'fastokens' package (>= {_MIN_FASTOKENS_VERSION}) is required "
"when VLLM_USE_FASTOKENS=1."
) from e
fastokens.patch_transformers()
class FastokensTokenizer(TokenizerLike):
@classmethod
def from_pretrained(
cls,
path_or_repo_id: str | Path,
*args,
trust_remote_code: bool = False,
revision: str | None = None,
download_dir: str | None = None,
**kwargs,
) -> HfTokenizer:
_apply_fastokens_patch()
return CachedHfTokenizer.from_pretrained(
path_or_repo_id,
*args,
trust_remote_code=trust_remote_code,
revision=revision,
download_dir=download_dir,
**kwargs,
try:
installed = version("fastokens")
except PackageNotFoundError:
installed = None
if installed is None or Version(installed) < Version(_MIN_FASTOKENS_VERSION):
raise ImportError(
f"fastokens >= {_MIN_FASTOKENS_VERSION} is required when "
f"VLLM_USE_FASTOKENS=1 (found {installed or 'unknown'})."
)
fastokens.patch_transformers()
+7 -1
View File
@@ -43,7 +43,6 @@ _MODEL_TYPES_WITH_INCORRECT_TOKENIZER_CLASS: set[str] = {"step3_vl"}
_VLLM_TOKENIZERS = {
"deepseek_v32": ("deepseek_v32", "DeepseekV32Tokenizer"),
"deepseek_v4": ("deepseek_v4", "DeepseekV4Tokenizer"),
"fastokens": ("fastokens", "FastokensTokenizer"),
"grok2": ("grok2", "Grok2Tokenizer"),
"hf": ("hf", "CachedHfTokenizer"),
"kimi_audio": ("kimi_audio", "KimiAudioTokenizer"),
@@ -204,6 +203,13 @@ def get_tokenizer(
**kwargs,
) -> _T:
"""Gets a tokenizer for the given model name via HuggingFace or ModelScope."""
if envs.VLLM_USE_FASTOKENS:
# Process-global, idempotent patch that swaps the Rust BPE backend
# of any HF fast tokenizer loaded afterwards. No-op for non-HF modes.
from .fastokens import apply_fastokens_patch
apply_fastokens_patch()
tokenizer_mode, tokenizer_name, args, kwargs = cached_resolve_tokenizer_args(
tokenizer_name,
*args,