[Feature][Pooling] Add offline fixed-length late chunking (#60164)

Signed-off-by: Linze-Shi <linzeshi0@gmail.com>
This commit is contained in:
Linze Shi
2026-10-13 11:16:21 +00:00
committed by GitHub
parent 426f3bd36b
commit e6e84f350c
15 changed files with 1147 additions and 7 deletions
+34
View File
@@ -90,6 +90,7 @@ The following [pooling parameters][vllm.PoolingParams] are supported.
```python
--8<-- "vllm/pooling_params.py:common-pooling-params"
--8<-- "vllm/pooling_params.py:embed-pooling-params"
--8<-- "vllm/pooling_params.py:token-embed-pooling-params"
```
### `LLM.encode`
@@ -108,6 +109,39 @@ data = output.outputs.data
print(f"Data: {data!r}")
```
### Fixed-length late chunking
Late chunking encodes the full input before averaging contextual token representations within each chunk. The chunk means pass through the model's embedding head, including its projection and optional normalization.
This mode is currently available through the offline `LLM.encode` API for plain-text input. With an instance configured for `token_embed`, enable it per request:
```python
from vllm import PoolingParams
from vllm.pooling_params import LateChunkingParams
outputs = llm.encode(
[document],
pooling_task="token_embed",
pooling_params=PoolingParams(
late_chunking_params=LateChunkingParams(chunk_size=256)
),
)
vectors = outputs[0].outputs.data # [num_chunks, embedding_dimension]
chunks = outputs[0].late_chunking.chunks
```
For `N` model-input tokens and chunk size `C`, the output contains `ceil(N / C)` vectors. The last chunk uses its actual length. Special tokens and instruction prefixes count toward both chunk boundaries and means. Omitting `late_chunking_params` preserves ordinary token embeddings.
Each vector row corresponds to one entry in `chunks`. Its `token_range` and `char_range` are zero-based, half-open ranges. Character ranges index the submitted Python string; use `document[start:end]` to retrieve the source text. A chunk containing only special tokens has `char_range=None`.
Use nonempty text within the model's context window and a fast tokenizer that provides aligned source offsets. Input truncation, explicit padding and renderer text normalization are not supported. The custom document/query outputs of `JinaForRanking` are also unsupported. Late chunking does not extend the model's context window or reduce its attention computation.
The [offline example](../../../examples/pooling/token_embed/late_chunking_offline.py) loads an E5 model and prints the vector shape, token ranges and source excerpts for two documents:
```bash
python examples/pooling/token_embed/late_chunking_offline.py --chunk-size 16
```
### `LLM.score`
The [score][vllm.entrypoints.pooling.offline.PoolingOfflineMixin.score] method outputs similarity scores between sentence pairs.