From ee0fd6984ad6579204a388cd80021fea081c8869 Mon Sep 17 00:00:00 2001 From: Angelo Ruocco Date: Wed, 17 Jun 2026 13:58:00 +0200 Subject: [PATCH] docs, kv_offloading: add docs for selective offload (#45279) Signed-off-by: Angelo Ruocco --- docs/features/kv_offloading_usage.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/docs/features/kv_offloading_usage.md b/docs/features/kv_offloading_usage.md index 39b826bfd5..93da2ed036 100644 --- a/docs/features/kv_offloading_usage.md +++ b/docs/features/kv_offloading_usage.md @@ -127,6 +127,29 @@ PYTHONHASHSEED=0 vllm serve ... - FS thread counts: tune `n_read_threads` and `n_write_threads` to the parallelism your storage can sustain. Reads are latency-sensitive on the prefill path, so prefer more read threads when prefill hit rates are high. - Sharing `root_dir` across runs: runs with the same model, `block_size`, parallelism layout, and dtype share files under the same `` subdirectory. Changing any of these produces a new subdirectory; old ones are orphaned but harmless. Delete them to reclaim disk. +## Per-Request Selective Offload + +Individual requests can cap how many of their tokens are eligible for offload by setting `max_offload_tokens` in the request's `kv_transfer_params`. Only the first `max_offload_tokens` tokens of the request are offloaded; blocks beyond that point are skipped on the store path. This is useful when a known prefix (e.g., a system prompt or shared context) is worth caching but later request-specific tokens are not. + +| Key | Type | Notes | +| --- | --- | --- | +| `max_offload_tokens` | non-negative `int` | Upper bound on tokens to offload for this request. `0` disables offload for the request entirely; omit the key (or set to `None`) for no cap. Non-`int`, negative, or `bool` values are rejected with a warning and treated as no cap. | + +!!! note + `max_offload_tokens` is experimental and subject to change. + +Example (OpenAI-compatible completions request): + +```json +{ + "model": "", + "prompt": "...", + "kv_transfer_params": { + "max_offload_tokens": 1024 + } +} +``` + ## Further Reading - [vLLM blog: KV Offloading Connector](https://vllm.ai/blog/2026-01-08-kv-offloading-connector) — motivation, architecture (DMA-based async transfer), and benchmarks (TTFT and throughput).