[Docs][PD][NIXL] Bidirectional kv-cache transfer (#43097)

Signed-off-by: NickLucche <nlucches@redhat.com>
This commit is contained in:
Nicolò Lucchesi
2026-05-20 09:02:36 +02:00
committed by GitHub
parent 7e4bc2cecb
commit 40651c0207
+164
View File
@@ -136,6 +136,170 @@ python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py \
- In bidirectional mode, the decoder caches KV blocks for multi-turn conversations. This TTL controls how long those blocks are held before being released. Unlike the prefiller lease, this TTL is not renewed via heartbeats.
- Example: `--kv-transfer-config '{"kv_connector_extra_config": {"decoder_kv_blocks_ttl": 600}}'`
## Bidirectional KV Transfer (Multi-turn)
In standard disaggregated prefilling, KV cache flows in one direction: Prefill (P) computes the KV cache and Decode (D) reads from P. For multi-turn conversations this is wasteful — D already holds the KV cache corresponding to the generated tokens from prior turns, yet P must recompute it from scratch on every new turn. Bidirectional KV transfer lets P **pull** existing KV blocks from D via RDMA before computing only the new tokens, significantly reducing Time-To-First-Token (TTFT) for long-prefill such as **multi-turn heavy scenarios**.
### How it works
The feature relies on a **stateful proxy** that sits between the client and the P/D instances. The proxy tracks `kv_transfer_params` returned by D at the end of each turn, and attaches them to the next turn's request so P knows which blocks to pull from D.
```mermaid
sequenceDiagram
participant Client
participant Proxy
participant P as Prefill (P)
participant D as Decode (D)
rect rgb(240, 240, 250)
note right of Client: Turn 1 — Cache Miss
Client->>Proxy: chat request + conversation_id
Proxy->>P: request (no remote blocks)
activate P
note over P: full prefill
P-->>Proxy: kv_transfer_params (P's blocks)
deactivate P
Proxy->>D: request + P's kv_transfer_params
activate D
D-->P: RDMA read (D pulls KV from P)
note over D: decode
D-->>Proxy: stream response + kv_transfer_params
deactivate D
note over Proxy: cache D's kv_transfer_params
Proxy-->>Client: response
end
rect rgb(255, 245, 235)
note right of Client: Turn 2+ — Cache Hit (Bidirectional)
Client->>Proxy: chat request + conversation_id
note over Proxy: lookup cached D blocks
Proxy->>P: request + D's remote_block_ids
activate P
P-->D: RDMA read (P pulls KV from D)
note over P: prefill new tokens only
P-->>Proxy: kv_transfer_params (P's blocks)
deactivate P
Proxy->>D: request + P's kv_transfer_params
activate D
D-->P: RDMA read (D pulls new KV from P)
note over D: decode
D-->>Proxy: stream response + kv_transfer_params
deactivate D
note over Proxy: update cached kv_transfer_params
Proxy-->>Client: response
end
```
**Turn 1 (cache miss):**
1. Client sends a chat request with a `conversation_id` to the proxy.
2. Proxy forwards the request to P with no remote block info — P computes the full KV cache.
3. Proxy forwards the request to D along with P's `kv_transfer_params` (block IDs, engine ID, host/port).
4. D reads KV blocks from P via RDMA (peer-to-peer pull), then generates the response.
5. D streams the response back through the proxy. The final chunk includes D's own `kv_transfer_params`.
6. Proxy caches D's `kv_transfer_params` keyed by `conversation_id`, then returns the response to the client.
**Turn 2+ (cache hit — bidirectional):**
1. Client sends the next turn with the same `conversation_id`.
2. Proxy looks up cached `kv_transfer_params` from the previous turn and attaches D's `remote_block_ids` to the request sent to P.
3. P reads the existing KV cache from D via RDMA (D→P pull), then computes KV only for the new tokens.
4. Proxy forwards the request to D with P's updated `kv_transfer_params`.
5. D reads the new KV blocks from P, generates the response, and returns updated `kv_transfer_params` which the proxy caches for the next turn.
### Configuration
Enable bidirectional KV transfer by setting `bidirectional_kv_xfer` in `kv_connector_extra_config` on **both** P and D instances:
```bash
vllm serve <MODEL> \
--kv-transfer-config '{
"kv_connector": "NixlConnector",
"kv_role": "kv_both",
"kv_connector_extra_config": {
"bidirectional_kv_xfer": true
}
}'
```
Additional configuration options in `kv_connector_extra_config`:
| Parameter | Default | Description |
| --------- | ------- | ----------- |
| `bidirectional_kv_xfer` | `false` | Enable bidirectional D→P KV transfer. |
| `kv_recompute_threshold` | `64` | Minimum number of remote tokens required to trigger a D→P pull. Below this threshold, P recomputes locally instead of pulling (to amortize transfer latency). |
| `decoder_kv_blocks_ttl` | `480` | TTL (seconds) for KV blocks cached on D for bidirectional reuse. Blocks are released after this duration. Not renewed via heartbeats. |
### Multi-turn proxy setup
Use the provided multi-turn proxy to manage `kv_transfer_params` caching across conversation turns:
```bash
python examples/disaggregated/disaggregated_serving/disagg_proxy_multiturn.py \
--host 0.0.0.0 --port 8000 \
--prefiller-host <P_IP> --prefiller-port 8100 \
--decoder-host <D_IP> --decoder-port 8200
```
The proxy supports multiple P and D instances via round-robin:
```bash
python examples/disaggregated/disaggregated_serving/disagg_proxy_multiturn.py \
--host 0.0.0.0 --port 8000 \
--prefiller-hosts <P_IP1> <P_IP2> --prefiller-ports 8100 8100 \
--decoder-hosts <D_IP1> <D_IP2> --decoder-ports 8200 8200
```
### Client usage
Include a `conversation_id` field in the request body to enable cross-turn KV reuse. Without it, the proxy cannot link turns and falls back to full recomputation.
```bash
# Turn 1
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-0.6B",
"conversation_id": "session-42",
"messages": [
{"role": "user", "content": "What is vLLM?"}
]
}'
# Turn 2 — same conversation_id triggers bidirectional KV pull
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-0.6B",
"conversation_id": "session-42",
"messages": [
{"role": "user", "content": "What is vLLM?"},
{"role": "assistant", "content": "vLLM is a high-throughput LLM serving engine..."},
{"role": "user", "content": "How does disaggregated prefilling work?"}
]
}'
```
!!! note
The `conversation_id` field is a non-standard extension to the OpenAI API. It is consumed by the proxy and not forwarded to the vLLM engine.
### Limitations
- Requires a stateful proxy (or equivalent router) to track and forward `kv_transfer_params` between turns.
- Currently supported on CUDA with device-buffer KV cache. Host-buffer support (e.g., for Intel XPU) is planned for future work.
!!! warning "Reasoning models with stripped thinking traces"
When using reasoning models (e.g. DeepSeek-R1) that produce thinking traces
(`<think>...</think>`), D's KV blocks cover the full token sequence including
thinking tokens. If the client strips thinking traces from the conversation
history before sending the next turn, the prompt P receives will be missing
tokens from the middle of what D generated. The block-alignment logic assumes
P's prompt is a prefix of D's sequence, so pulling KV blocks from D in this
case transfers cache computed for the wrong token positions, producing
incorrect results.
We currently assume the router is able to detect such mismatch across turns. See [#43094](https://github.com/vllm-project/vllm/issues/43094).
## Multi-Instance Setup
### Multiple Prefiller Instances on Different Machines