[Docs][PD][NIXL] Bidirectional kv-cache transfer (#43097)
Signed-off-by: NickLucche <nlucches@redhat.com>
This commit is contained in:
@@ -136,6 +136,170 @@ python tests/v1/kv_connector/nixl_integration/toy_proxy_server.py \
|
||||
- In bidirectional mode, the decoder caches KV blocks for multi-turn conversations. This TTL controls how long those blocks are held before being released. Unlike the prefiller lease, this TTL is not renewed via heartbeats.
|
||||
- Example: `--kv-transfer-config '{"kv_connector_extra_config": {"decoder_kv_blocks_ttl": 600}}'`
|
||||
|
||||
## Bidirectional KV Transfer (Multi-turn)
|
||||
|
||||
In standard disaggregated prefilling, KV cache flows in one direction: Prefill (P) computes the KV cache and Decode (D) reads from P. For multi-turn conversations this is wasteful — D already holds the KV cache corresponding to the generated tokens from prior turns, yet P must recompute it from scratch on every new turn. Bidirectional KV transfer lets P **pull** existing KV blocks from D via RDMA before computing only the new tokens, significantly reducing Time-To-First-Token (TTFT) for long-prefill such as **multi-turn heavy scenarios**.
|
||||
|
||||
### How it works
|
||||
|
||||
The feature relies on a **stateful proxy** that sits between the client and the P/D instances. The proxy tracks `kv_transfer_params` returned by D at the end of each turn, and attaches them to the next turn's request so P knows which blocks to pull from D.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant Proxy
|
||||
participant P as Prefill (P)
|
||||
participant D as Decode (D)
|
||||
|
||||
rect rgb(240, 240, 250)
|
||||
note right of Client: Turn 1 — Cache Miss
|
||||
Client->>Proxy: chat request + conversation_id
|
||||
Proxy->>P: request (no remote blocks)
|
||||
activate P
|
||||
note over P: full prefill
|
||||
P-->>Proxy: kv_transfer_params (P's blocks)
|
||||
deactivate P
|
||||
Proxy->>D: request + P's kv_transfer_params
|
||||
activate D
|
||||
D-->P: RDMA read (D pulls KV from P)
|
||||
note over D: decode
|
||||
D-->>Proxy: stream response + kv_transfer_params
|
||||
deactivate D
|
||||
note over Proxy: cache D's kv_transfer_params
|
||||
Proxy-->>Client: response
|
||||
end
|
||||
|
||||
rect rgb(255, 245, 235)
|
||||
note right of Client: Turn 2+ — Cache Hit (Bidirectional)
|
||||
Client->>Proxy: chat request + conversation_id
|
||||
note over Proxy: lookup cached D blocks
|
||||
Proxy->>P: request + D's remote_block_ids
|
||||
activate P
|
||||
P-->D: RDMA read (P pulls KV from D)
|
||||
note over P: prefill new tokens only
|
||||
P-->>Proxy: kv_transfer_params (P's blocks)
|
||||
deactivate P
|
||||
Proxy->>D: request + P's kv_transfer_params
|
||||
activate D
|
||||
D-->P: RDMA read (D pulls new KV from P)
|
||||
note over D: decode
|
||||
D-->>Proxy: stream response + kv_transfer_params
|
||||
deactivate D
|
||||
note over Proxy: update cached kv_transfer_params
|
||||
Proxy-->>Client: response
|
||||
end
|
||||
```
|
||||
|
||||
**Turn 1 (cache miss):**
|
||||
|
||||
1. Client sends a chat request with a `conversation_id` to the proxy.
|
||||
2. Proxy forwards the request to P with no remote block info — P computes the full KV cache.
|
||||
3. Proxy forwards the request to D along with P's `kv_transfer_params` (block IDs, engine ID, host/port).
|
||||
4. D reads KV blocks from P via RDMA (peer-to-peer pull), then generates the response.
|
||||
5. D streams the response back through the proxy. The final chunk includes D's own `kv_transfer_params`.
|
||||
6. Proxy caches D's `kv_transfer_params` keyed by `conversation_id`, then returns the response to the client.
|
||||
|
||||
**Turn 2+ (cache hit — bidirectional):**
|
||||
|
||||
1. Client sends the next turn with the same `conversation_id`.
|
||||
2. Proxy looks up cached `kv_transfer_params` from the previous turn and attaches D's `remote_block_ids` to the request sent to P.
|
||||
3. P reads the existing KV cache from D via RDMA (D→P pull), then computes KV only for the new tokens.
|
||||
4. Proxy forwards the request to D with P's updated `kv_transfer_params`.
|
||||
5. D reads the new KV blocks from P, generates the response, and returns updated `kv_transfer_params` which the proxy caches for the next turn.
|
||||
|
||||
### Configuration
|
||||
|
||||
Enable bidirectional KV transfer by setting `bidirectional_kv_xfer` in `kv_connector_extra_config` on **both** P and D instances:
|
||||
|
||||
```bash
|
||||
vllm serve <MODEL> \
|
||||
--kv-transfer-config '{
|
||||
"kv_connector": "NixlConnector",
|
||||
"kv_role": "kv_both",
|
||||
"kv_connector_extra_config": {
|
||||
"bidirectional_kv_xfer": true
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
Additional configuration options in `kv_connector_extra_config`:
|
||||
|
||||
| Parameter | Default | Description |
|
||||
| --------- | ------- | ----------- |
|
||||
| `bidirectional_kv_xfer` | `false` | Enable bidirectional D→P KV transfer. |
|
||||
| `kv_recompute_threshold` | `64` | Minimum number of remote tokens required to trigger a D→P pull. Below this threshold, P recomputes locally instead of pulling (to amortize transfer latency). |
|
||||
| `decoder_kv_blocks_ttl` | `480` | TTL (seconds) for KV blocks cached on D for bidirectional reuse. Blocks are released after this duration. Not renewed via heartbeats. |
|
||||
|
||||
### Multi-turn proxy setup
|
||||
|
||||
Use the provided multi-turn proxy to manage `kv_transfer_params` caching across conversation turns:
|
||||
|
||||
```bash
|
||||
python examples/disaggregated/disaggregated_serving/disagg_proxy_multiturn.py \
|
||||
--host 0.0.0.0 --port 8000 \
|
||||
--prefiller-host <P_IP> --prefiller-port 8100 \
|
||||
--decoder-host <D_IP> --decoder-port 8200
|
||||
```
|
||||
|
||||
The proxy supports multiple P and D instances via round-robin:
|
||||
|
||||
```bash
|
||||
python examples/disaggregated/disaggregated_serving/disagg_proxy_multiturn.py \
|
||||
--host 0.0.0.0 --port 8000 \
|
||||
--prefiller-hosts <P_IP1> <P_IP2> --prefiller-ports 8100 8100 \
|
||||
--decoder-hosts <D_IP1> <D_IP2> --decoder-ports 8200 8200
|
||||
```
|
||||
|
||||
### Client usage
|
||||
|
||||
Include a `conversation_id` field in the request body to enable cross-turn KV reuse. Without it, the proxy cannot link turns and falls back to full recomputation.
|
||||
|
||||
```bash
|
||||
# Turn 1
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"conversation_id": "session-42",
|
||||
"messages": [
|
||||
{"role": "user", "content": "What is vLLM?"}
|
||||
]
|
||||
}'
|
||||
|
||||
# Turn 2 — same conversation_id triggers bidirectional KV pull
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"conversation_id": "session-42",
|
||||
"messages": [
|
||||
{"role": "user", "content": "What is vLLM?"},
|
||||
{"role": "assistant", "content": "vLLM is a high-throughput LLM serving engine..."},
|
||||
{"role": "user", "content": "How does disaggregated prefilling work?"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
!!! note
|
||||
The `conversation_id` field is a non-standard extension to the OpenAI API. It is consumed by the proxy and not forwarded to the vLLM engine.
|
||||
|
||||
### Limitations
|
||||
|
||||
- Requires a stateful proxy (or equivalent router) to track and forward `kv_transfer_params` between turns.
|
||||
- Currently supported on CUDA with device-buffer KV cache. Host-buffer support (e.g., for Intel XPU) is planned for future work.
|
||||
|
||||
!!! warning "Reasoning models with stripped thinking traces"
|
||||
When using reasoning models (e.g. DeepSeek-R1) that produce thinking traces
|
||||
(`<think>...</think>`), D's KV blocks cover the full token sequence including
|
||||
thinking tokens. If the client strips thinking traces from the conversation
|
||||
history before sending the next turn, the prompt P receives will be missing
|
||||
tokens from the middle of what D generated. The block-alignment logic assumes
|
||||
P's prompt is a prefix of D's sequence, so pulling KV blocks from D in this
|
||||
case transfers cache computed for the wrong token positions, producing
|
||||
incorrect results.
|
||||
|
||||
We currently assume the router is able to detect such mismatch across turns. See [#43094](https://github.com/vllm-project/vllm/issues/43094).
|
||||
|
||||
## Multi-Instance Setup
|
||||
|
||||
### Multiple Prefiller Instances on Different Machines
|
||||
|
||||
Reference in New Issue
Block a user