[Frontend] Reuse prefill token ids on the decode chat path for disaggregated serving (#48145)

Signed-off-by: Seiji Eicher <seiji@anyscale.com>
This commit is contained in:
Seiji Eicher
2026-07-29 10:02:57 +02:00
committed by GitHub
parent 65a1a16594
commit 6370e53f24
6 changed files with 198 additions and 13 deletions
+28
View File
@@ -49,6 +49,34 @@ Now supports 9 types of connectors:
--kv-transfer-config '{"kv_connector":"FlexKVConnectorV1","kv_role":"kv_both"}'
```
## Reusing prefill token ids on decode
!!! note
This applies to disaggregated prefill and decode serving on the `/v1/chat/completions` endpoint, using a KV connector configured as in the Usage example above. It is experimental and subject to change.
In disaggregated serving, the prefill and decode stages both render the chat prompt from `messages` and tokenize it. Because the prefill stage has already produced the token ids, the decode stage can reuse them and skip its own templating and tokenization. The output is otherwise identical to a normal chat completion: it is detokenized to text, and tool and reasoning parsing, streaming, and structured output constraints all still apply.
The token ids are passed to the decode stage through `kv_transfer_params`, the dict already attached to the decode request to coordinate the transfer:
1. Send the prefill request with `return_token_ids` enabled, and read `prompt_token_ids` from the response.
2. Set `kv_transfer_params["prompt_token_ids"]` to those ids on the decode request. `messages` is still required, but its content is not tokenized when the ids are present.
```python
prefill = client.chat.completions.create(
model=model,
messages=messages,
extra_body={"return_token_ids": True, "kv_transfer_params": {"do_remote_decode": True}},
)
ids = prefill.prompt_token_ids
decode = client.chat.completions.create(
model=model,
messages=messages,
stream=True,
extra_body={"kv_transfer_params": {"do_remote_prefill": True, "prompt_token_ids": ids}},
)
```
## Development
We implement disaggregated prefilling by running 2 vLLM instances. One for prefill (we call it prefill instance) and one for decode (we call it decode instance), and then use a connector to transfer the prefill KV caches and results from prefill instance to decode instance.