[Frontend] Reuse prefill token ids on the decode chat path for disaggregated serving (#48145)
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
This commit is contained in:
@@ -49,6 +49,34 @@ Now supports 9 types of connectors:
|
||||
--kv-transfer-config '{"kv_connector":"FlexKVConnectorV1","kv_role":"kv_both"}'
|
||||
```
|
||||
|
||||
## Reusing prefill token ids on decode
|
||||
|
||||
!!! note
|
||||
This applies to disaggregated prefill and decode serving on the `/v1/chat/completions` endpoint, using a KV connector configured as in the Usage example above. It is experimental and subject to change.
|
||||
|
||||
In disaggregated serving, the prefill and decode stages both render the chat prompt from `messages` and tokenize it. Because the prefill stage has already produced the token ids, the decode stage can reuse them and skip its own templating and tokenization. The output is otherwise identical to a normal chat completion: it is detokenized to text, and tool and reasoning parsing, streaming, and structured output constraints all still apply.
|
||||
|
||||
The token ids are passed to the decode stage through `kv_transfer_params`, the dict already attached to the decode request to coordinate the transfer:
|
||||
|
||||
1. Send the prefill request with `return_token_ids` enabled, and read `prompt_token_ids` from the response.
|
||||
2. Set `kv_transfer_params["prompt_token_ids"]` to those ids on the decode request. `messages` is still required, but its content is not tokenized when the ids are present.
|
||||
|
||||
```python
|
||||
prefill = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=messages,
|
||||
extra_body={"return_token_ids": True, "kv_transfer_params": {"do_remote_decode": True}},
|
||||
)
|
||||
ids = prefill.prompt_token_ids
|
||||
|
||||
decode = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=messages,
|
||||
stream=True,
|
||||
extra_body={"kv_transfer_params": {"do_remote_prefill": True, "prompt_token_ids": ids}},
|
||||
)
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
We implement disaggregated prefilling by running 2 vLLM instances. One for prefill (we call it prefill instance) and one for decode (we call it decode instance), and then use a connector to transfer the prefill KV caches and results from prefill instance to decode instance.
|
||||
|
||||
Reference in New Issue
Block a user