diff --git a/docs/assets/features/disagg_prefill/high_level_design.png b/docs/assets/features/disagg_prefill/high_level_design.png new file mode 100644 index 0000000000..ce9b1c8827 Binary files /dev/null and b/docs/assets/features/disagg_prefill/high_level_design.png differ diff --git a/docs/assets/features/disagg_prefill/workflow.png b/docs/assets/features/disagg_prefill/workflow.png new file mode 100644 index 0000000000..9e773f4fa0 Binary files /dev/null and b/docs/assets/features/disagg_prefill/workflow.png differ diff --git a/docs/features/disagg_prefill.md b/docs/features/disagg_prefill.md index c0c32594f2..996ef00a6b 100644 --- a/docs/features/disagg_prefill.md +++ b/docs/features/disagg_prefill.md @@ -19,6 +19,18 @@ Two main reasons: Please refer to for the example usage of disaggregated prefilling. +Now supports 5 types of connectors: + +- **SharedStorageConnector**: refer to for the example usage of SharedStorageConnector disaggregated prefilling. +- **LMCacheConnectorV1**: refer to for the example usage of LMCacheConnectorV1 disaggregated prefilling which uses NIXL as the underlying KV transmission. +- **NixlConnector**: refer to for the example usage of NixlConnector disaggregated prefilling which support fully async send/recv. +- **P2pNcclConnector**: refer to for the example usage of P2pNcclConnector disaggregated prefilling. +- **MultiConnector**: take advantage of the kv_connector_extra_config: dict[str, Any] already present in KVTransferConfig to stash all the connectors we want in an ordered list of kwargs.such as: + + ```bash + --kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both"},{"kv_connector":"SharedStorageConnector","kv_role":"kv_both","kv_connector_extra_config":{"shared_storage_path":"local_storage"}}]}}' + ``` + ## Benchmarks Please refer to for disaggregated prefilling benchmarks. @@ -48,6 +60,19 @@ The workflow of disaggregated prefilling is as follows: The `buffer` corresponds to `insert` API in LookupBuffer, and the `drop_select` corresponds to `drop_select` API in LookupBuffer. +Now every process in vLLM will have a corresponding connector. Specifically, we have: + +- Scheduler connector: the connector that locates in the same process as the scheduler process. It schedules the KV cache transfer ops. +- Worker connectors: the connectors that locate in the worker processes. They execute KV cache transfer ops. + +Here is a figure illustrating how the above 2 connectors are organized: + +![Disaggregated prefilling high level design](../assets/features/disagg_prefill/high_level_design.png) + +The figure below shows how the worker connector works with the attention module to achieve layer-by-layer KV cache store and load: + +![Disaggregated prefilling workflow](../assets/features/disagg_prefill/workflow.png) + ## Third-party contributions Disaggregated prefilling is highly related to infrastructure, so vLLM relies on third-party connectors for production-level disaggregated prefilling (and vLLM team will actively review and merge new PRs for third-party connectors).