diff --git a/docs/features/disagg_prefill.md b/docs/features/disagg_prefill.md index f7d3f9a70f..5167d61239 100644 --- a/docs/features/disagg_prefill.md +++ b/docs/features/disagg_prefill.md @@ -23,7 +23,7 @@ Now supports 6 types of connectors: - **ExampleConnector**: refer to [examples/offline_inference/disaggregated-prefill-v1/run.sh](../../examples/offline_inference/disaggregated-prefill-v1/run.sh) for the example usage of ExampleConnector disaggregated prefilling. - **LMCacheConnectorV1**: refer to [examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh](../../examples/others/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh) for the example usage of LMCacheConnectorV1 disaggregated prefilling which uses NIXL as the underlying KV transmission. -- **NixlConnector**: refer to [tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh](../../tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh) for the example usage of NixlConnector disaggregated prefilling which support fully async send/recv. For detailed usage guide, see [NixlConnector Usage Guide](nixl_connector_usage.md). +- **NixlConnector**: refer to [tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh](../../tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh) for the example usage of NixlConnector disaggregated prefilling which support fully async send/recv. For detailed usage guide, see [NixlConnector Usage Guide](nixl_connector_usage.md). For feature compatibility details, see [NixlConnector Compatibility Matrix](nixl_connector_compatibility.md). - **P2pNcclConnector**: refer to [examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/disagg_example_p2p_nccl_xpyd.sh](../../examples/online_serving/disaggregated_serving_p2p_nccl_xpyd/disagg_example_p2p_nccl_xpyd.sh) for the example usage of P2pNcclConnector disaggregated prefilling. - **MooncakeConnector**: refer to [examples/online_serving/disaggregated_serving/mooncake_connector/run_mooncake_connector.sh](../../examples/online_serving/disaggregated_serving/mooncake_connector/run_mooncake_connector.sh) for the example usage of ExampleConnector disaggregated prefilling. For detailed usage guide, see [MooncakeConnector Usage Guide](mooncake_connector_usage.md). - **MultiConnector**: take advantage of the kv_connector_extra_config: dict[str, Any] already present in KVTransferConfig to stash all the connectors we want in an ordered list of kwargs.such as: diff --git a/docs/features/nixl_connector_compatibility.md b/docs/features/nixl_connector_compatibility.md new file mode 100644 index 0000000000..5541cd99bd --- /dev/null +++ b/docs/features/nixl_connector_compatibility.md @@ -0,0 +1,104 @@ +# NixlConnector Compatibility Matrix + +This page documents the feature compatibility of **disaggregated prefilling with the NixlConnector**. For general usage instructions, see the [NixlConnector Usage Guide](nixl_connector_usage.md). For an overview of disaggregated prefilling, see [Disaggregated Prefilling](disagg_prefill.md). + +!!! note + This page reflects the current state of the codebase and is subject to change as features evolve. Entries marked 🟠 or ❌ may link to tracking issues. See the [NIXL connector roadmap](https://github.com/vllm-project/vllm/issues/33702) for upcoming feature development. + +**Legend:** + +- ✅ = Fully supported +- 🟠 = Partial support (see footnotes) +- ❌ = Not supported +- ❔ = Unknown / not yet validated +- 🚧 = Work in progress + +!!! info "Universally supported features" + The following features work with **all** model architectures when using NixlConnector PD disaggregated serving: + + [Chunked Prefill](../configuration/optimization.md#chunked-prefill) | + [APC (Prefix Caching)](automatic_prefix_caching.md) | + [Data Parallel](../serving/data_parallel_deployment.md) | + CUDA graph | + Logprobs | + Prompt Logprobs | + [Prompt Embeds](prompt_embeds.md) | + Multiple NIXL backends (UCX, GDS, LIBFABRIC, etc.) + +## Model Architecture x Capability + + + +| Model type | Basic PD | Spec Decode | Hetero TP | Cross-layer blocks | SWA | Host buffer | Hetero block size | +| - | - | - | - | - | - | - | - | +| Dense Transformers | ✅ | ✅1 | ✅ | ✅2 | ✅ | ✅ | 🟠3 | +| MLA (e.g. DeepSeek-V2/V3) | ✅ | ✅1 | 🟠4 | ✅2 | ✅ | ✅ | 🟠3 | +| Sparse MLA (e.g. DeepSeek-V3.2) | ✅ | ✅1 | 🟠4 | ✅2 | ✅ | ✅ | 🟠3 | +| Hybrid SSM / Mamba | ✅ | ❔ | 🚧5 | ❌ | ✅ | ✅ | ❌6 | +| MoE | ✅ | ✅1 | ✅ | ✅2 | ✅ | ✅ | 🟠3 | +| Multimodal | ❔ | ❔ | ❔ | ❔ | ❔ | ❔ | ❔ | +| Encoder-Decoder | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | + +1 P and D instances must use the same speculation configuration. + +2 Requires `FLASH_ATTN` or `FLASHINFER` backend **and** `HND` KV cache layout. Enable via `--kv-transfer-config '{"kv_connector_extra_config": {"enable_cross_layers_blocks": "True"}}'`. + +3 Supported only when HMA is **not** required (i.e., non-hybrid models). Block IDs are remapped automatically. Only P block size < D block size is supported. + +4 MLA KV cache is replicated across TP workers, so heterogeneous TP works but there is no head-splitting. When P TP > D TP, only a single read is executed (redundant ranks are skipped). D TP > P TP also works. + +5 Hybrid SSM (Mamba) models require **homogeneous TP** (`P TP == D TP`). Heterogeneous TP is not yet supported for Mamba layers. + +6 HMA (required by hybrid models) does not support different remote block sizes. + +## Configuration Notes + +### What must match between P and D + +By default, a **compatibility hash** is checked during handshake. P and D instances must agree on: + +- vLLM version and NIXL connector version +- Model (architecture, dtype, number of KV heads, head size, number of hidden layers) +- Attention backend +- KV cache dtype (`cache_dtype`) + +!!! warning + Disable the hash check with `--kv-transfer-config '{"kv_connector_extra_config": {"enforce_handshake_compat": false}}'` at your own risk. + +### What can safely differ between P and D + +- `tensor-parallel-size` (heterogeneous TP, subject to model restrictions above) +- `block-size` (heterogeneous block size, subject to restrictions above) +- Number of KV cache blocks (determined by available memory on each instance) + +### KV cache layout + +- NixlConnector defaults to **`HND`** layout for optimal transfer performance (non-MLA models). +- `NHD` layout is supported but does **not** allow heterogeneous TP head splitting. +- Experimental `HND` ↔ `NHD` permute: enable via `--kv-transfer-config '{"enable_permute_local_kv": true}'`. Not supported with HMA. + +### Quantized KV cache + +[Quantized KV cache](quantization/quantized_kvcache.md) (e.g., FP8) requires both P and D instances to use the **same** `cache_dtype`. Mismatched cache dtypes will fail the compatibility hash check during handshake. + +- **Static quantization** (scales loaded from checkpoint): ✅ Supported. Scales are loaded independently by each instance from the model checkpoint. +- **Dynamic quantization** (scales computed at runtime): ❌ Not supported. Per-block scales are not transferred alongside KV cache data. +- **Packed-layout scales** (scales stored inline with weights): ✅ Supported. Scales are transferred together with the KV cache blocks. diff --git a/docs/features/nixl_connector_usage.md b/docs/features/nixl_connector_usage.md index a9039f0daf..ea66861515 100644 --- a/docs/features/nixl_connector_usage.md +++ b/docs/features/nixl_connector_usage.md @@ -2,6 +2,8 @@ NixlConnector is a high-performance KV cache transfer connector for vLLM's disaggregated prefilling feature. It provides fully asynchronous send/receive operations using the NIXL library for efficient cross-process KV cache transfer. +For feature compatibility details (supported model architectures, TP configurations, and feature interactions), see the [NixlConnector Compatibility Matrix](nixl_connector_compatibility.md). + ## Prerequisites ### Installation