[KVConnector][NIXL] Support attention-HMA layouts in pipeline-parallel push prefill (#50494)

Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Yiliu Dong <91178480+qianlihuang@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
qizixi
2026-09-15 15:43:56 -07:00
committed by GitHub
co-authored by Claude Yiliu Dong Codex
parent c6fa1f05d1
commit 0fefffc934
8 changed files with 713 additions and 49 deletions
@@ -104,6 +104,24 @@ By default, a **compatibility hash** is checked during handshake. P and D instan
- `LBNHC` (token-major, formerly `NHD`) layout is supported but does **not** allow heterogeneous TP head splitting.
- Experimental `LBHNC` ↔ `LBNHC` permute: enable via `--kv-transfer-config '{"enable_permute_local_kv": true}'`. Not supported with HMA.
### Pipeline parallelism
The default (pull) NixlConnector does **not** support
`pipeline-parallel-size > 1` together with a hybrid KV cache (HMA)
layout: region indices are not a stable identity across the
prefill/decode layer split, so the connector raises at startup. Use the
push connector (`NixlPushConnector`) for pipeline parallelism with
hybrid KV caches — it routes transfers by layer-name (member) identity,
so a PP-sharded prefiller can write into a `PP=1` decoder. See
[NIXL push-mode KV transfer](../design/nixl_kv_push_connector.md).
Current push PP + HMA limitations:
- Only the prefiller (producer) may be PP-sharded; decode-side PP is not supported.
- Hybrid SSM/Mamba layouts are not supported under PP.
- HMA requires the same block size on P and D.
- Attention-HMA member routing requires decode TP to be no greater than prefill TP.
### Quantized KV cache
[Quantized KV cache](quantization/quantized_kvcache.md) (e.g., FP8) requires both P and D instances to use the **same** `cache_dtype`. Mismatched cache dtypes will fail the compatibility hash check during handshake.