[KVConnector][NIXL] Support attention-HMA layouts in pipeline-parallel push prefill (#50494)
Signed-off-by: zixi-qi <zixi@inferact.ai> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Yiliu Dong <91178480+qianlihuang@users.noreply.github.com> Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
co-authored by
Claude
Yiliu Dong
Codex
parent
c6fa1f05d1
commit
0fefffc934
@@ -104,6 +104,24 @@ By default, a **compatibility hash** is checked during handshake. P and D instan
|
||||
- `LBNHC` (token-major, formerly `NHD`) layout is supported but does **not** allow heterogeneous TP head splitting.
|
||||
- Experimental `LBHNC` ↔ `LBNHC` permute: enable via `--kv-transfer-config '{"enable_permute_local_kv": true}'`. Not supported with HMA.
|
||||
|
||||
### Pipeline parallelism
|
||||
|
||||
The default (pull) NixlConnector does **not** support
|
||||
`pipeline-parallel-size > 1` together with a hybrid KV cache (HMA)
|
||||
layout: region indices are not a stable identity across the
|
||||
prefill/decode layer split, so the connector raises at startup. Use the
|
||||
push connector (`NixlPushConnector`) for pipeline parallelism with
|
||||
hybrid KV caches — it routes transfers by layer-name (member) identity,
|
||||
so a PP-sharded prefiller can write into a `PP=1` decoder. See
|
||||
[NIXL push-mode KV transfer](../design/nixl_kv_push_connector.md).
|
||||
|
||||
Current push PP + HMA limitations:
|
||||
|
||||
- Only the prefiller (producer) may be PP-sharded; decode-side PP is not supported.
|
||||
- Hybrid SSM/Mamba layouts are not supported under PP.
|
||||
- HMA requires the same block size on P and D.
|
||||
- Attention-HMA member routing requires decode TP to be no greater than prefill TP.
|
||||
|
||||
### Quantized KV cache
|
||||
|
||||
[Quantized KV cache](quantization/quantized_kvcache.md) (e.g., FP8) requires both P and D instances to use the **same** `cache_dtype`. Mismatched cache dtypes will fail the compatibility hash check during handshake.
|
||||
|
||||
Reference in New Issue
Block a user