[KVConnector][NIXL] Support attention-HMA layouts in pipeline-parallel push prefill (#50494)

Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Yiliu Dong <91178480+qianlihuang@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
qizixi
2026-09-15 15:43:56 -07:00
committed by GitHub
co-authored by Claude Yiliu Dong Codex
parent c6fa1f05d1
commit 0fefffc934
8 changed files with 713 additions and 49 deletions
+44
View File
@@ -181,6 +181,50 @@ The completion notif sent from P to D after a WRITE is the existing
is D's own request id, taken from the registration), so the D-side
accounting code is unchanged.
## Pipeline parallelism and hybrid KV caches
The pull connector requires the local and remote workers to expose a
congruent list of KV regions — region *i* here corresponds to region
*i* there. That assumption breaks under pipeline parallelism (PP)
combined with a hybrid (HMA) KV layout:
* a PP-sharded prefiller (**P**) holds only a slice of the model's
layers while the `PP=1` decoder (**D**) holds them all, so region
counts differ;
* with HMA, several layer names are pooled into one region and the layer
that represents a pooled region can differ between P and D.
`NixlPushConnector` handles this for PP-sharded producers by routing
**by layer-name (member) identity** instead of by region index. Each
worker advertises which layer names back each of its NIXL regions in the
handshake metadata (`NixlAgentMetadata.region_members`). A producer that
needs member routing derives its member-major layout once, when it
registers its KV caches, and every transfer it issues uses that order.
`add_remote_agent` then selects exactly the remote regions this stage
owns and reorders them to match, so both sides stay paired regardless of
how each remote rank happens to order its metadata.
Addresses, block lengths, strides, and per-region capacities follow the same
member order. Descriptor offsets use each member's region capacity, so P and
D need not allocate the same number of blocks. Physical allocations are still
registered once, even when multiple members share them.
Invariants enforced when the remote regions are aligned:
* every locally owned member must be advertised exactly once by the
remote; a missing member fails the handshake rather than silently
leaving that layer's KV stale, and remote-only members (owned by other
PP stages) are ignored;
* a remote that omits member metadata while the local layout requires
member routing fails the handshake instead of falling back to
region-index routing;
* member order is a property of the local layout alone, so the same local
source descriptors serve every remote engine and TP rank; only the
remote descriptor list is rebuilt per rank.
Decode-side PP is unsupported because completions are counted per
consumer rank. Mamba/SSM hybrids are unsupported under PP.
## Scheduler-side responsibilities
`NixlPushConnectorScheduler` extends the base scheduler with:
@@ -104,6 +104,24 @@ By default, a **compatibility hash** is checked during handshake. P and D instan
- `LBNHC` (token-major, formerly `NHD`) layout is supported but does **not** allow heterogeneous TP head splitting.
- Experimental `LBHNC` ↔ `LBNHC` permute: enable via `--kv-transfer-config '{"enable_permute_local_kv": true}'`. Not supported with HMA.
### Pipeline parallelism
The default (pull) NixlConnector does **not** support
`pipeline-parallel-size > 1` together with a hybrid KV cache (HMA)
layout: region indices are not a stable identity across the
prefill/decode layer split, so the connector raises at startup. Use the
push connector (`NixlPushConnector`) for pipeline parallelism with
hybrid KV caches — it routes transfers by layer-name (member) identity,
so a PP-sharded prefiller can write into a `PP=1` decoder. See
[NIXL push-mode KV transfer](../design/nixl_kv_push_connector.md).
Current push PP + HMA limitations:
- Only the prefiller (producer) may be PP-sharded; decode-side PP is not supported.
- Hybrid SSM/Mamba layouts are not supported under PP.
- HMA requires the same block size on P and D.
- Attention-HMA member routing requires decode TP to be no greater than prefill TP.
### Quantized KV cache
[Quantized KV cache](quantization/quantized_kvcache.md) (e.g., FP8) requires both P and D instances to use the **same** `cache_dtype`. Mismatched cache dtypes will fail the compatibility hash check during handshake.