[KVConnector][NIXL] Support attention-HMA layouts in pipeline-parallel push prefill (#50494)
Signed-off-by: zixi-qi <zixi@inferact.ai> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Yiliu Dong <91178480+qianlihuang@users.noreply.github.com> Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
co-authored by
Claude
Yiliu Dong
Codex
parent
c6fa1f05d1
commit
0fefffc934
@@ -181,6 +181,50 @@ The completion notif sent from P to D after a WRITE is the existing
|
||||
is D's own request id, taken from the registration), so the D-side
|
||||
accounting code is unchanged.
|
||||
|
||||
## Pipeline parallelism and hybrid KV caches
|
||||
|
||||
The pull connector requires the local and remote workers to expose a
|
||||
congruent list of KV regions — region *i* here corresponds to region
|
||||
*i* there. That assumption breaks under pipeline parallelism (PP)
|
||||
combined with a hybrid (HMA) KV layout:
|
||||
|
||||
* a PP-sharded prefiller (**P**) holds only a slice of the model's
|
||||
layers while the `PP=1` decoder (**D**) holds them all, so region
|
||||
counts differ;
|
||||
* with HMA, several layer names are pooled into one region and the layer
|
||||
that represents a pooled region can differ between P and D.
|
||||
|
||||
`NixlPushConnector` handles this for PP-sharded producers by routing
|
||||
**by layer-name (member) identity** instead of by region index. Each
|
||||
worker advertises which layer names back each of its NIXL regions in the
|
||||
handshake metadata (`NixlAgentMetadata.region_members`). A producer that
|
||||
needs member routing derives its member-major layout once, when it
|
||||
registers its KV caches, and every transfer it issues uses that order.
|
||||
`add_remote_agent` then selects exactly the remote regions this stage
|
||||
owns and reorders them to match, so both sides stay paired regardless of
|
||||
how each remote rank happens to order its metadata.
|
||||
|
||||
Addresses, block lengths, strides, and per-region capacities follow the same
|
||||
member order. Descriptor offsets use each member's region capacity, so P and
|
||||
D need not allocate the same number of blocks. Physical allocations are still
|
||||
registered once, even when multiple members share them.
|
||||
|
||||
Invariants enforced when the remote regions are aligned:
|
||||
|
||||
* every locally owned member must be advertised exactly once by the
|
||||
remote; a missing member fails the handshake rather than silently
|
||||
leaving that layer's KV stale, and remote-only members (owned by other
|
||||
PP stages) are ignored;
|
||||
* a remote that omits member metadata while the local layout requires
|
||||
member routing fails the handshake instead of falling back to
|
||||
region-index routing;
|
||||
* member order is a property of the local layout alone, so the same local
|
||||
source descriptors serve every remote engine and TP rank; only the
|
||||
remote descriptor list is rebuilt per rank.
|
||||
|
||||
Decode-side PP is unsupported because completions are counted per
|
||||
consumer rank. Mamba/SSM hybrids are unsupported under PP.
|
||||
|
||||
## Scheduler-side responsibilities
|
||||
|
||||
`NixlPushConnectorScheduler` extends the base scheduler with:
|
||||
|
||||
@@ -104,6 +104,24 @@ By default, a **compatibility hash** is checked during handshake. P and D instan
|
||||
- `LBNHC` (token-major, formerly `NHD`) layout is supported but does **not** allow heterogeneous TP head splitting.
|
||||
- Experimental `LBHNC` ↔ `LBNHC` permute: enable via `--kv-transfer-config '{"enable_permute_local_kv": true}'`. Not supported with HMA.
|
||||
|
||||
### Pipeline parallelism
|
||||
|
||||
The default (pull) NixlConnector does **not** support
|
||||
`pipeline-parallel-size > 1` together with a hybrid KV cache (HMA)
|
||||
layout: region indices are not a stable identity across the
|
||||
prefill/decode layer split, so the connector raises at startup. Use the
|
||||
push connector (`NixlPushConnector`) for pipeline parallelism with
|
||||
hybrid KV caches — it routes transfers by layer-name (member) identity,
|
||||
so a PP-sharded prefiller can write into a `PP=1` decoder. See
|
||||
[NIXL push-mode KV transfer](../design/nixl_kv_push_connector.md).
|
||||
|
||||
Current push PP + HMA limitations:
|
||||
|
||||
- Only the prefiller (producer) may be PP-sharded; decode-side PP is not supported.
|
||||
- Hybrid SSM/Mamba layouts are not supported under PP.
|
||||
- HMA requires the same block size on P and D.
|
||||
- Attention-HMA member routing requires decode TP to be no greater than prefill TP.
|
||||
|
||||
### Quantized KV cache
|
||||
|
||||
[Quantized KV cache](quantization/quantized_kvcache.md) (e.g., FP8) requires both P and D instances to use the **same** `cache_dtype`. Mismatched cache dtypes will fail the compatibility hash check during handshake.
|
||||
|
||||
Reference in New Issue
Block a user