[KVConnector][NIXL] Support packed MLA KV layouts in pipeline-parallel push prefill (#50499)

Signed-off-by: zixi-qi <zixi@inferact.ai>
This commit is contained in:
qizixi
2026-09-28 18:50:46 +02:00
committed by GitHub
parent 94d1462924
commit 6d3ea3c2c9
6 changed files with 357 additions and 28 deletions
+29 -11
View File
@@ -195,35 +195,53 @@ combined with a hybrid (HMA) KV layout:
that represents a pooled region can differ between P and D.
`NixlPushConnector` handles this for PP-sharded producers by routing
**by layer-name (member) identity** instead of by region index. Each
**by layer name** instead of by region index. Each
worker advertises which layer names back each of its NIXL regions in the
handshake metadata (`NixlAgentMetadata.region_members`). A producer that
needs member routing derives its member-major layout once, when it
needs layer routing derives its transfer layer order once, when it
registers its KV caches, and every transfer it issues uses that order.
`add_remote_agent` then selects exactly the remote regions this stage
owns and reorders them to match, so both sides stay paired regardless of
how each remote rank happens to order its metadata.
Addresses, block lengths, strides, and per-region capacities follow the same
member order. Descriptor offsets use each member's region capacity, so P and
layer order. Descriptor offsets use each layer's region capacity, so P and
D need not allocate the same number of blocks. Physical allocations are still
registered once, even when multiple members share them.
registered once, even when multiple layers share them.
Packed MLA caches (block-outermost layouts such as DeepSeek-V4's) store the
pages of every layer for one block in a single row. A PP stage holds only its
own layers, so its rows are shorter and each layer sits at a different offset:
```text
D (PP=1), block b: | L0 | L1 | L2 | L3 | L4 | L5 |
P (stage 2), block b: | L3 | L4 | L5 |
```
`PP=1` peers transfer whole rows. They also advertise where each layer's page
sits in the row (`NixlAgentMetadata.packed_member_layouts`), so a PP producer
writes each of its pages to the matching slot in D's row. Page sizes must
match; offsets, row strides, and block counts may differ. Because PP stages can
list the same attention backends in a different order, packed push compares the
sorted set of backend names during the handshake. Pull mode is unchanged.
Invariants enforced when the remote regions are aligned:
* every locally owned member must be advertised exactly once by the
remote; a missing member fails the handshake rather than silently
leaving that layer's KV stale, and remote-only members (owned by other
* every locally owned layer must be advertised exactly once by the
remote; a missing layer fails the handshake rather than silently
leaving that layer's KV stale, and remote-only layers (owned by other
PP stages) are ignored;
* a remote that omits member metadata while the local layout requires
member routing fails the handshake instead of falling back to
* a remote that omits layer metadata while the local layout requires
layer routing fails the handshake instead of falling back to
region-index routing;
* member order is a property of the local layout alone, so the same local
* layer order is a property of the local layout alone, so the same local
source descriptors serve every remote engine and TP rank; only the
remote descriptor list is rebuilt per rank.
Decode-side PP is unsupported because completions are counted per
consumer rank. Mamba/SSM hybrids are unsupported under PP.
consumer rank. Mamba/SSM hybrids are unsupported under PP. Packed PP push
requires MLA caches and equal P/D block sizes. MLA pages are replicated
across TP ranks, so producer TP1 to consumer TP2 is supported.
## Scheduler-side responsibilities
@@ -120,7 +120,8 @@ Current push PP + HMA limitations:
- Only the prefiller (producer) may be PP-sharded; decode-side PP is not supported.
- Hybrid SSM/Mamba layouts are not supported under PP.
- HMA requires the same block size on P and D.
- Attention-HMA member routing requires decode TP to be no greater than prefill TP.
- Non-MLA attention-HMA layer routing requires decode TP to be no greater than prefill TP.
- Packed layouts under PP require MLA caches. Local and remote packed block strides may differ; layer pages must have equal sizes. MLA supports decode TP greater than prefill TP because its KV is replicated.
### Quantized KV cache