[KVConnector][NIXL] Support attention-HMA layouts in pipeline-parallel push prefill (#50494)

Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Yiliu Dong <91178480+qianlihuang@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
qizixi
2026-09-15 15:43:56 -07:00
committed by GitHub
co-authored by Claude Yiliu Dong Codex
parent c6fa1f05d1
commit 0fefffc934
8 changed files with 713 additions and 49 deletions
+44
View File
@@ -181,6 +181,50 @@ The completion notif sent from P to D after a WRITE is the existing
is D's own request id, taken from the registration), so the D-side
accounting code is unchanged.
## Pipeline parallelism and hybrid KV caches
The pull connector requires the local and remote workers to expose a
congruent list of KV regions — region *i* here corresponds to region
*i* there. That assumption breaks under pipeline parallelism (PP)
combined with a hybrid (HMA) KV layout:
* a PP-sharded prefiller (**P**) holds only a slice of the model's
layers while the `PP=1` decoder (**D**) holds them all, so region
counts differ;
* with HMA, several layer names are pooled into one region and the layer
that represents a pooled region can differ between P and D.
`NixlPushConnector` handles this for PP-sharded producers by routing
**by layer-name (member) identity** instead of by region index. Each
worker advertises which layer names back each of its NIXL regions in the
handshake metadata (`NixlAgentMetadata.region_members`). A producer that
needs member routing derives its member-major layout once, when it
registers its KV caches, and every transfer it issues uses that order.
`add_remote_agent` then selects exactly the remote regions this stage
owns and reorders them to match, so both sides stay paired regardless of
how each remote rank happens to order its metadata.
Addresses, block lengths, strides, and per-region capacities follow the same
member order. Descriptor offsets use each member's region capacity, so P and
D need not allocate the same number of blocks. Physical allocations are still
registered once, even when multiple members share them.
Invariants enforced when the remote regions are aligned:
* every locally owned member must be advertised exactly once by the
remote; a missing member fails the handshake rather than silently
leaving that layer's KV stale, and remote-only members (owned by other
PP stages) are ignored;
* a remote that omits member metadata while the local layout requires
member routing fails the handshake instead of falling back to
region-index routing;
* member order is a property of the local layout alone, so the same local
source descriptors serve every remote engine and TP rank; only the
remote descriptor list is rebuilt per rank.
Decode-side PP is unsupported because completions are counted per
consumer rank. Mamba/SSM hybrids are unsupported under PP.
## Scheduler-side responsibilities
`NixlPushConnectorScheduler` extends the base scheduler with:
@@ -104,6 +104,24 @@ By default, a **compatibility hash** is checked during handshake. P and D instan
- `LBNHC` (token-major, formerly `NHD`) layout is supported but does **not** allow heterogeneous TP head splitting.
- Experimental `LBHNC` ↔ `LBNHC` permute: enable via `--kv-transfer-config '{"enable_permute_local_kv": true}'`. Not supported with HMA.
### Pipeline parallelism
The default (pull) NixlConnector does **not** support
`pipeline-parallel-size > 1` together with a hybrid KV cache (HMA)
layout: region indices are not a stable identity across the
prefill/decode layer split, so the connector raises at startup. Use the
push connector (`NixlPushConnector`) for pipeline parallelism with
hybrid KV caches — it routes transfers by layer-name (member) identity,
so a PP-sharded prefiller can write into a `PP=1` decoder. See
[NIXL push-mode KV transfer](../design/nixl_kv_push_connector.md).
Current push PP + HMA limitations:
- Only the prefiller (producer) may be PP-sharded; decode-side PP is not supported.
- Hybrid SSM/Mamba layouts are not supported under PP.
- HMA requires the same block size on P and D.
- Attention-HMA member routing requires decode TP to be no greater than prefill TP.
### Quantized KV cache
[Quantized KV cache](quantization/quantized_kvcache.md) (e.g., FP8) requires both P and D instances to use the **same** `cache_dtype`. Mismatched cache dtypes will fail the compatibility hash check during handshake.
@@ -918,6 +918,7 @@ def _make_mock_worker_for_desc_ids(
worker = MagicMock(spec=NixlConnectorWorker)
worker.num_regions = num_regions
worker._uses_region_group_mapping = False
worker._transfer_layer_group_ids = ()
worker._has_mamba = has_mamba
worker._group_spec_types = group_spec_types
worker.block_len_per_layer = block_len_per_layer or [100]
@@ -12,6 +12,7 @@ mid-decode (silent corruption of an unrelated request).
"""
from collections import defaultdict
from threading import Event, Lock
from unittest.mock import MagicMock, patch
import numpy as np
@@ -112,6 +113,7 @@ def test_local_descriptors_follow_each_region_pool_capacity():
worker.block_len_per_layer = [16, 16]
worker.block_stride_per_layer = [16, 16]
worker.region_num_blocks = [2, 3]
worker._transfer_layer_region_indices = ()
descriptors = worker._build_fa_local([100, 1000], block_size_ratio=1)
@@ -119,7 +121,8 @@ def test_local_descriptors_follow_each_region_pool_capacity():
@pytest.mark.cpu_test
def test_overlaid_transfer_groups_share_region_geometry():
@pytest.mark.parametrize("push_pp", [False, True])
def test_overlaid_transfer_groups_share_region_geometry(push_pp):
"""Groups overlaid on one allocation share its transfer region."""
import msgspec
@@ -127,6 +130,9 @@ def test_overlaid_transfer_groups_share_region_geometry():
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.metadata import (
NixlAgentMetadata,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.push_worker import (
NixlPushConnectorWorker,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker import (
NixlConnectorWorker,
)
@@ -153,7 +159,14 @@ def test_overlaid_transfer_groups_share_region_geometry():
}
groups = [KVCacheGroupSpec([layer_name], spec) for layer_name in caches]
worker = object.__new__(NixlConnectorWorker)
worker_cls = NixlPushConnectorWorker if push_pp else NixlConnectorWorker
worker = object.__new__(worker_cls)
if push_pp:
worker._push_writer_stop = Event()
worker._push_writer_wake = Event()
worker._push_writer_thread = MagicMock()
worker._sending_transfers_lock = Lock()
worker._sending_transfers = defaultdict(list)
worker.tp_rank = 0
worker.world_size = 1
worker.transfer_tp_rank = 0
@@ -198,6 +211,9 @@ def test_overlaid_transfer_groups_share_region_geometry():
worker._desc_is_dram_by_block_size = {}
worker._desc_pos_by_block_size = {}
worker._dram_src_handles_by_block_size = {}
worker._transfer_layer_names = ()
worker._transfer_layer_region_indices = ()
worker._transfer_layer_group_ids = ()
worker._region_is_mla = []
worker.block_len_per_layer = []
worker.block_stride_per_layer = []
@@ -205,7 +221,8 @@ def test_overlaid_transfer_groups_share_region_geometry():
worker.use_host_buffer = False
worker.host_xfer_buffers = {}
worker.device_kv_caches = {}
worker.pp_size = 1
worker.pp_size = 2 if push_pp else 1
worker._is_hma_required = True
worker.dcp_size = 1
worker.pcp_size = 1
worker.kv_buffer_device = "cuda"
@@ -240,7 +257,13 @@ def test_overlaid_transfer_groups_share_region_geometry():
expected_addrs = [
backing.data_ptr() + block * block_stride for block in range(num_blocks)
]
assert worker.src_blocks_data[:, 0].tolist() == expected_addrs
num_desc_regions = 2 if push_pp else 1
assert worker.src_blocks_data[:, 0].tolist() == expected_addrs * num_desc_regions
assert worker.num_descs == num_blocks * num_desc_regions
assert (
worker.dst_region_num_blocks[worker.engine_id]
== [num_blocks] * num_desc_regions
)
metadata = msgspec.msgpack.decode(
worker.xfer_handshake_metadata.agent_metadata_bytes,
@@ -248,6 +271,7 @@ def test_overlaid_transfer_groups_share_region_geometry():
)
assert metadata.region_group_ids == [-1]
assert metadata.region_num_blocks == [num_blocks]
assert metadata.region_members == ([["layer.0", "layer.1"]] if push_pp else [])
assert worker._block_ids_by_region(([0], [2]), worker.region_group_ids) == [[0, 2]]
@@ -32,9 +32,13 @@ from unittest.mock import MagicMock, patch
import msgspec
import pytest
from vllm.distributed.kv_transfer.kv_connector.utils import TransferTopology
from vllm.distributed.kv_transfer.kv_connector.v1.base import (
KVConnectorTransferResults,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker import (
NixlBaseConnectorWorker,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.metadata import (
PUSH_REG_NOTIF_PREFIX,
NixlAgentMetadata,
@@ -43,6 +47,7 @@ from vllm.distributed.kv_transfer.kv_connector.v1.nixl.metadata import (
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.push_worker import (
NixlPushConnectorWorker,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.tp_mapping import TPMapping
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.utils import (
get_base_request_id,
)
@@ -354,6 +359,7 @@ class _StubWriterWorker(NixlPushConnectorWorker):
w._failed_recv_reqs = queue.Queue()
w._recv_failures = set()
w._recving_transfers = defaultdict(list)
w._is_hma_required = False
w._reqs_to_process = set()
w._reqs_to_send = {}
w.consumer_notification_counts_by_req = defaultdict(int)
@@ -361,12 +367,16 @@ class _StubWriterWorker(NixlPushConnectorWorker):
w.pcp_rank = 0
w.pcp_dcp_sharded = False
w.world_size = 1
w.pp_size = 1
w.engine_id = "test-decode-engine"
w._remote_agents = {}
w._handshake_lock = threading.RLock()
w._physical_blocks_per_logical_kv_block = 1
w._uses_region_group_mapping = False
w.region_group_ids = [0]
w._transfer_layer_names = ()
w._transfer_layer_region_indices = ()
w._transfer_layer_group_ids = ()
w.dst_region_num_blocks = {}
w.dst_region_group_ids = {}
w.dst_uses_region_group_mapping = {}
@@ -922,12 +932,20 @@ class TestPushWriterNotifs:
# Both handles were still cleaned up.
assert request_id not in w._sending_transfers
def test_completed_push_send_waits_for_consumer_notif(self):
@pytest.mark.parametrize(("pp_size", "is_hma"), [(1, False), (2, True)])
def test_completed_push_send_waits_for_consumer_notif(self, pp_size, is_hma):
"""P-side sends are completed by consumer notifs / lease expiry,
not by local WRITE-handle completion."""
w = self._pollable_worker()
w.pp_size = pp_size
w._is_hma_required = is_hma
request_id = self._make_sending_req(w)
w.nixl_wrapper.check_xfer_state.return_value = "DONE"
w.nixl_wrapper.check_xfer_state.side_effect = ["DONE", "PROC", "DONE"]
results = w.get_transfer_results()
assert request_id not in results.finished_sending
assert w._sending_transfers[request_id] == [102]
assert request_id in w._reqs_to_send
results = w.get_transfer_results()
@@ -940,6 +958,12 @@ class TestPushWriterNotifs:
results = w.get_transfer_results()
assert request_id in results.finished_sending
assert request_id not in w._reqs_to_send
assert request_id not in w._reqs_to_process
assert w._evict_finished_inbox.get_nowait() == request_id
assert w.get_transfer_results().finished_sending == set()
assert w._evict_finished_inbox.empty()
assert w.nixl_wrapper.release_xfer_handle.call_count == 2
w.xfer_stats.record_kv_expired_req.assert_not_called()
# ----------------------------------------------------------------- #
@@ -1576,3 +1600,375 @@ class TestPushPrefixCaching:
local, remote = self._written_block_ids(w)
assert local == [10, 11, 12]
assert remote == [500, 501, 502]
@pytest.mark.parametrize("layer_name_routing", [False, True])
def test_different_region_groups_require_layer_routing(self, layer_name_routing):
"""A layer mapped to a pooled remote region can still receive its blocks."""
w, engine_id = self._worker_driving_xfer()
w.dst_region_group_ids[engine_id] = [-1]
if layer_name_routing:
w._transfer_layer_group_ids = (0,)
reg = _registration_data("req-groups", local_block_ids=([500, 501],))
if not layer_name_routing:
with pytest.raises(NotImplementedError, match="cache-group layouts"):
NixlPushConnectorWorker._do_start_push_kv(
w, "req-groups", ([10, 11],), reg
)
w.nixl_wrapper.make_prepped_xfer.assert_not_called()
return
NixlPushConnectorWorker._do_start_push_kv(w, "req-groups", ([10, 11],), reg)
local, remote = self._written_block_ids(w)
assert local == [10, 11]
assert remote == [500, 501]
def _agent_metadata(
region_layers: list[list[str]],
base_addresses: list[int],
block_lens: list[int],
block_strides: list[int] | None = None,
) -> NixlAgentMetadata:
return NixlAgentMetadata(
engine_id="remote-engine",
agent_metadata=b"agent",
kv_caches_base_addr=base_addresses,
device_id=7,
num_blocks=2,
block_lens=block_lens,
block_strides=list(block_lens if block_strides is None else block_strides),
kv_cache_layout="LBHNC",
block_size=16,
ssm_sizes=(0, 0),
attn_backend_name="FLASH_ATTN",
physical_blocks_per_logical_kv_block=1,
region_members=region_layers,
)
def _layer_routing_worker(
region_layers: list[list[str]],
group_by_layer: dict[str, int],
pp_size: int = 2,
is_hma: bool = True,
) -> _StubWriterWorker:
worker = _StubWriterWorker.fresh()
worker.pp_size = pp_size
worker.dcp_size = 1
worker.block_size = 16
worker._has_mamba = False
worker._is_hma_required = is_hma
worker.kv_cache_config = MagicMock(transfer_group_index_by_layer=group_by_layer)
worker.transfer_topo = MagicMock()
worker._set_region_layers(region_layers)
return worker
@pytest.mark.parametrize(
("region_num_blocks", "expected"),
[(None, [1, 2, 15, 21, 22]), ([4, 7, 5], [1, 2, 9, 12, 13])],
)
def test_layer_group_ids_route_descriptor_blocks(region_num_blocks, expected):
worker = _layer_routing_worker(
[["a", "a.swa"], ["b"]], {"a": 0, "a.swa": 1, "b": 0}
)
desc_ids = worker._compute_desc_ids(
block_ids=[[1, 2], [5]],
dst_num_blocks=10,
block_size_ratio=None,
physical_blocks_per_logical=1,
region_num_blocks=region_num_blocks,
)
assert desc_ids.tolist() == expected
def test_layer_metadata_round_trip():
metadata = _agent_metadata([["L0", "L1"]], [0x10000], [256])
metadata.region_num_blocks = [4]
metadata.region_group_ids = [-1]
metadata.region_names = ["L0"]
metadata.region_mem_types = ["VRAM"]
encoded = msgspec.msgpack.encode(metadata)
assert msgspec.msgpack.Decoder(NixlAgentMetadata).decode(encoded) == metadata
def test_layer_identity_gate_preserves_the_non_hma_path():
assert _layer_routing_worker([["a"]], {"a": 0})._transfer_layer_region_indices == (
0,
)
# A base worker (pull) never routes by layer name.
pull = object.__new__(NixlBaseConnectorWorker)
pull._transfer_layer_names = ()
pull._transfer_layer_region_indices = ()
pull._transfer_layer_group_ids = ()
pull._set_region_layers([["a"]])
assert pull._transfer_layer_region_indices == ()
# A non-HMA local layout does not require layer-name routing.
assert (
_layer_routing_worker(
[["a"]], {"a": 0}, is_hma=False
)._transfer_layer_region_indices
== ()
)
# PP=1 has congruent local/remote regions and keeps the region route even
# when its allocator is hybrid.
assert (
_layer_routing_worker(
[["a"]], {"a": 0}, pp_size=1
)._transfer_layer_region_indices
== ()
)
def test_layer_alignment_fails_loud_when_remote_omits_layers():
# Falling back to region-index routing here would silently transfer stale
# KV, so an unannounced peer must fail the handshake instead.
worker = _layer_routing_worker([["a"]], {"a": 0})
with pytest.raises(AssertionError, match="no region_members"):
worker._align_remote_regions_by_layer(_agent_metadata([], [0xA000], [128]))
def test_layer_alignment_expands_pooled_regions():
worker = _layer_routing_worker(
[["a", "a.swa"], ["b"]], {"a": 0, "a.swa": 1, "b": 0}
)
assert worker._transfer_layer_names == ("a", "a.swa", "b")
assert worker._transfer_layer_region_indices == (0, 0, 1)
assert worker._transfer_layer_group_ids == (0, 1, 0)
metadata = _agent_metadata([["a", "a.swa"], ["b"]], [0xA000, 0xB000], [128, 128])
metadata.region_num_blocks = [4, 6]
metadata.region_group_ids = [-1, 0]
metadata.region_names = ["a", "b"]
metadata.region_mem_types = ["VRAM", "VRAM"]
worker._align_remote_regions_by_layer(metadata)
assert metadata.kv_caches_base_addr == [0xA000, 0xA000, 0xB000]
assert metadata.block_lens == [128, 128, 128]
assert metadata.block_strides == [128, 128, 128]
assert metadata.region_members == [["a"], ["a.swa"], ["b"]]
assert metadata.region_num_blocks == [4, 4, 6]
assert metadata.region_group_ids == [-1, -1, 0]
assert metadata.region_names == ["a", "a.swa", "b"]
assert metadata.region_mem_types == ["VRAM", "VRAM", "VRAM"]
def test_layer_alignment_filters_and_reorders_a_pp_stage():
worker = _layer_routing_worker([["l2"], ["l3"]], {"l2": 0, "l3": 1})
metadata = _agent_metadata(
[["l2"], ["l0"], ["l3"], ["l1"]],
[0xC000, 0xA000, 0xD000, 0xB000],
[65536, 65536, 32768, 32768],
[131072, 131072, 65536, 65536],
)
metadata.region_num_blocks = [4, 6, 8, 10]
metadata.region_group_ids = [0, 0, 1, 1]
metadata.region_names = ["l2", "l0", "l3", "l1"]
metadata.region_mem_types = ["VRAM"] * 4
worker._align_remote_regions_by_layer(metadata)
assert worker._transfer_layer_region_indices == (0, 1)
assert worker._transfer_layer_group_ids == (0, 1)
assert metadata.kv_caches_base_addr == [0xC000, 0xD000]
assert metadata.block_lens == [65536, 32768]
assert metadata.block_strides == [131072, 65536]
assert metadata.region_num_blocks == [4, 8]
assert metadata.region_group_ids == [0, 1]
assert metadata.region_names == ["l2", "l3"]
assert metadata.region_mem_types == ["VRAM", "VRAM"]
@pytest.mark.parametrize(
("local_num_blocks", "remote_num_blocks"),
[([4, 4], [4, 4, 4]), ([4, 6], [3, 5, 7])],
)
def test_layer_descriptors_pair_layers_across_asymmetric_pp_split(
local_num_blocks, remote_num_blocks
):
"""A PP split can leave each stage a different mix of attention types.
Stage 1 owns L3 (full) pooled with L4 (sliding) in region 0, and L5 (full)
alone in region 1. The consumer holds every layer and pools them
differently, so region indices cannot be paired positionally.
"""
worker = _layer_routing_worker([["L3", "L4"], ["L5"]], {"L3": 0, "L4": 1, "L5": 0})
worker.block_len_per_layer = [128, 128]
worker.block_stride_per_layer = [256, 512]
worker._region_is_mla = [False, False]
worker.num_blocks = 4
worker.region_num_blocks = local_num_blocks
worker.device_id = 0
consumer = _agent_metadata(
[["L0", "L1"], ["L3", "L2"], ["L5", "L4"]],
[0xA000, 0xB000, 0xC000],
[128, 128, 128],
[256, 512, 1024],
)
consumer.num_blocks = 4
consumer.region_num_blocks = remote_num_blocks
worker._align_remote_regions_by_layer(consumer)
# L3 -> remote region 1, L4 -> 2, L5 -> 2. Pairing by index would have sent
# L5 to region 1 and L4 to region 2's sibling.
assert consumer.kv_caches_base_addr == [0xB000, 0xC000, 0xC000]
plan = TPMapping(((0,), (0,)), (0,), {0: 0}, 0)
local_descs = worker._build_fa_local([0x1000, 0x2000], block_size_ratio=1)
remote_descs = worker._build_fa_remote(plan, consumer, block_size_ratio=1)
local_counts = [local_num_blocks[i] for i in worker._transfer_layer_region_indices]
local_ids = worker._compute_desc_ids(
[[1, 2], [3]], 4, None, 1, region_num_blocks=local_counts
)
remote_ids = worker._compute_desc_ids(
[[0, 1], [2]], 4, None, 1, region_num_blocks=consumer.region_num_blocks
)
assert len(local_descs) == sum(local_counts)
assert len(remote_descs) == sum(consumer.region_num_blocks)
# Each pair addresses the same layer's blocks despite different pooling
# and strides. L3 and L5 use group 0; L4 uses group 1.
assert local_descs[local_ids].tolist() == [
[0x1100, 128, 0],
[0x1200, 128, 0],
[0x1300, 128, 0],
[0x2200, 128, 0],
[0x2400, 128, 0],
]
assert remote_descs[remote_ids].tolist() == [
[0xB000, 128, 7],
[0xB200, 128, 7],
[0xC800, 128, 7],
[0xC000, 128, 7],
[0xC400, 128, 7],
]
def test_layer_alignment_is_canonical_across_remote_orderings():
local, groups = [["x"], ["y"]], {"x": 0, "y": 1}
rank0 = _agent_metadata([["x"], ["y"]], [0x1000, 0x2000], [64, 128], [256, 512])
rank1 = _agent_metadata([["y"], ["x"]], [0x2000, 0x1000], [128, 64], [512, 256])
_layer_routing_worker(local, groups)._align_remote_regions_by_layer(rank0)
_layer_routing_worker(local, groups)._align_remote_regions_by_layer(rank1)
assert rank0.kv_caches_base_addr == rank1.kv_caches_base_addr == [0x1000, 0x2000]
assert rank0.block_lens == rank1.block_lens == [64, 128]
assert rank0.block_strides == rank1.block_strides == [256, 512]
def test_layer_alignment_is_idempotent():
worker = _layer_routing_worker(
[["a", "a.swa"], ["b"]], {"a": 0, "a.swa": 1, "b": 0}
)
metadata = _agent_metadata([["a", "a.swa"], ["b"]], [0xA000, 0xB000], [128, 128])
metadata.region_num_blocks = [4, 6]
metadata.region_group_ids = [-1, 0]
metadata.region_names = ["a", "b"]
metadata.region_mem_types = ["VRAM", "VRAM"]
worker._align_remote_regions_by_layer(metadata)
aligned = msgspec.msgpack.encode(metadata)
worker._align_remote_regions_by_layer(metadata)
assert msgspec.msgpack.encode(metadata) == aligned
def test_layer_alignment_rejects_missing_local_layer():
worker = _layer_routing_worker([["l0"], ["l1"]], {"l0": 0, "l1": 1})
metadata = _agent_metadata([["l0"]], [0xA000], [128])
with pytest.raises(AssertionError, match="missing locally owned layers"):
worker._align_remote_regions_by_layer(metadata)
def test_layer_alignment_rejects_duplicate_remote_layer():
worker = _layer_routing_worker([["a"]], {"a": 0})
metadata = _agent_metadata([["a"], ["a"]], [0xA000, 0xB000], [128, 128])
with pytest.raises(AssertionError, match="in multiple regions"):
worker._align_remote_regions_by_layer(metadata)
def test_layer_alignment_rejects_inconsistent_remote_metadata():
worker = _layer_routing_worker([["a"]], {"a": 0})
metadata = _agent_metadata([["a"], ["b"]], [0xA000], [128])
with pytest.raises(AssertionError, match="lengths disagree"):
worker._align_remote_regions_by_layer(metadata)
@pytest.mark.parametrize(
"field",
["region_num_blocks", "region_group_ids", "region_names", "region_mem_types"],
)
def test_layer_alignment_rejects_inconsistent_region_geometry(field):
worker = _layer_routing_worker([["a"]], {"a": 0})
metadata = _agent_metadata([["a"], ["b"]], [0xA000, 0xB000], [128, 128])
setattr(metadata, field, [])
original = msgspec.msgpack.encode(metadata)
with pytest.raises(AssertionError, match="lengths disagree"):
worker._align_remote_regions_by_layer(metadata)
assert msgspec.msgpack.encode(metadata) == original
def test_set_region_layers_rejects_duplicate_local_layer():
with pytest.raises(AssertionError, match="spans multiple NIXL regions"):
_layer_routing_worker([["a"], ["a"]], {"a": 0})
def test_set_region_layers_rejects_layer_outside_any_kv_group():
with pytest.raises(AssertionError, match="outside any local group"):
_layer_routing_worker([["a"], ["b"]], {"a": 0})
@pytest.mark.parametrize(
("local_block_size", "remote_block_size", "remote_tp_size", "error"),
[
(32, 16, 1, "identical P/D block sizes"),
(16, 32, 1, "identical P/D block sizes"),
(16, 16, 2, "decode TP greater"),
],
)
def test_layer_handshake_rejects_unsupported_geometry(
local_block_size: int, remote_block_size: int, remote_tp_size: int, error: str
):
"""Reject unsupported peers without registering agents or transfer state."""
metadata = _agent_metadata([["a"]], [0xA000], [128])
metadata.block_size = remote_block_size
worker = _layer_routing_worker([["a"]], {"a": 0})
worker.block_size = local_block_size
worker.block_len_per_layer = [128]
worker.use_mla = False
worker.nixl_wrapper = MagicMock()
worker.kv_caches_base_addr = defaultdict(dict)
worker.dst_num_blocks = {}
worker.tp_mappings = {}
worker.transfer_topo = TransferTopology(
tp_rank=0,
tp_size=1,
block_size=local_block_size,
engine_id=worker.engine_id,
is_mla=False,
is_mamba=False,
total_num_kv_heads=8,
attn_backends=[],
)
with pytest.raises(NotImplementedError, match=error):
worker.add_remote_agent(metadata, remote_tp_size=remote_tp_size)
worker.nixl_wrapper.add_remote_agent.assert_not_called()
worker.nixl_wrapper.prep_xfer_dlist.assert_not_called()
assert not worker.tp_mappings
assert not worker.dst_num_blocks
assert not worker.kv_caches_base_addr
with pytest.raises(KeyError):
worker.transfer_topo.get_engine_info(metadata.engine_id)
@@ -164,6 +164,10 @@ class NixlBaseConnectorWorker:
# Overridden by NixlPushConnectorWorker.
_TRANSFER_MODE: str = "pull"
# Layer-name routing is supported only by NixlPushConnector for HMA KV
# caches under PP.
_supports_pp_hma = False
def _compute_desc_ids(
self,
block_ids: BlockIds,
@@ -191,28 +195,42 @@ class NixlBaseConnectorWorker:
num_blocks = dst_num_blocks
if block_size_ratio is not None:
num_blocks = int(num_blocks * block_size_ratio)
num_regions = self.num_regions or len(self.block_len_per_layer)
use_layer_name_routing = bool(self._transfer_layer_group_ids)
num_desc_regions = (
len(self._transfer_layer_group_ids)
if use_layer_name_routing
else self.num_regions or len(self.block_len_per_layer)
)
if region_num_blocks is None:
region_num_blocks = [num_blocks] * num_regions
region_num_blocks = [num_blocks] * num_desc_regions
elif block_size_ratio is not None:
region_num_blocks = [
int(count * block_size_ratio) for count in region_num_blocks
]
if region_num_blocks:
num_regions = len(region_num_blocks)
assert len(region_num_blocks) == num_regions
if region_num_blocks and not use_layer_name_routing:
num_desc_regions = len(region_num_blocks)
assert len(region_num_blocks) == num_desc_regions
region_offsets = np.cumsum([0, *region_num_blocks[:-1]])
num_fa_descs = sum(region_num_blocks)
if use_layer_name_routing:
# Descriptors in layer order, using each layer's KV-cache group.
return np.concatenate(
[
np.asarray(block_ids[g], dtype=np.int32) + region_offsets[k]
for k, g in enumerate(self._transfer_layer_group_ids)
]
)
# All-attention fast path.
if num_ssm_regions == 0:
if region_group_ids is None:
region_group_ids = self.region_group_ids
if uses_region_group_mapping is None:
uses_region_group_mapping = self._uses_region_group_mapping
if not region_group_ids and num_regions == 1:
if not region_group_ids and num_desc_regions == 1:
region_group_ids = [0]
assert len(region_group_ids) == num_regions
assert len(region_group_ids) == num_desc_regions
if uses_region_group_mapping is None:
uses_region_group_mapping = len(set(region_group_ids)) > 1
region_group_ids_array = np.asarray(region_group_ids, dtype=np.int32)
@@ -392,6 +410,103 @@ class NixlBaseConnectorWorker:
"""
return region_idx < len(self._region_is_mla) and self._region_is_mla[region_idx]
def _set_region_layers(self, region_layers: list[list[str]]) -> None:
layer_names = [layer_name for region in region_layers for layer_name in region]
assert len(layer_names) == len(set(layer_names)), (
"A KV cache layer spans multiple NIXL regions"
)
self.region_members: list[list[str]] = region_layers
if not self._requires_layer_name_routing():
return
group_by_layer = self.kv_cache_config.transfer_group_index_by_layer
ungrouped = [name for name in layer_names if name not in group_by_layer]
assert not ungrouped, f"KV cache layers outside any local group: {ungrouped}"
self._transfer_layer_names = tuple(layer_names)
self._transfer_layer_region_indices = tuple(
region_idx
for region_idx, region in enumerate(region_layers)
for _ in region
)
self._transfer_layer_group_ids = tuple(
group_by_layer[name] for name in layer_names
)
def _requires_layer_name_routing(self) -> bool:
"""Whether PP push must match HMA layers by name, not region index."""
return (
self._supports_pp_hma
and self.pp_size > 1
and self._is_hma_required
and not self._has_mamba
)
def _align_remote_regions_by_layer(
self, nixl_agent_meta: NixlAgentMetadata
) -> None:
"""Select remote regions for this PP stage in local layer order."""
remote_region_layers = nixl_agent_meta.region_members
# Region-index routing here would silently transfer stale KV.
assert remote_region_layers, "Remote advertised no region_members"
assert (
len(nixl_agent_meta.kv_caches_base_addr)
== len(nixl_agent_meta.block_lens)
== len(nixl_agent_meta.block_strides)
== len(remote_region_layers)
), "Remote region metadata lengths disagree"
assert all(
values is None or len(values) == len(remote_region_layers)
for values in (
nixl_agent_meta.region_num_blocks,
nixl_agent_meta.region_group_ids,
nixl_agent_meta.region_names,
nixl_agent_meta.region_mem_types,
)
), "Remote region metadata lengths disagree"
remote_region_by_layer: dict[str, int] = {}
for region_idx, layer_names in enumerate(remote_region_layers):
for layer_name in layer_names:
assert layer_name not in remote_region_by_layer, (
f"Remote advertised layer {layer_name!r} in multiple regions"
)
remote_region_by_layer[layer_name] = region_idx
missing = [
name
for name in self._transfer_layer_names
if name not in remote_region_by_layer
]
assert not missing, f"Remote is missing locally owned layers: {missing}"
remote_regions = [
remote_region_by_layer[name] for name in self._transfer_layer_names
]
nixl_agent_meta.kv_caches_base_addr = [
nixl_agent_meta.kv_caches_base_addr[i] for i in remote_regions
]
nixl_agent_meta.block_lens = [
nixl_agent_meta.block_lens[i] for i in remote_regions
]
nixl_agent_meta.block_strides = [
nixl_agent_meta.block_strides[i] for i in remote_regions
]
if nixl_agent_meta.region_num_blocks is not None:
nixl_agent_meta.region_num_blocks = [
nixl_agent_meta.region_num_blocks[i] for i in remote_regions
]
if nixl_agent_meta.region_group_ids is not None:
nixl_agent_meta.region_group_ids = [
nixl_agent_meta.region_group_ids[i] for i in remote_regions
]
if nixl_agent_meta.region_mem_types is not None:
nixl_agent_meta.region_mem_types = [
nixl_agent_meta.region_mem_types[i] for i in remote_regions
]
if nixl_agent_meta.region_names is not None:
nixl_agent_meta.region_names = list(self._transfer_layer_names)
# One layer per region keeps a second alignment pass a no-op.
nixl_agent_meta.region_members = [[name] for name in self._transfer_layer_names]
def __init__(
self,
vllm_config: "VllmConfig",
@@ -638,18 +753,26 @@ class NixlBaseConnectorWorker:
# transfers into the matching sub-range of a PP=1 remote's regions.
self.pp_size = vllm_config.parallel_config.pipeline_parallel_size
self._remote_region_offset = 0
# PP push slices regions per layer (uniform count); HMA breaks that.
if self.pp_size > 1 and self._is_hma_required:
raise NotImplementedError(
"NixlPushConnector does not support pipeline_parallel_size > 1 "
"with hybrid KV cache layouts (HMA) yet."
)
# Decode-side PP is unsupported (completions counted per consumer rank).
if vllm_config.kv_transfer_config.kv_role == "kv_consumer" and self.pp_size > 1:
raise NotImplementedError(
"NixlPushConnector consumer (decode) does not support "
"pipeline_parallel_size > 1."
)
if self.pp_size > 1:
if self._is_hma_required and not self._supports_pp_hma:
raise NotImplementedError(
"NixlConnector (pull) does not support pipeline_parallel_size "
"> 1 with hybrid KV cache layouts (HMA); use NixlPushConnector "
"for PP + HMA."
)
# PP push routes HMA (hybrid) attention layouts by layer name;
# Mamba/SSM hybrids are not yet supported under PP.
if self._has_mamba:
raise NotImplementedError(
"NixlPushConnector does not support pipeline_parallel_size > 1 "
"with Mamba/SSM hybrid KV cache layouts yet."
)
# Decode-side PP is unsupported (completions counted per consumer rank).
if vllm_config.kv_transfer_config.kv_role == "kv_consumer":
raise NotImplementedError(
"NixlPushConnector consumer (decode) does not support "
"pipeline_parallel_size > 1."
)
# Keep heartbeat handshakes to a PP-sharded producer notif-only.
self._hb_handshake_notif_only = False
@@ -773,6 +896,13 @@ class NixlBaseConnectorWorker:
# within a block (BLHNC/BHLNC), where stride > block_len.
self.block_stride_per_layer = list[int]()
# Region layer membership is populated for HMA layouts.
self.region_members = []
# Local layer order, set by ``_set_region_layers`` only when
# routing by layer name. Empty means region-index routing.
self._transfer_layer_names = tuple[str, ...]()
self._transfer_layer_region_indices = tuple[int, ...]()
self._transfer_layer_group_ids = tuple[int, ...]()
# Per-engine TP mappings. Generated during handshake.
self.tp_mappings: dict[EngineId, TPMapping] = {}
@@ -1306,6 +1436,11 @@ class NixlBaseConnectorWorker:
):
compressed_region_owners.setdefault(cache.data_ptr(), cache)
track_region_layers = (
self._supports_pp_hma and self._is_hma_required and not self._has_mamba
)
region_layers: list[list[str]] = []
# K and V are packed into the content dim, so each attention layer is a
# single NIXL region whose block transfers as one unit. Mamba layers instead
# register separate conv/ssm sub-regions (see `_build_mamba_local`).
@@ -1508,6 +1643,12 @@ class NixlBaseConnectorWorker:
self._region_is_mla.append(is_mla_region)
region_mem_types.append(mem_type)
if track_region_layers:
if region_index == len(region_layers):
region_layers.append([layer_name])
else:
region_layers[region_index].append(layer_name)
if isinstance(layer_spec, MambaSpec):
if layer_spec.tp_replicated:
assert self._is_csa_linear
@@ -1563,7 +1704,9 @@ class NixlBaseConnectorWorker:
if self._has_mamba:
self.region_num_blocks = [self.num_blocks] * self.num_regions
if self.pp_size > 1:
self._set_region_layers(region_layers)
if self.pp_size > 1 and not self._is_hma_required:
start_layer, end_layer = self.model_config.get_layers_start_end_indices(
self.vllm_config.parallel_config
)
@@ -1573,7 +1716,12 @@ class NixlBaseConnectorWorker:
self._remote_region_offset = regions_per_layer * start_layer
# Total local FA descriptors (boundary between FA and mamba descs).
self.num_descs = sum(self.region_num_blocks)
xfer_region_num_blocks = (
[self.region_num_blocks[i] for i in self._transfer_layer_region_indices]
if self._transfer_layer_region_indices
else self.region_num_blocks
)
self.num_descs = sum(xfer_region_num_blocks)
self._mixed_mem_types = len(set(region_mem_types)) > 1
if self._mixed_mem_types:
@@ -1597,7 +1745,7 @@ class NixlBaseConnectorWorker:
self.device_kv_caches = kv_caches
self.dst_num_blocks[self.engine_id] = self.num_blocks
self.dst_region_num_blocks[self.engine_id] = self.region_num_blocks
self.dst_region_num_blocks[self.engine_id] = xfer_region_num_blocks
self.dst_region_group_ids[self.engine_id] = self.region_group_ids
self.dst_uses_region_group_mapping[self.engine_id] = (
self._uses_region_group_mapping
@@ -1647,6 +1795,7 @@ class NixlBaseConnectorWorker:
region_mem_types=self.region_mem_types,
dcp_size=self.dcp_size,
pcp_size=self.pcp_size,
region_members=self.region_members,
)
# Wrap metadata in payload with hash for defensive decoding
assert self.compat_hash is not None
@@ -1805,12 +1954,16 @@ class NixlBaseConnectorWorker:
base_addresses: list[int],
block_size_ratio: int,
) -> np.ndarray:
"""Build local FA descriptors for all layers as an Nx3 uint64 array."""
"""Build local FA descriptors as an Nx3 uint64 array."""
assert self.transfer_topo is not None
assert base_addresses, "Local KV cache base addresses must not be empty."
device_id = self.device_id
region_indices = self._transfer_layer_region_indices or range(
len(base_addresses)
)
parts: list[np.ndarray] = []
for i, base_addr in enumerate(base_addresses):
for i in region_indices:
base_addr = base_addresses[i]
block_len = self.block_len_per_layer[i] // block_size_ratio
logical_blocks = np.repeat(
np.arange(self.region_num_blocks[i], dtype=np.uint64),
@@ -1834,7 +1987,7 @@ class NixlBaseConnectorWorker:
nixl_agent_meta: NixlAgentMetadata,
block_size_ratio: int,
) -> np.ndarray:
"""Build remote FA descriptors for all layers as an Nx3 uint64 array."""
"""Build remote FA descriptors as an Nx3 uint64 array."""
assert self.transfer_topo is not None
assert nixl_agent_meta.kv_caches_base_addr, (
"Remote KV cache base addresses must not be empty."
@@ -1851,10 +2004,14 @@ class NixlBaseConnectorWorker:
device_id = nixl_agent_meta.device_id
block_strides = nixl_agent_meta.block_strides
parts: list[np.ndarray] = []
use_layer_name_routing = bool(self._transfer_layer_region_indices)
for i, base_addr in enumerate(nixl_agent_meta.kv_caches_base_addr):
replicated = self._is_region_replicated(i)
local_region = (
self._transfer_layer_region_indices[i] if use_layer_name_routing else i
)
replicated = self._is_region_replicated(local_region)
# Read our whole local region size from remote..
local_block_len = self.block_len_per_layer[i]
local_block_len = self.block_len_per_layer[local_region]
remote_kv_block_len = local_block_len // block_size_ratio
if block_size_ratio > 1:
# ..using remote kv_block_len as transfer unit
@@ -2021,9 +2178,22 @@ class NixlBaseConnectorWorker:
)
return self._remote_agents[engine_id][(0, remote_tp_rank)]
assert self.transfer_topo is not None
transfer_topo = self.transfer_topo
# Number of physical regions registered locally (one per layer/tensor).
num_local_regions = len(self.block_len_per_layer)
if (
if self._transfer_layer_region_indices:
if self.block_size != nixl_agent_meta.block_size:
raise NotImplementedError(
"Attention-HMA push requires identical P/D block sizes."
)
if remote_tp_size > transfer_topo.tp_size and not self.use_mla:
raise NotImplementedError(
"Attention-HMA push does not support decode TP greater than "
"prefill TP yet"
)
self._align_remote_regions_by_layer(nixl_agent_meta)
elif (
self.pp_size > 1
and len(nixl_agent_meta.kv_caches_base_addr) > num_local_regions
):
@@ -2038,6 +2208,10 @@ class NixlBaseConnectorWorker:
]
nixl_agent_meta.block_lens = nixl_agent_meta.block_lens[start:end]
nixl_agent_meta.block_strides = nixl_agent_meta.block_strides[start:end]
if nixl_agent_meta.region_num_blocks is not None:
nixl_agent_meta.region_num_blocks = nixl_agent_meta.region_num_blocks[
start:end
]
if nixl_agent_meta.region_group_ids is not None:
nixl_agent_meta.region_group_ids = nixl_agent_meta.region_group_ids[
start:end
@@ -2048,14 +2222,8 @@ class NixlBaseConnectorWorker:
nixl_agent_meta.region_mem_types = nixl_agent_meta.region_mem_types[
start:end
]
if nixl_agent_meta.region_num_blocks is not None:
nixl_agent_meta.region_num_blocks = nixl_agent_meta.region_num_blocks[
start:end
]
### Register remote engine in TransferTopology (idempotent).
assert self.transfer_topo is not None
transfer_topo = self.transfer_topo
physical_blocks_per_logical = (
nixl_agent_meta.physical_blocks_per_logical_kv_block
)
@@ -2349,6 +2517,9 @@ class NixlBaseConnectorWorker:
# the per-rank KV head ratio rather than the raw tp_ratio, because GQA
# replication caps per-rank heads at 1 when tp > total_kv_heads
# (issue #45330). Mamba uses the ssm_sizes counterpart, so skip here.
local_regions = self._transfer_layer_region_indices or range(
len(self.block_len_per_layer)
)
if self._has_mamba and self.use_mla:
# Hybrid MLA+SSM (e.g. KimiLinear's KDA+MLA): regions are
# kernel-granularity views of the mamba-unified page. The MLA
@@ -2365,7 +2536,7 @@ class NixlBaseConnectorWorker:
f"remote={nixl_agent_meta.block_lens}."
)
elif not self._has_mamba:
assert len(self.block_len_per_layer) == len(nixl_agent_meta.block_lens), (
assert len(local_regions) == len(nixl_agent_meta.block_lens), (
"Number of KV layers must match between prefill and decode"
)
model_replicated = self.use_mla or self.transfer_topo.is_kv_replicated(
@@ -2374,8 +2545,9 @@ class NixlBaseConnectorWorker:
total_kv_heads = self.transfer_topo.total_num_kv_heads
local_heads = self.transfer_topo.local_physical_heads
remote_heads = max(1, total_kv_heads // remote_tp_size)
for i, local_len in enumerate(self.block_len_per_layer):
replicated = model_replicated or self._is_region_replicated(i)
for i, region_idx in enumerate(local_regions):
local_len = self.block_len_per_layer[region_idx]
replicated = model_replicated or self._is_region_replicated(region_idx)
remote_len = nixl_agent_meta.block_lens[i]
if replicated:
assert local_len // block_size_ratio == remote_len, (
@@ -2406,8 +2578,7 @@ class NixlBaseConnectorWorker:
# TP workers that handhshake with same remote have same #blocks.
assert self.dst_num_blocks[remote_engine_id] == nixl_agent_meta.num_blocks
# Same number of regions/~layers.
assert len(nixl_agent_meta.kv_caches_base_addr) == len(self.block_len_per_layer)
assert len(nixl_agent_meta.kv_caches_base_addr) == len(local_regions)
num_remote_regions = len(nixl_agent_meta.kv_caches_base_addr)
if nixl_agent_meta.region_num_blocks is not None:
assert len(nixl_agent_meta.region_num_blocks) == num_remote_regions
@@ -2423,7 +2594,12 @@ class NixlBaseConnectorWorker:
if nixl_agent_meta.region_group_ids is not None:
assert len(nixl_agent_meta.region_group_ids) == num_remote_regions
if nixl_agent_meta.region_names is not None:
assert nixl_agent_meta.region_names == self.region_names, (
expected_names = (
list(self._transfer_layer_names)
if self._transfer_layer_names
else self.region_names
)
assert nixl_agent_meta.region_names == expected_names, (
"NIXL transfer regions must name the same model caches in the "
"same order"
)
@@ -2,7 +2,7 @@
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Metadata dataclasses and helpers for the NIXL connector."""
from dataclasses import dataclass
from dataclasses import dataclass, field
from typing import Any
from vllm.config import VllmConfig
@@ -47,8 +47,9 @@ PUSH_REG_NOTIF_PREFIX = b"PUSH_REG:"
# 9: Add block_strides
# 10: Add dense virtual transfer pages for compressed MLA caches
# 11: Add per-region transfer geometry and memory types to NixlAgentMetadata
# 12: Add per-region member names for PP push
#
NIXL_CONNECTOR_VERSION: int = 11
NIXL_CONNECTOR_VERSION: int = 12
@dataclass
@@ -71,6 +72,8 @@ class NixlAgentMetadata:
region_mem_types: list[str] | None = None
dcp_size: int = 1
pcp_size: int = 1
# Layer names sharing each advertised region, in region order.
region_members: list[list[str]] = field(default_factory=list)
@dataclass
@@ -83,6 +83,8 @@ class NixlPushConnectorWorker(NixlBaseConnectorWorker):
# Distinguishes push from pull in the NIXL compatibility hash.
_TRANSFER_MODE: str = "push"
_supports_pp_hma = True
def __init__(
self,
vllm_config: "VllmConfig",
@@ -530,7 +532,7 @@ class NixlPushConnectorWorker(NixlBaseConnectorWorker):
local_region_groups = self.region_group_ids
remote_region_groups = self.dst_region_group_ids[engine_id]
groups_differ = local_region_groups != remote_region_groups
if groups_differ:
if groups_differ and not self._transfer_layer_group_ids:
raise NotImplementedError(
"NixlPushConnector does not support different producer and "
"consumer cache-group layouts"