[Feature][Multimodal] Add cross-encoder caching to Mooncake P2P (#56242)

Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Co-authored-by: Tianyu Guo <guoty@inferact.ai>
Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
Zhou ziheng
2026-09-17 13:16:41 +08:00
committed by GitHub
co-authored by Tianyu Guo jiangkuaixue123 Codex
parent 2d8a5c5741
commit 2743dc4f92
15 changed files with 1863 additions and 154 deletions
+109
View File
@@ -0,0 +1,109 @@
# Cross-Encoder Output Reuse
Cross-Encoder output reuse lets an Encoder load image embeddings computed by
another Encoder from a shared Mooncake Store. Local cache misses check the Store
before encoding; newly computed outputs are published asynchronously. Both paths
use the existing `ECMooncakeConnector` P2P delivery to Prefill.
## Requirements
- A working [disaggregated Encoder](disagg_encoder.md) deployment using
`ECMooncakeConnector`. See the [Mooncake integration example](../../tests/v1/ec_connector/integration/run_epd_mooncake_ec_full_pipeline.sh)
for the base E/PD configuration.
- Model Runner V2 (`VLLM_USE_V2_MODEL_RUNNER=1`), Encoder TP=1, and no dynamic LoRA.
- `--mm-processor-cache-gb` greater than zero to preserve content identifiers.
- Mooncake 0.3.12 or later with a running RAM Store. Disk offloading
(`enable_offload: true`) is not supported.
Shared reuse supports images with contiguous 2D FP16, BF16 or FP32 encoder outputs.
Other modalities use normal encoding; non-contiguous outputs are not published.
## Usage
On each Encoder, add `cross_encoder_cache: true` to the existing producer
`--ec-transfer-config`, preserving its P2P settings:
```json
{
"ec_connector": "ECMooncakeConnector",
"ec_role": "ec_producer",
"ec_connector_extra_config": {
"cross_encoder_cache": true
}
}
```
Prefill keeps its existing `ECMooncakeConnector` consumer configuration and does
not need a Store client.
Create a JSON file pointing to an existing Store. For an independently managed
RAM pool, use `standalone-store` with `global_segment_size` set to zero:
```json
{
"metadata_server": "http://STORE_HOST:2379/metadata",
"master_server_address": "STORE_HOST:50051",
"protocol": "tcp",
"device_name": "",
"mode": "standalone-store",
"global_segment_size": 0,
"local_buffer_size": "4GB"
}
```
Replace the addresses with your Store endpoints and set the path on each Encoder
before starting vLLM:
```bash
export MOONCAKE_CONFIG_PATH=/path/to/mooncake_config.json
```
This connects to an existing Store; it does not start the storage services.
For Store setup and tenant configuration, see the
[Mooncake Store guide](mooncake_store_connector_usage.md#prerequisites).
## Configuration
These options belong in the producer's `ec_connector_extra_config`:
| Option | Default | Description |
| --- | --- | --- |
| `cross_encoder_cache` | `false` | Enable shared output reuse. |
| `embedding_cache_prefix` | `""` | Namespace prefix for shared embeddings. |
| `embedding_model_identity` | Configured model path | Override the model field in Store keys. Matching multimodal identifiers are still required. |
| `store_max_pending_items` | `32` | Maximum pending publications per Encoder. Must be positive. |
| `store_max_pending_bytes` | `2147483648` (2 GiB) | Maximum retained tensor storage for pending publications per Encoder. Must be positive. |
| `store_read_buffer_bytes` | `134217728` (128 MiB) | Maximum reusable CPU staging buffer per Encoder; pinned for CUDA outputs. Must be positive. |
Encoders sharing outputs must use the same immutable weights, compatible
preprocessing and matching multimodal identifiers. Use the same configured model
path for automatic identifiers: `embedding_model_identity` alone does not enable
reuse across different paths. Change `embedding_cache_prefix` when replacing
weights in place or changing output-affecting settings. Caller-provided UUIDs
must consistently identify the same input; see [cached inputs](multimodal_inputs.md#cached-inputs).
## Limitations
- Store reads are synchronous. Reuse skips Encoder computation, but not
preprocessing, P2P delivery or the scheduler's Encoder budget reservation.
- Hits are read in batches through one lazily allocated, registered CPU staging
buffer. Each chunk is copied to independent output tensors before the buffer
is reused. Reads are split by `store_read_buffer_bytes`; an individual object
larger than this capacity (including its 24-byte header) falls back to encoding.
Staging memory is additional to the Store's `local_buffer_size` and is released
on healthy shutdown. The first load includes allocation and registration costs.
- Publication is best-effort. A request can finish before its outputs reach the
Store, so concurrent cold requests may still encode the same image.
- Publication budgets count each retained backing storage once across pending
views. Its charge is released after the last view is safely reclaimed.
Exceeding a budget skips the write. Store client buffers and the P2P pool
consume additional memory.
- Cache misses and recoverable read errors fall back to encoding. Recoverable
publication errors skip the write. Unexpected native errors or unconfirmed
I/O completion or buffer release can fail the worker.
- Incompatible objects are rejected without replacement; they can trigger
repeated fallback until evicted.
- An independent Store can retain embeddings across Encoder restarts. Capacity,
eviction and resilience to Store failures depend on the Store deployment.
- The `protocol:v3` namespace isolates the compact embedding format from older
cache objects. Encoders using different format versions do not share hits.
+3
View File
@@ -45,6 +45,9 @@ Below ready-to-run scripts shows the workflow:
## 3 Test Script
For optional cross-Encoder output reuse while retaining Mooncake P2P delivery,
see [Cross-encoder output reuse](cross_encoder_cache.md).
Please refer to the directories `tests/v1/ec_connector`
## 4 Development
@@ -0,0 +1,726 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import ctypes
import gc
import struct
import threading
import weakref
from concurrent.futures import Future, ThreadPoolExecutor
from dataclasses import replace
from types import SimpleNamespace
from unittest.mock import MagicMock, patch
import pytest
import torch
from tests.v1.ec_connector.unit.utils import create_ec_vllm_config
from vllm.config import MultiModalConfig
from vllm.distributed.ec_transfer.ec_connector.base import ECConnectorRole
from vllm.distributed.ec_transfer.ec_connector.mooncake.config import MooncakeECConfig
from vllm.distributed.ec_transfer.ec_connector.mooncake.metadata import (
ECMooncakeConnectorMetadata,
ECMooncakePushSpec,
)
from vllm.distributed.ec_transfer.ec_connector.mooncake_ec_connector import (
ECMooncakeConnector,
)
from vllm.distributed.ec_transfer.ec_connector.mooncake_store_embedding import (
backend as store_backend,
)
from vllm.distributed.ec_transfer.ec_connector.mooncake_store_embedding import (
store_client,
)
from vllm.distributed.ec_transfer.ec_connector.mooncake_store_embedding.data import (
TensorSpec,
build_embedding_namespace,
make_embedding_key,
)
pytestmark = pytest.mark.cpu_test
KEY = (
"test@embedding@model:model@revision:revision"
"@encoder:encoder@dtype:torch.float32@protocol:v3"
)
SPEC = TensorSpec((2, 2), "torch.float32", 16)
def test_embedding_key_combines_namespace_and_identifier():
config = create_ec_vllm_config(ec_role="ec_producer")
config.ec_transfer_config.ec_connector_extra_config.update(
embedding_cache_prefix="team@cache", embedding_model_identity="shared/model"
)
config.model_config.revision = "rev:1"
config.model_config.dtype = torch.float32
config.model_config.multimodal_config = None
assert make_embedding_key(build_embedding_namespace(config), "image@1") == (
"team@cache@embedding@model:shared/model@revision:rev:1"
"@encoder:encoder:default@dtype:torch.float32@protocol:v3@id:image@1"
)
def test_shared_reuse_requires_content_stable_identifiers():
"""Encoder-only cache-off rendering uses process-local counter identifiers."""
config = create_ec_vllm_config(ec_role="ec_producer")
config.ec_transfer_config.ec_connector = "ECMooncakeConnector"
config.ec_transfer_config.ec_connector_extra_config["cross_encoder_cache"] = True
config.model_config.multimodal_config = MultiModalConfig(mm_processor_cache_gb=0)
config.use_v2_model_runner = True
config.lora_config = None
with pytest.raises(ValueError, match="mm_processor_cache_gb"):
MooncakeECConfig.from_vllm_config(config)
class MemoryStore:
"""Only the native boundary is faked; codec, resolver and publisher are real."""
def __init__(self):
self.objects: dict[str, bytes] = {}
self.registered: dict[int, int] = {}
def close(self):
return 0
def batch_is_exist(self, keys):
return [int(k in self.objects) for k in keys]
def register_buffer(self, addr, size):
self.registered[addr] = size
return 0
def unregister_buffer(self, addr):
del self.registered[addr]
return 0
def batch_put_from_multi_buffers(self, keys, pointers, sizes, config):
[key], [ptrs], [lengths] = keys, pointers, sizes
self.objects[key] = b"".join(
ctypes.string_at(p, n) for p, n in zip(ptrs, lengths, strict=True)
)
return [0]
def batch_get_into(self, keys, pointers, sizes):
results = []
for key, ptr, size in zip(keys, pointers, sizes, strict=True):
assert any(
base <= ptr and ptr + size <= base + capacity
for base, capacity in self.registered.items()
)
obj = self.objects.get(key)
if obj is None:
results.append(-704)
elif len(obj) > size:
results.append(-600)
else:
ctypes.memmove(ptr, obj, len(obj))
results.append(len(obj))
return results
@pytest.fixture
def backend():
instance = store_backend.MooncakeEmbeddingStoreBackend(
store_client.MooncakeEmbeddingStoreClient(
MemoryStore(), read_buffer_bytes=1024
),
KEY,
max_pending_items=32,
max_pending_bytes=2 * 1024**3,
)
yield instance
# Some tests deliberately poison I/O; always stop the Python executor.
instance._executor.shutdown(wait=True)
if not instance.store_client._poisoned and (
instance.store_client._read_buffer is not None
or instance.store_client._put_header is not None
):
instance.store_client.close()
def test_tensor_round_trip(backend):
client, native = backend.store_client, backend.store_client.store
native.register_buffer = MagicMock(wraps=native.register_buffer)
header_ptr = None
for index, dtype in enumerate((torch.float16, torch.bfloat16, torch.float32)):
tensor = torch.arange(6, dtype=dtype).reshape(index + 1, -1)
key = make_embedding_key(KEY, str(dtype))
client.put_tensor(key, tensor)
pointer = ctypes.addressof(client._put_header)
header_ptr = header_ptr or pointer
assert pointer == header_ptr
expected = TensorSpec(tuple(tensor.shape), str(dtype), tensor.nbytes)
with torch.device("meta"):
loaded = client.load_tensors({key: expected}, "cpu")[key]
assert client._read_buffer.device.type == "cpu"
assert loaded.dtype == dtype and torch.equal(loaded, tensor)
assert len(native.objects[key]) == 24 + tensor.nbytes
assert (
sum(c.args[0] == header_ptr for c in native.register_buffer.call_args_list) == 1
)
assert len(native.registered) == 2
@pytest.mark.parametrize("capacity", [80, 120])
def test_batch_reads_reuse_registration_without_aliasing_outputs(backend, capacity):
client, native = backend.store_client, backend.store_client.store
client._read_buffer_bytes = capacity
tensors = {key: torch.full((2, 2), float(i)) for i, key in enumerate("abc")}
for key, tensor in tensors.items():
client.put_tensor(key, tensor)
native.register_buffer = MagicMock(wraps=native.register_buffer)
native.unregister_buffer = MagicMock(wraps=native.unregister_buffer)
native.batch_get_into = MagicMock(wraps=native.batch_get_into)
expected = dict.fromkeys(tensors, SPEC)
first = client.load_tensors(expected, "cpu")
for key, tensor in tensors.items():
native.objects[key] = native.objects[key][:24] + struct.pack(
"<4f", *[tensor[0, 0].item() + 10] * 4
)
second = client.load_tensors(expected, "cpu")
assert all(torch.equal(first[key], tensor) for key, tensor in tensors.items())
assert all(torch.equal(second[key], tensor + 10) for key, tensor in tensors.items())
assert native.batch_get_into.call_count == (4 if capacity == 80 else 2)
native.register_buffer.assert_called_once()
native.unregister_buffer.assert_not_called()
client.close()
assert native.unregister_buffer.call_count == 2
assert not native.registered
@pytest.mark.parametrize("mode", ["miss", "oversized"])
def test_unloadable_inputs_do_not_allocate_staging(backend, mode):
client, native = backend.store_client, backend.store_client.store
client._read_buffer_bytes = 39
if mode == "oversized":
client.put_tensor(make_embedding_key(KEY, "a"), torch.ones(2, 2))
native.register_buffer = MagicMock(wraps=native.register_buffer)
assert not backend.resolve_inputs({"a": SPEC}, {}, "cpu")
assert client._read_buffer is None
if mode == "miss":
assert client._put_header is None
client.close()
native.register_buffer.assert_not_called()
@pytest.mark.parametrize(
"results", [[40, -800], [40], [40, True], [[40], [40]], None, [40, 41]]
)
def test_unsafe_batch_result_retains_whole_staging(backend, results):
client, native = backend.store_client, backend.store_client.store
native.batch_get_into = MagicMock(return_value=results)
native.unregister_buffer = MagicMock(wraps=native.unregister_buffer)
for _ in range(2):
with pytest.raises(store_client.EmbeddingStoreError, match="unconfirmed"):
client.load_tensors(dict.fromkeys("ab", SPEC), "cpu")
native.batch_get_into.assert_called_once()
with pytest.raises(store_client.EmbeddingStoreError, match="unconfirmed"):
client.close()
native.unregister_buffer.assert_not_called()
assert client._read_buffer is client._unsafe_owners[0]
@pytest.mark.parametrize("failure", ["allocation", "copy", "record", "completion"])
def test_copy_failure_preserves_staging_lifetime(backend, failure):
"""Emulate CUDA allocation/event boundaries without requiring a GPU."""
client = backend.store_client
for key in "ab":
client.put_tensor(key, torch.ones(2, 2))
allocate, copy = torch.empty, torch.Tensor.copy_
targets: list[torch.Tensor] = []
copies: list[torch.Tensor] = []
ready = MagicMock()
if failure == "record":
ready.record.side_effect = RuntimeError("record failure")
elif failure == "completion":
ready.synchronize.side_effect = RuntimeError("copy failure")
def allocate_on_cpu(*args, **kwargs):
is_target = kwargs.get("device") == torch.device("cuda")
if is_target and targets and failure == "allocation":
raise torch.OutOfMemoryError("allocation failure")
kwargs.update(device="cpu", pin_memory=False)
tensor = allocate(*args, **kwargs)
if is_target:
targets.append(tensor)
return tensor
def record_copy(target, source, **kwargs):
if copies and failure == "copy":
raise RuntimeError("copy failure")
copies.append(target)
return copy(target, source, **kwargs)
with (
patch.object(torch, "empty", side_effect=allocate_on_cpu),
patch.object(torch, "Event", return_value=ready),
patch.object(torch.accelerator, "current_stream"),
patch.object(torch.Tensor, "copy_", new=record_copy),
pytest.raises(RuntimeError, match="allocation failure|unconfirmed"),
):
client.load_tensors(dict.fromkeys("ab", SPEC), "cuda")
if failure == "allocation":
assert not copies and not client._poisoned
else:
assert len(copies) == (1 if failure == "copy" else 2)
assert client._poisoned
assert any(owner is client._read_buffer for owner in client._unsafe_owners)
assert any(owner is ready for owner in client._unsafe_owners)
assert all(
any(owner is target for owner in client._unsafe_owners)
for target in targets
)
with pytest.raises(store_client.EmbeddingStoreError, match="unconfirmed"):
client.close()
def test_non_matrix_output_is_rejected_before_registration(backend):
client = backend.store_client
client.store.register_buffer = MagicMock()
with pytest.raises(store_client.EmbeddingStoreOperationError, match="2D"):
client.put_tensor(make_embedding_key(KEY, "a"), torch.ones(2, 2, 2))
client.store.register_buffer.assert_not_called()
def make_worker(backend):
from vllm.distributed.ec_transfer.ec_connector.mooncake.worker import (
ECMooncakeWorker,
)
worker = object.__new__(ECMooncakeWorker)
worker.is_producer = True
worker.is_consumer = False
worker._output_store = backend
worker._buffer_device = "cpu"
worker._producer_pushes = MagicMock()
worker._producer_pushes.reserve.return_value = (MagicMock(), True)
worker._producer_pushes.poll.return_value = []
worker._producer_pushes.cancel_requests.return_value = []
worker._producer_pushes.pending = False
worker._control_executor = MagicMock()
worker._consumer_memory = MagicMock()
worker._consumer_memory.drain_reclaimed.return_value = set()
worker._completed_loads = set()
worker._failed_loads = set()
worker._failed_saves = set()
worker._push_ready = threading.Event()
worker._flush_pending_pushes = MagicMock()
worker._bind_push_source = MagicMock()
connector = object.__new__(ECMooncakeConnector)
connector._worker = worker
connector._role = ECConnectorRole.WORKER
connector._is_producer = True
connector._is_consumer = False
connector._connector_metadata = None
return connector
@pytest.mark.parametrize(
"mode",
[
"miss",
"legacy",
"hit",
"partial",
"get-failure",
"short-data",
"lease-expired",
"registration-failure",
],
)
def test_reuse_pipeline(backend, mode):
from vllm.multimodal.inputs import (
MultiModalFeatureSpec,
MultiModalKwargsItem,
PlaceholderRange,
)
from vllm.v1.worker.gpu.ec_connector import ActiveECConnector
from vllm.v1.worker.gpu.mm.encoder_cache import EncoderCache
from vllm.v1.worker.gpu.mm.encoder_runner import EncoderRunner
from vllm.v1.worker.gpu.model_states.interface import ModelState
native = backend.store_client.store
values = {"a": 1.0, "b": 2.0, "c": 3.0}
if mode == "legacy":
old_namespace = KEY.replace("@protocol:v3", "@protocol:v2")
for key in "abc":
native.objects[make_embedding_key(old_namespace, key)] = b"old format"
if mode == "partial":
hits = {"a", "c"}
elif mode not in ("miss", "legacy"):
hits = set("abc")
else:
hits = set()
for key in hits:
backend.store_client.put_tensor(
make_embedding_key(KEY, key), torch.full((2, 2), values[key])
)
if mode == "short-data":
key = make_embedding_key(KEY, "b")
native.objects[key] = native.objects[key][:-1]
hits.remove("b")
if mode in ("get-failure", "lease-expired"):
read = native.batch_get_into
def failed_read(keys, pointers, sizes):
results = read(keys, pointers, sizes)
results[keys.index(make_embedding_key(KEY, "b"))] = (
-707 if mode == "lease-expired" else -704
)
return results
native.batch_get_into = failed_read
hits.remove("b")
if mode == "registration-failure":
native.register_buffer = MagicMock(return_value=-500)
hits.clear()
connector = make_worker(backend)
metadata = ECMooncakeConnectorMetadata(
store_candidates=dict.fromkeys("abc", SPEC),
pushes=[
ECMooncakePushSpec(
mm_hash=k,
shape=(2, 2),
dtype="float32",
nbytes=16,
consumer_zmq="ipc:///tmp/test",
transfer_id=k,
request_id="req",
)
for k in "abc"
],
)
cache = EncoderCache()
cache.mm_features["req"] = [
MultiModalFeatureSpec(
data=MultiModalKwargsItem({}),
modality="image",
identifier=k,
mm_position=PlaceholderRange(offset=i * 2, length=2),
)
for i, k in enumerate("abc")
]
runner = EncoderRunner(
model=None,
max_num_tokens=8,
hidden_size=2,
encoder_cache=cache,
dtype=torch.float32,
device=torch.device("cpu"),
)
runner.enable_timing = False
missing = [k for k in "abc" if k not in hits]
runner.execute_mm_encoder = MagicMock(
return_value=[torch.full((2, 2), values[k] + 10) for k in missing]
)
state = SimpleNamespace(encoder_runner=runner, encoder_cache=cache)
with patch(
"vllm.v1.worker.gpu.ec_connector.get_ec_transfer", return_value=connector
):
adapter = ActiveECConnector(SimpleNamespace(), cache.encoder_outputs)
with adapter.maybe_get_output(
SimpleNamespace(ec_connector_metadata=metadata, finished_req_ids=set())
) as output:
assert runner.prepare_mm_inputs({"req": [0, 1, 2]})[0] == missing
ModelState.execute_mm_encoder(state, {"req": [0, 1, 2]})
assert runner.execute_mm_encoder.call_count == bool(missing)
assert output.ec_connector_worker_meta.pending_saves is False
assert all(
torch.equal(
cache.encoder_outputs[k],
torch.full((2, 2), values[k] + (0 if k in hits else 10)),
)
for k in "abc"
)
assert {
c.args[1] for c in connector._worker._bind_push_source.call_args_list
} == set("abc")
backend.shutdown()
assert not native.registered
if mode in ("miss", "legacy"):
other = store_backend.MooncakeEmbeddingStoreBackend(
store_client.MooncakeEmbeddingStoreClient(native, read_buffer_bytes=1024),
KEY,
max_pending_items=32,
max_pending_bytes=2 * 1024**3,
)
try:
loaded: dict[str, torch.Tensor] = {}
assert other.resolve_inputs(
dict.fromkeys("abc", SPEC),
loaded,
"cpu",
) == set("abc")
assert all(torch.equal(loaded[k], cache.encoder_outputs[k]) for k in "abc")
finally:
other.shutdown()
@pytest.mark.parametrize(
"mismatch",
["revision", "shape", "dtype", "magic", "short-header", "short-data", "long-data"],
)
def test_incompatible_output_is_a_miss(backend, mismatch):
key = make_embedding_key(KEY, "a")
dtype = torch.float16 if mismatch == "dtype" else torch.float32
backend.store_client.put_tensor(key, torch.ones(2, 2, dtype=dtype))
if mismatch == "revision":
backend.namespace = KEY.replace("@revision:revision", "@revision:other")
objects = backend.store_client.store.objects
if mismatch == "magic":
objects[key] = b"\0" * 4 + objects[key][4:]
elif mismatch == "short-header":
objects[key] = objects[key][:23]
elif mismatch == "short-data":
objects[key] = objects[key][:-1]
elif mismatch == "long-data":
objects[key] += b"extra"
expected = replace(SPEC, shape=(4, 1)) if mismatch == "shape" else SPEC
outputs: dict[str, torch.Tensor] = {}
assert not backend.resolve_inputs({"a": expected}, outputs, "cpu")
assert not outputs
tensor = torch.ones(2, 2)
backend.store_client.put_tensor(
make_embedding_key(backend.namespace, "valid"), tensor
)
assert backend.resolve_inputs({"valid": SPEC}, outputs, "cpu") == {"valid"}
assert torch.equal(outputs["valid"], tensor)
assert len(backend.store_client.store.registered) == 2
def test_partial_registration_rejection_releases_publication(backend):
"""A rejected header registration must not prevent a later publication."""
native = backend.store_client.store
register = native.register_buffer
put = native.batch_put_from_multi_buffers
native.register_buffer = lambda addr, size: (
-500 if size == 24 else register(addr, size)
)
native.batch_put_from_multi_buffers = MagicMock()
tensor = torch.ones(2, 2)
owner = weakref.ref(tensor)
assert backend._enqueue_save("a", tensor)
storage_owner = weakref.ref(tensor.untyped_storage())
future = backend._pending["a"][0]
with pytest.raises(store_client.EmbeddingStoreOperationError, match="register"):
future.result(timeout=5)
del tensor
gc.collect()
assert owner() is None and not native.registered
native.batch_put_from_multi_buffers.assert_not_called()
backend.resolve_inputs({}, {}, "cpu")
assert backend._pending_bytes == 0
gc.collect()
assert storage_owner() is None # The failed Future is still alive.
assert backend.store_client._put_header is None
native.register_buffer = register
native.batch_put_from_multi_buffers = put
backend.store_client.put_tensor("retry", torch.ones(2, 2))
assert len(native.registered) == 1
@pytest.mark.parametrize(
"operation",
["read-unregister", "header-unregister", "put", "rejected-put", "unregister"],
)
def test_native_io_owners(backend, operation):
"""Uncertain I/O retains owners; completed rejection frees them even via Future."""
native, client = backend.store_client.store, backend.store_client
put = native.batch_put_from_multi_buffers
backend.max_pending_bytes = SPEC.nbytes
tensor = torch.ones(2, 2)
owner = weakref.ref(tensor)
rejected = operation == "rejected-put"
if operation in ("read-unregister", "header-unregister"):
key = make_embedding_key(KEY, "a")
client.put_tensor(key, tensor)
if operation == "read-unregister":
client.load_tensors({key: SPEC}, "cpu")
native.unregister_buffer = MagicMock(return_value=-500)
with pytest.raises(store_client.EmbeddingStoreError, match="unregister"):
client.close()
owner = weakref.ref(
client._read_buffer
if operation == "read-unregister"
else client._put_header
)
else:
if operation == "unregister":
native.unregister_buffer = MagicMock(return_value=-500)
else:
native.batch_put_from_multi_buffers = MagicMock(
return_value=[-1700 if rejected else -800]
)
assert backend._enqueue_save("a", tensor)
future = backend._pending["a"][0]
with pytest.raises(store_client.EmbeddingStoreError):
future.result(timeout=5)
del tensor
gc.collect()
if rejected:
assert owner() is None # A live Future must not retain the tensor.
tensor = torch.ones(4, 4) # Too large for admission after reaping.
caller_owner = weakref.ref(tensor)
backend._step_candidates = {"next"}
backend.save_output("next", tensor)
del tensor
gc.collect()
assert caller_owner() is None
assert backend._pending_bytes == 0 and len(native.registered) == 1
header = client._put_header
native.batch_put_from_multi_buffers = put
client.put_tensor("retry", torch.ones(1, 4))
assert client._put_header is header
else:
assert owner() is not None and native.registered
header = bytes(client._put_header)
with pytest.raises(store_client.EmbeddingStoreError, match="unconfirmed"):
client.put_tensor("retry", torch.ones(1, 4))
assert bytes(client._put_header) == header
with pytest.raises(store_client.EmbeddingStoreError, match="unconfirmed"):
client.close()
def test_publication_budget_charges_and_releases_backing_storage(backend):
tensor = torch.arange(16, dtype=torch.float32)[:4].view(2, 2)
owner = weakref.ref(tensor)
future: Future[None] = Future()
backend.max_pending_items = 1
backend.max_pending_bytes = tensor.nbytes
with patch.object(backend._executor, "submit", return_value=future):
assert not backend._enqueue_save("a", tensor)
backend.max_pending_bytes = 64
assert backend._enqueue_save("a", tensor)
storage_owner = weakref.ref(tensor.untyped_storage())
assert not backend._enqueue_save("b", tensor)
del tensor
gc.collect()
backend.reap() # The final step-end poll precedes publication completion.
assert owner() is not None
backend._save(make_embedding_key(KEY, "a"), backend._pending["a"][1])
future.set_result(None)
assert owner() is None and storage_owner() is not None
def load_after_reaping(*args):
assert not backend._pending and backend._pending_bytes == 0
assert storage_owner() is None
return {}
with patch.object(backend.store_client, "load_tensors", load_after_reaping):
assert not backend.resolve_inputs({"b": SPEC}, {}, "cpu")
@pytest.mark.parametrize("rejected", [False, True])
def test_shared_storage_budget_is_released_after_last_publication(backend, rejected):
batch = torch.arange(16, dtype=torch.float32).view(4, 4)
first: Future[None] = Future()
last: Future[None] = Future()
separate: Future[None] = Future()
backend.max_pending_items = 2
backend.max_pending_bytes = batch.nbytes
with patch.object(backend._executor, "submit", side_effect=[first, last, separate]):
assert backend._enqueue_save("a", batch[:2])
assert backend._enqueue_save("b", batch[2:])
assert backend._pending_bytes == batch.nbytes
assert not backend._enqueue_save("a", batch[:2])
assert not backend._enqueue_save("extra", batch[:1])
if rejected:
first.set_exception(store_client.EmbeddingStoreOperationError("rejected"))
else:
first.set_result(None)
backend.reap()
assert backend._pending_bytes == batch.nbytes
assert not backend._enqueue_save("c", torch.ones(2, 2))
last.set_result(None)
backend.reap()
assert backend._pending_bytes == 0
assert backend._enqueue_save("c", torch.ones(2, 2))
assert backend._pending_bytes == 16
separate.set_result(None)
backend.reap()
assert backend._pending_bytes == 0
@pytest.mark.parametrize("shared_storage", [False, True])
def test_completion_race_preserves_failure_and_reclaims_other_items(
backend, shared_storage
):
class RacingFuture(Future[None]):
def done(self):
observed = super().done()
if not observed:
self.set_exception(store_client.EmbeddingStoreError("uncertain"))
return observed
racing = RacingFuture()
other: Future[None] = Future()
tensor = torch.ones(2, 2)
other_tensor = tensor[:1] if shared_storage else torch.ones(2, 2)
with patch.object(backend._executor, "submit", side_effect=[racing, other]):
assert backend._enqueue_save("race", tensor)
assert backend._enqueue_save("other", other_tensor)
backend.reap()
assert backend._pending_bytes == (16 if shared_storage else 32)
other.set_result(None)
with pytest.raises(store_client.EmbeddingStoreError):
backend.reap()
assert backend._pending_bytes == 16
with pytest.raises(store_client.EmbeddingStoreError):
backend._enqueue_save("later", torch.ones(2, 2))
def test_readiness_and_shutdown_do_not_block_p2p_completion(backend):
ready, waiting, entered = [threading.Event() for _ in range(3)]
original = backend._executor.shutdown
def closing(*args, **kwargs):
entered.set()
return original(*args, **kwargs)
native = backend.store_client.store
def close_store():
assert ready.is_set() and not native.registered
assert backend._pending_bytes == 0
return 0
native.close = MagicMock(side_effect=close_store)
put = native.batch_put_from_multi_buffers
def waiting_put(*args):
waiting.set()
assert ready.wait(5)
assert backend.store_client._put_header is not None and native.registered
return put(*args)
native.batch_put_from_multi_buffers = waiting_put
connector = make_worker(backend)
connector.bind_connector_metadata(
ECMooncakeConnectorMetadata(store_candidates={"a": SPEC})
)
cache: dict[str, torch.Tensor] = {}
with (
patch.object(
store_backend, "_record_tensor_ready_event", return_value=MagicMock()
),
patch.object(backend._executor, "shutdown", side_effect=closing),
ThreadPoolExecutor(max_workers=1) as closer,
):
connector.start_save_caches(encoder_cache=cache)
cache["a"] = torch.arange(4, dtype=torch.float32).view(2, 2)
connector.save_caches(cache, "a")
assert connector.build_connector_worker_meta().pending_saves is False
assert waiting.wait(5) and not backend.store_client.store.objects
close = closer.submit(backend.shutdown)
try:
assert entered.wait(5) and not close.done()
assert not backend._enqueue_save("late", torch.ones(2, 2))
finally:
ready.set()
close.result(timeout=5)
native.close.assert_called_once_with()
assert not backend.store_client.store.registered and backend._pending_bytes == 0
assert backend.store_client.store.objects[make_embedding_key(KEY, "a")][
24:
] == struct.pack("<4f", 0, 1, 2, 3)
@@ -17,6 +17,7 @@ import pytest
import torch
from tests.v1.attention.utils import dense_kv_cache_views
from vllm.distributed import mooncake_store
from vllm.distributed.kv_events import KVEventAggregator
from vllm.distributed.kv_transfer.kv_connector.v1.mooncake import (
rdma_utils,
@@ -402,6 +403,7 @@ def _install_fake_mooncake(monkeypatch, store_instance: MagicMock):
fake_store_module = types.ModuleType("mooncake.store")
fake_store_module.MooncakeDistributedStore = lambda: store_instance # type: ignore[attr-defined]
fake_store_module.ReplicateConfig = FakeReplicateConfig # type: ignore[attr-defined]
fake_store_module.ObjectDataType = SimpleNamespace(TENSOR=1) # type: ignore[attr-defined]
fake_mooncake_module = types.ModuleType("mooncake")
fake_mooncake_module.store = fake_store_module # type: ignore[attr-defined]
monkeypatch.setitem(sys.modules, "mooncake", fake_mooncake_module)
@@ -590,7 +592,7 @@ def test_pool_key_cache_prefix_namespaces_and_disambiguates():
def test_default_local_buffer_size_matches_pr40900():
"""PR-40900 shipped a 4 GiB default for local_buffer_size; the dual-mode
patch preserves it (and the JSON key) so unchanged PR-40900 configs work."""
assert worker.DEFAULT_LOCAL_BUFFER_SIZE == 4 * 1024**3
assert mooncake_store.DEFAULT_LOCAL_BUFFER_SIZE == 4 * 1024**3
def test_get_requester_local_hostname_prefers_override(monkeypatch):
@@ -2957,6 +2959,7 @@ def test_requester_worker_group_semantics_string_true_enables(
fake_store_module = types.ModuleType("mooncake.store")
fake_store_module.MooncakeDistributedStore = lambda: store # type: ignore[attr-defined]
fake_store_module.ReplicateConfig = FakeReplicateConfig # type: ignore[attr-defined]
fake_store_module.ObjectDataType = SimpleNamespace(TENSOR=1) # type: ignore[attr-defined]
fake_mooncake_module = types.ModuleType("mooncake")
fake_mooncake_module.store = fake_store_module # type: ignore[attr-defined]
monkeypatch.setitem(sys.modules, "mooncake", fake_mooncake_module)
@@ -4377,10 +4380,10 @@ def test_config_defaults_to_embedded():
"""A JSON without explicit mode parses as embedded with 4 GiB segment."""
cfg = _make_config()
assert cfg.mode == "embedded"
assert cfg.global_segment_size == worker.DEFAULT_GLOBAL_SEGMENT_SIZE
assert cfg.local_buffer_size == worker.DEFAULT_LOCAL_BUFFER_SIZE
assert cfg.global_segment_size == mooncake_store.DEFAULT_GLOBAL_SEGMENT_SIZE
assert cfg.local_buffer_size == mooncake_store.DEFAULT_LOCAL_BUFFER_SIZE
assert cfg.enable_offload is False
assert cfg.tenant_id == worker.DEFAULT_TENANT_ID
assert cfg.tenant_id == mooncake_store.DEFAULT_TENANT_ID
def test_config_pr40900_unchanged(tmp_path):
@@ -4402,7 +4405,7 @@ def test_config_pr40900_unchanged(tmp_path):
assert cfg.global_segment_size == 4 * 1024**3
assert cfg.local_buffer_size == 4 * 1024**3
assert cfg.enable_offload is False
assert cfg.tenant_id == worker.DEFAULT_TENANT_ID
assert cfg.tenant_id == mooncake_store.DEFAULT_TENANT_ID
def test_config_from_file_normalizes_tenant_id(tmp_path):
@@ -4432,7 +4435,7 @@ def test_config_from_file_normalizes_empty_tenant_id_to_default(tmp_path):
cfg = worker.MooncakeStoreConfig.from_file(config_path)
assert cfg.tenant_id == worker.DEFAULT_TENANT_ID
assert cfg.tenant_id == mooncake_store.DEFAULT_TENANT_ID
def test_config_from_file_rejects_non_string_tenant_id(tmp_path):
@@ -4584,7 +4587,20 @@ def test_topology_embedded_cpu_only(tmp_path, monkeypatch):
assert w.disk_offload_buffer_budget_bytes is None
def test_topology_forwards_non_default_tenant_id(tmp_path, monkeypatch):
def _create_store_client_for_tenant_test(consumer):
if consumer == "ec":
from vllm.distributed.ec_transfer.ec_connector.mooncake_store_embedding.store_client import ( # noqa: E501
create_mooncake_embedding_store_client,
)
return create_mooncake_embedding_store_client()
return worker.MooncakeStoreWorker(
_make_vllm_config(rank=1), _make_kv_cache_config()
)
@pytest.mark.parametrize("consumer", ["kv", "ec"])
def test_topology_forwards_non_default_tenant_id(tmp_path, monkeypatch, consumer):
store = MagicMock()
store.setup.return_value = 0
_install_fake_mooncake(monkeypatch, store)
@@ -4605,12 +4621,13 @@ def test_topology_forwards_non_default_tenant_id(tmp_path, monkeypatch):
),
)
worker.MooncakeStoreWorker(_make_vllm_config(rank=1), _make_kv_cache_config())
_create_store_client_for_tenant_test(consumer)
assert store.setup.call_args.kwargs == {"tenant_id": "tenant-a"}
def test_non_default_tenant_preserves_setup_type_error(tmp_path, monkeypatch):
@pytest.mark.parametrize("consumer", ["kv", "ec"])
def test_non_default_tenant_preserves_setup_type_error(tmp_path, monkeypatch, consumer):
store = MagicMock()
setup_error = TypeError(
"setup(): incompatible function arguments; "
@@ -4636,7 +4653,7 @@ def test_non_default_tenant_preserves_setup_type_error(tmp_path, monkeypatch):
)
with pytest.raises(TypeError) as exc_info:
worker.MooncakeStoreWorker(_make_vllm_config(rank=1), _make_kv_cache_config())
_create_store_client_for_tenant_test(consumer)
assert exc_info.value is setup_error
+2
View File
@@ -23,6 +23,8 @@ class ECTransferConfig:
safetensors) and ``ECMooncakeConnector`` (Mooncake TransferEngine RDMA;
requires ``mooncake-transfer-engine`` and matching producer/consumer
``ec_connector_extra_config``; see ``mooncake_ec_connector`` module docstring).
Set ``cross_encoder_cache`` in Mooncake extra config to reuse shared
Encoder outputs from Store before encoding, retaining P2P delivery.
"""
engine_id: str | None = None
@@ -69,6 +69,10 @@ class MooncakeECConfig:
control_timeout_ms: int
push_wait_timeout_s: float
pool_size: int
cross_encoder_cache: bool = False
store_max_pending_items: int = 32
store_max_pending_bytes: int = 2 * 1024**3
store_read_buffer_bytes: int = 128 * 1024**2
@classmethod
def from_vllm_config(cls, vllm_config: VllmConfig) -> MooncakeECConfig:
@@ -94,6 +98,24 @@ class MooncakeECConfig:
"ec_buffer_size", ec_config.ec_buffer_size
)
get = ec_config.get_from_extra_config
shared_reuse = get("cross_encoder_cache", False)
if not isinstance(shared_reuse, bool):
raise ValueError("cross_encoder_cache must be a boolean")
if shared_reuse:
if not ec_config.is_encode_only:
raise ValueError(
"cross_encoder_cache requires an Encoder-only producer"
)
if not vllm_config.use_v2_model_runner:
raise ValueError("cross_encoder_cache requires Model Runner V2")
if vllm_config.lora_config is not None:
raise ValueError("cross_encoder_cache does not support LoRA")
mm_config = vllm_config.model_config.multimodal_config
if mm_config is not None and mm_config.mm_processor_cache_gb == 0:
raise ValueError(
"cross_encoder_cache requires mm_processor_cache_gb > 0 "
"to preserve content identifiers across Encoders and restarts"
)
control_port = int(ec_config.ec_port) + (
parallel_config.data_parallel_index * parallel_config.tensor_parallel_size
)
@@ -120,4 +142,14 @@ class MooncakeECConfig:
"push_wait_timeout_s", get("push_wait_timeout_s", 60)
),
pool_size=registered_buffer_size,
cross_encoder_cache=shared_reuse,
store_max_pending_items=_positive_int(
"store_max_pending_items", get("store_max_pending_items", 32)
),
store_max_pending_bytes=_positive_int(
"store_max_pending_bytes", get("store_max_pending_bytes", 2 * 1024**3)
),
store_read_buffer_bytes=_positive_int(
"store_read_buffer_bytes", get("store_read_buffer_bytes", 128 * 1024**2)
),
)
@@ -10,6 +10,9 @@ from vllm.distributed.ec_transfer.ec_connector.base import (
ECConnectorMetadata,
ECConnectorWorkerMetadata,
)
from vllm.distributed.ec_transfer.ec_connector.mooncake_store_embedding.data import (
TensorSpec,
)
@dataclass
@@ -46,6 +49,7 @@ class ECMooncakeConnectorMetadata(ECConnectorMetadata):
loads: list[ECMooncakeLoadSpec] = field(default_factory=list)
pushes: list[ECMooncakePushSpec] = field(default_factory=list)
freed: list[str] | None = None
store_candidates: dict[str, TensorSpec] = field(default_factory=dict)
@dataclass
@@ -45,6 +45,9 @@ from vllm.distributed.ec_transfer.ec_connector.mooncake.state import (
from vllm.distributed.ec_transfer.ec_connector.mooncake.transfer import (
ensure_mooncake_available,
)
from vllm.distributed.ec_transfer.ec_connector.mooncake_store_embedding.data import (
TensorSpec,
)
from vllm.distributed.ec_transfer.ec_connector.utils import (
PlaceholderMetadataResolver,
collect_ec_item_metadata,
@@ -76,6 +79,8 @@ class ECMooncakeScheduler:
ensure_mooncake_available()
config = MooncakeECConfig.from_vllm_config(vllm_config)
self._store_candidates: list[tuple[str, str, TensorSpec]] = []
self._cross_encoder_cache = config.cross_encoder_cache
self._is_producer = config.is_producer
self._is_consumer = config.is_consumer
self._control_addr = config.control_addr
@@ -427,28 +432,40 @@ class ECMooncakeScheduler:
transfer_id = f"{request.request_id}:{index}"
if not consumer_zmq or transfer_id in self._prepared_push_transfer_ids:
return
num_tokens = request.get_num_encoder_embeds(index)
dtype = self._model_config.dtype
assert isinstance(dtype, torch.dtype)
assert self._encoder_cache_hidden_dim is not None
dtype_name = str(dtype).split(".")[-1]
shape = (num_tokens, self._encoder_cache_hidden_dim)
nbytes = math.prod(shape) * dtype.itemsize
output = self._encoder_output_spec(request, index)
self._pushes_to_prepare[transfer_id] = ECMooncakePushSpec(
mm_hash=mm_hash,
nbytes=nbytes,
shape=shape,
dtype=dtype_name,
nbytes=output.nbytes,
shape=output.shape,
dtype=output.dtype.removeprefix("torch."),
consumer_zmq=str(consumer_zmq),
transfer_id=transfer_id,
request_id=request.request_id,
)
self._prepared_push_transfer_ids.add(transfer_id)
def _encoder_output_spec(self, request: Any, index: int) -> TensorSpec:
dtype = self._model_config.dtype
assert isinstance(dtype, torch.dtype)
assert self._encoder_cache_hidden_dim is not None
shape = (request.get_num_encoder_embeds(index), self._encoder_cache_hidden_dim)
return TensorSpec(shape, str(dtype), math.prod(shape) * dtype.itemsize)
def update_state_after_alloc(self, request: Any, index: int) -> None:
self._local_cache.add(request.mm_features[index].identifier)
if self._is_producer:
self._prepare_push_spec(request, index)
if (
self._cross_encoder_cache
and request.mm_features[index].modality == "image"
):
self._store_candidates.append(
(
request.request_id,
request.mm_features[index].identifier,
self._encoder_output_spec(request, index),
)
)
def update_state_after_free(self, request: Any, index: int) -> None:
if not self._is_consumer:
@@ -474,6 +491,13 @@ class ECMooncakeScheduler:
meta = ECMooncakeConnectorMetadata(
freed=scheduler_output.free_encoder_mm_hashes
)
preempted = scheduler_output.preempted_req_ids or set()
meta.store_candidates = {
identifier: output
for request_id, identifier, output in self._store_candidates
if request_id not in preempted
}
self._store_candidates.clear()
for push_spec in self._pushes_to_prepare.values():
meta.pushes.append(push_spec)
self._pushes_to_prepare.clear()
@@ -49,6 +49,9 @@ from vllm.distributed.ec_transfer.ec_connector.mooncake.transfer import (
MooncakeTransfer,
ensure_mooncake_available,
)
from vllm.distributed.ec_transfer.ec_connector.mooncake_store_embedding.data import (
build_embedding_namespace,
)
from vllm.logger import init_logger
from vllm.utils.network_utils import get_ip
@@ -59,6 +62,10 @@ _T = TypeVar("_T")
if TYPE_CHECKING:
from vllm.config import VllmConfig
from ..mooncake_store_embedding.backend import (
MooncakeEmbeddingStoreBackend,
)
_RESERVATION_REFRESH_SECONDS = _RESERVATION_TTL_SECONDS / 2
_MAX_CANCELLED_TRANSFER_IDS = 1 << 16
_CANCEL_ATTEMPTS = 2
@@ -87,6 +94,17 @@ class ECMooncakeWorker:
def __init__(self, vllm_config: VllmConfig) -> None:
ensure_mooncake_available()
config = MooncakeECConfig.from_vllm_config(vllm_config)
self._store_config = (
(
build_embedding_namespace(vllm_config),
config.store_max_pending_items,
config.store_max_pending_bytes,
config.store_read_buffer_bytes,
)
if config.cross_encoder_cache
else None
)
self._output_store: MooncakeEmbeddingStoreBackend | None = None
self.is_producer = config.is_producer
self.is_consumer = config.is_consumer
self._buffer_device = config.buffer_device
@@ -173,6 +191,21 @@ class ECMooncakeWorker:
self._is_receiving_rank = True
def start_services(self) -> None:
if self._store_config is not None and self._output_store is None:
from ..mooncake_store_embedding.backend import (
MooncakeEmbeddingStoreBackend,
)
from ..mooncake_store_embedding.store_client import (
create_mooncake_embedding_store_client,
)
namespace, max_items, max_bytes, read_bytes = self._store_config
self._output_store = MooncakeEmbeddingStoreBackend(
create_mooncake_embedding_store_client(read_buffer_bytes=read_bytes),
namespace,
max_pending_items=max_items,
max_pending_bytes=max_bytes,
)
if not self.is_consumer or self._control_server is not None:
return
self._resolve_consumer_rank()
@@ -470,6 +503,14 @@ class ECMooncakeWorker:
tensor = encoder_cache.get(mm_hash)
if tensor is not None:
self._bind_push_source(tensor, mm_hash)
if self._output_store is not None:
loaded = self._output_store.resolve_inputs(
metadata.store_candidates, encoder_cache, self._buffer_device
)
# Loaded entries precede the runner's new-output snapshot, so they
# need explicit P2P source binding here.
for mm_hash in loaded:
self._bind_push_source(encoder_cache[mm_hash], mm_hash)
def _reserve_batch(self, records: list[ProducerPushRecord]) -> None:
"""Resolve independent item futures with one reserve RPC per shard."""
@@ -851,6 +892,8 @@ class ECMooncakeWorker:
if not self.is_producer:
return None, None
if self._output_store is not None:
self._output_store.reap()
for record in self._producer_pushes.cancel_requests(finished_req_ids):
self._producer_pushes.submit_cancel(
record,
@@ -866,6 +909,8 @@ class ECMooncakeWorker:
return
tensor = encoder_cache[mm_hash]
self._bind_push_source(tensor, mm_hash)
if self._output_store is not None:
self._output_store.save_output(mm_hash, tensor)
def build_connector_worker_meta(self) -> ECMooncakeWorkerMetadata | None:
if self.is_consumer and not self._is_receiving_rank:
@@ -938,3 +983,6 @@ class ECMooncakeWorker:
)
self._producer_memory.close()
self._transfer.close()
if self._output_store is not None:
self._output_store.shutdown()
self._output_store = None
@@ -0,0 +1,3 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Mooncake Store support for cross-Encoder output reuse."""
@@ -0,0 +1,214 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Encoder-side Mooncake Store resolver and publisher."""
from __future__ import annotations
import threading
import traceback
from collections.abc import Mapping
from concurrent.futures import Future, ThreadPoolExecutor
from dataclasses import dataclass
import torch
from vllm.logger import init_logger
from .data import TensorSpec, make_embedding_key
from .store_client import (
EmbeddingStoreError,
EmbeddingStoreOperationError,
MooncakeEmbeddingStoreClient,
)
logger = init_logger(__name__)
@dataclass
class _StoreSave:
tensor: torch.Tensor | None
ready_event: torch.Event | None
storage_ptr: int
def release(self) -> None:
self.tensor = None
self.ready_event = None
class MooncakeEmbeddingStoreBackend:
"""Resolve immutable encoder outputs and publish misses asynchronously."""
def __init__(
self,
store_client: MooncakeEmbeddingStoreClient,
namespace: str,
*,
max_pending_items: int,
max_pending_bytes: int,
) -> None:
self.store_client = store_client
self.namespace = namespace
self.max_pending_items = max_pending_items
self.max_pending_bytes = max_pending_bytes
self._executor = ThreadPoolExecutor(
max_workers=1,
thread_name_prefix="ec-mooncake-store",
)
self._pending: dict[str, tuple[Future[None], _StoreSave]] = {}
self._pending_storages: dict[int, tuple[torch.UntypedStorage, int]] = {}
self._pending_lock = threading.Lock()
self._pending_bytes = 0
self._step_candidates: set[str] = set()
self._closed = False
self._fatal_error: BaseException | None = None
def save_output(self, identifier: str, tensor: torch.Tensor) -> None:
if identifier in self._step_candidates:
self.reap()
self._enqueue_save(identifier, tensor)
def resolve_inputs(
self,
expected_tensors: Mapping[str, TensorSpec],
encoder_cache: dict[str, torch.Tensor],
device: torch.device | str,
) -> set[str]:
"""Load outputs before encoding; unresolved items retain normal computation."""
self.reap()
self._step_candidates = set(expected_tensors)
candidates = [key for key in expected_tensors if key not in encoder_cache]
if not candidates:
return set()
pool_keys = [
make_embedding_key(self.namespace, identifier) for identifier in candidates
]
try:
exists = self.store_client.batch_exists(pool_keys)
except (EmbeddingStoreOperationError, OSError):
logger.warning(
"Mooncake embedding Store lookup failed; falling back to encoder",
exc_info=True,
)
return set()
hits = {
pool_key: expected_tensors[identifier]
for identifier, pool_key, hit in zip(
candidates, pool_keys, exists, strict=True
)
if hit
}
try:
tensors = self.store_client.load_tensors(hits, device)
except (EmbeddingStoreOperationError, OSError):
logger.warning(
"Mooncake embedding Store GET failed; falling back to encoder",
exc_info=True,
)
return set()
loaded: set[str] = set()
for identifier, pool_key in zip(candidates, pool_keys, strict=True):
if pool_key in tensors:
encoder_cache[identifier] = tensors[pool_key]
loaded.add(identifier)
return loaded
def _enqueue_save(self, identifier: str, tensor: torch.Tensor) -> bool:
"""Admit contiguous outputs, charging each retained storage once."""
if not tensor.is_contiguous():
return False
storage = tensor.untyped_storage()
storage_ptr = storage.data_ptr()
with self._pending_lock:
if self._fatal_error is not None:
raise EmbeddingStoreError(
"Store publisher has failed"
) from self._fatal_error
if self._closed or identifier in self._pending:
return False
_, ref_count = self._pending_storages.get(storage_ptr, (storage, 0))
budget_bytes = 0 if ref_count else storage.nbytes()
if (
len(self._pending) >= self.max_pending_items
or self._pending_bytes + budget_bytes > self.max_pending_bytes
):
return False
# Only admitted work creates an event. Runtime errors propagate.
save = _StoreSave(
tensor,
_record_tensor_ready_event(tensor),
storage_ptr,
)
future = self._executor.submit(
self._save, make_embedding_key(self.namespace, identifier), save
)
self._pending[identifier] = (future, save)
# Keep the storage alive until reaping so its address cannot be reused.
self._pending_storages[storage_ptr] = (storage, ref_count + 1)
self._pending_bytes += budget_bytes
return True
def _save(self, pool_key: str, save: _StoreSave) -> None:
try:
if self.store_client.exists(pool_key):
return
tensor = save.tensor
assert tensor is not None
if save.ready_event is not None:
save.ready_event.synchronize()
self.store_client.put_tensor(pool_key, tensor)
except BaseException as error:
# Returned client frames can otherwise keep tensors via a Future's
# traceback. Uncertain I/O owners remain retained by the client.
traceback.clear_frames(error.__traceback__)
raise
finally:
tensor = None
save.release()
def reap(self) -> None:
with self._pending_lock:
for identifier, (future, save) in list(self._pending.items()):
# Observe once; a racing completion is handled on the next poll.
if not future.done():
continue
try:
future.result() # Already done: never waits on Store I/O.
except (EmbeddingStoreOperationError, OSError) as error:
logger.warning("Store PUT failed for %s: %s", identifier, error)
# This failure is consumed; do not retain publisher/caller frames.
error.__traceback__ = None
except BaseException as error:
# The client may still own unsafe native-I/O buffers. Keep
# their charge and fatal result until worker termination.
if self._fatal_error is None:
self._fatal_error = error
continue
del self._pending[identifier]
storage, ref_count = self._pending_storages[save.storage_ptr]
if ref_count == 1:
del self._pending_storages[save.storage_ptr]
self._pending_bytes -= storage.nbytes()
else:
self._pending_storages[save.storage_ptr] = (storage, ref_count - 1)
fatal = self._fatal_error
if fatal is not None:
raise fatal
def shutdown(self) -> None:
with self._pending_lock:
if self._closed:
return
self._closed = True
self._executor.shutdown(wait=True, cancel_futures=False)
self.reap()
self.store_client.close()
def _record_tensor_ready_event(tensor: torch.Tensor) -> torch.Event | None:
if tensor.device.type != "cuda":
return None
event = torch.Event()
event.record(torch.accelerator.current_stream(tensor.device))
return event
@@ -0,0 +1,51 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Content identities and output contracts for the shared Encoder Store."""
from __future__ import annotations
from dataclasses import dataclass
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from vllm.config import VllmConfig
EMBEDDING_CACHE_KEY_VERSION = "v3"
def make_embedding_key(namespace: str, identifier: str) -> str:
return f"{namespace}@id:{identifier}"
@dataclass(frozen=True)
class TensorSpec:
"""Contiguous output contract used by the Encoder and Store codec."""
shape: tuple[int, ...]
dtype: str
nbytes: int
def build_embedding_namespace(vllm_config: VllmConfig) -> str:
model_config = vllm_config.model_config
ec_config = vllm_config.ec_transfer_config
assert ec_config is not None
extra_config = ec_config.ec_connector_extra_config
multimodal_config = model_config.multimodal_config
encoder_hash = (
multimodal_config.compute_hash()
if multimodal_config is not None
else "encoder:default"
)
cache_prefix = str(extra_config.get("embedding_cache_prefix", ""))
model_name = str(extra_config.get("embedding_model_identity", model_config.model))
prefix = f"{cache_prefix}@" if cache_prefix else ""
return (
f"{prefix}embedding"
f"@model:{model_name}"
f"@revision:{model_config.revision or 'default'}"
f"@encoder:{encoder_hash}"
f"@dtype:{model_config.dtype}"
f"@protocol:{EMBEDDING_CACHE_KEY_VERSION}"
)
@@ -0,0 +1,463 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Thin Mooncake Store client for embedding objects.
Originally adapted from vLLM PR #47302 (0d1f71f5d36c). The v3 key namespace
uses a compact header for complete 2D embeddings, not that PR's tensor format.
"""
from __future__ import annotations
import ctypes
import struct
import threading
from collections.abc import Callable, Iterator, Mapping
from contextlib import contextmanager
from enum import IntEnum
from typing import Any
import torch
from vllm.distributed.ec_transfer.ec_connector.mooncake_store_embedding.data import (
TensorSpec,
)
from vllm.distributed.mooncake_store import MooncakeStoreConfig, setup_mooncake_store
from vllm.logger import init_logger
from vllm.utils.network_utils import get_ip
logger = init_logger(__name__)
# A failed transfer can still use registered memory after the binding returns.
# Poisoned clients retain their owners until process exit, including if normal
# worker teardown drops the connector. New operations are rejected.
_UNSAFE_CLIENTS: list[Any] = []
# Values from Mooncake v0.3.12 / v0.3.12.post1 types.h (ErrorCode).
# Python exposes integer returns, not this enum.
class _MooncakeErrorCode(IntEnum):
NO_AVAILABLE_HANDLE = -200
INVALID_PARAMS = -600
INVALID_REPLICA = -702
REPLICA_IS_NOT_READY = -703
OBJECT_NOT_FOUND = -704
OBJECT_ALREADY_EXISTS = -705
LEASE_EXPIRED = -707
RPC_FAIL = -900
RPC_TIMEOUT = -901
TENANT_QUOTA_EXCEEDED = -1700
# For RAM batch reads and single-replica writes, these rejections occur
# before submission or after completed I/O. Transfer timeouts can leave I/O
# in flight; all other failures retain registered buffers and poison the client.
_SAFE_IO_REJECTIONS = frozenset(
{
_MooncakeErrorCode.NO_AVAILABLE_HANDLE,
_MooncakeErrorCode.REPLICA_IS_NOT_READY,
_MooncakeErrorCode.OBJECT_NOT_FOUND,
_MooncakeErrorCode.OBJECT_ALREADY_EXISTS,
_MooncakeErrorCode.RPC_FAIL,
_MooncakeErrorCode.RPC_TIMEOUT,
}
)
# Batch GET validates capacity/replicas before I/O and checks leases afterwards.
# PUT rejects tenant quotas before allocating replicas or submitting transfers.
_SAFE_GET_REJECTIONS = _SAFE_IO_REJECTIONS | {
_MooncakeErrorCode.INVALID_PARAMS,
_MooncakeErrorCode.INVALID_REPLICA,
_MooncakeErrorCode.LEASE_EXPIRED,
}
_SAFE_PUT_REJECTIONS = _SAFE_IO_REJECTIONS | {_MooncakeErrorCode.TENANT_QUOTA_EXCEEDED}
_MOONCAKE_TENSOR_OBJECT_MAGIC = 0x4D4F4F4E
# Magic, dtype, token count, hidden dimension. Format versioning is in the key.
_MOONCAKE_TENSOR_HEADER = struct.Struct("<IIQQ")
_MOONCAKE_DTYPE_TO_TORCH_DTYPE = {
0: "torch.float32",
11: "torch.float16",
12: "torch.bfloat16",
}
_TORCH_DTYPE_TO_MOONCAKE_DTYPE = {
value: key for key, value in _MOONCAKE_DTYPE_TO_TORCH_DTYPE.items()
}
def create_mooncake_embedding_store_client(
read_buffer_bytes: int = 128 * 1024**2,
) -> MooncakeEmbeddingStoreClient:
try:
from mooncake.store import ( # type: ignore
MooncakeDistributedStore,
ObjectDataType,
ReplicateConfig,
)
except ImportError as e:
raise ImportError(
"Install mooncake with Store support to enable cross_encoder_cache."
) from e
from vllm.distributed.kv_transfer.kv_connector.v1.mooncake import rdma_utils
config = MooncakeStoreConfig.load_from_config()
if config.enable_offload:
raise ValueError(
"cross_encoder_cache supports a RAM Store; disable enable_offload"
)
store = MooncakeDistributedStore()
local_ip = get_ip()
local_hostname = rdma_utils.get_requester_local_hostname(local_ip)
setup_mooncake_store(store, config, local_hostname)
logger.info(
"Initialized embedding Mooncake store mode=%s global_segment_size=%d "
"local_buffer_size=%d",
config.mode,
config.global_segment_size,
config.local_buffer_size,
)
replicate_config = ReplicateConfig()
replicate_config.data_type = ObjectDataType.TENSOR
return MooncakeEmbeddingStoreClient(
store, replicate_config=replicate_config, read_buffer_bytes=read_buffer_bytes
)
class EmbeddingStoreError(RuntimeError):
"""A Store failure that must propagate to the worker."""
class EmbeddingStoreOperationError(EmbeddingStoreError):
"""A rejected or completed operation that permits normal Encoder fallback."""
def _batch_get_completed_safely(results: Any, capacities: list[int]) -> bool:
return (
isinstance(results, list)
and len(results) == len(capacities)
and all(
type(result) is int
and (0 <= result <= capacity or result in _SAFE_GET_REJECTIONS)
for result, capacity in zip(results, capacities, strict=True)
)
)
class MooncakeEmbeddingStoreClient:
"""Wraps Mooncake object and buffer APIs used by embedding transfer."""
def __init__(
self,
store: Any,
replicate_config: Any | None = None,
*,
read_buffer_bytes: int = 128 * 1024**2,
):
self._lifetime_lock = threading.Lock()
self._unsafe_owners: list[Any] = []
self._poisoned = False
self.store = store
self.replicate_config = replicate_config
self._read_buffer_bytes = read_buffer_bytes
self._read_buffer: torch.Tensor | None = None
self._put_header: ctypes.Array[ctypes.c_char] | None = None
def _check_healthy(self) -> None:
if self._poisoned:
raise EmbeddingStoreError(
"Mooncake I/O completion is unconfirmed; worker must exit"
)
def _poison(self, owners: list[Any]) -> None:
with self._lifetime_lock:
self._unsafe_owners.extend(owners)
if not self._poisoned:
self._poisoned = True
_UNSAFE_CLIENTS.append(self)
@contextmanager
def _registered_io(
self,
buffers: list[tuple[Any, int, int]],
) -> Iterator[Callable[[int], None]]:
self._check_healthy()
registered = []
submitted = False
confirmed = False
def confirm(result: int) -> None:
nonlocal confirmed
if type(result) is not int or (
result < 0 and result not in _SAFE_PUT_REJECTIONS
):
raise EmbeddingStoreError(
"Mooncake I/O completion is unconfirmed; retaining buffers"
)
confirmed = True
try:
for owner, addr, size in buffers:
ret = self.store.register_buffer(addr, size)
if ret != 0:
raise EmbeddingStoreOperationError(
f"Failed to register embedding buffer: {ret}"
)
registered.append(addr)
submitted = True
yield confirm
except BaseException as error:
if submitted and not confirmed:
self._poison([owner for owner, _, _ in buffers])
raise EmbeddingStoreError(
"Mooncake I/O completion is unconfirmed; retaining buffers"
) from error
raise
finally:
if not submitted or confirmed:
try:
for addr in reversed(registered):
self._unregister_buffer(addr)
except BaseException:
self._poison([owner for owner, _, _ in buffers])
raise
def close(self) -> None:
"""Close the supported Mooncake binding after all I/O is drained."""
self._check_healthy()
if self._read_buffer is not None:
try:
self._unregister_buffer(self._read_buffer.data_ptr())
except BaseException:
self._poison([self._read_buffer])
raise
self._read_buffer = None
if self._put_header is not None:
try:
self._unregister_buffer(ctypes.addressof(self._put_header))
except BaseException:
self._poison([self._put_header])
raise
self._put_header = None
ret = self.store.close()
if ret != 0:
raise EmbeddingStoreError(
f"failed to close embedding Mooncake Store: {ret}"
)
def exists(self, pool_key: str) -> bool:
return self.batch_exists([pool_key])[0]
def batch_exists(self, keys: list[str]) -> list[bool]:
self._check_healthy()
if not keys:
return []
states = self.store.batch_is_exist(keys)
if len(states) != len(keys):
raise RuntimeError(
"Mooncake Store returned an unexpected number of lookup results"
)
return [state == 1 for state in states]
def load_tensors(
self, expected: Mapping[str, TensorSpec], device: torch.device | str
) -> dict[str, torch.Tensor]:
"""Synchronously load runner inputs into independently owned tensors."""
self._check_healthy()
device = torch.device(device)
loaded: dict[str, torch.Tensor] = {}
chunk: dict[str, TensorSpec] = {}
chunk_bytes = 0
for key, spec in expected.items():
size = _MOONCAKE_TENSOR_HEADER.size + spec.nbytes
if size > self._read_buffer_bytes:
logger.warning(
"Skipping Store GET for %s: object bytes=%d exceed read buffer=%d",
key,
size,
self._read_buffer_bytes,
)
continue
if chunk_bytes + size > self._read_buffer_bytes:
loaded.update(self._load_chunk(chunk, device))
chunk = {}
chunk_bytes = 0
chunk[key] = spec
chunk_bytes += size
if chunk:
loaded.update(self._load_chunk(chunk, device))
return loaded
def _load_chunk(
self, expected: dict[str, TensorSpec], device: torch.device
) -> dict[str, torch.Tensor]:
self._check_healthy()
if self._read_buffer is None:
buffer = torch.empty(
self._read_buffer_bytes,
dtype=torch.uint8,
device="cpu",
pin_memory=device.type == "cuda",
)
ret = self.store.register_buffer(buffer.data_ptr(), buffer.nbytes)
if ret != 0:
raise EmbeddingStoreOperationError(
f"Failed to register embedding read buffer: {ret}"
)
self._read_buffer = buffer
buffer = self._read_buffer
base = buffer.data_ptr()
sizes = [
_MOONCAKE_TENSOR_HEADER.size + spec.nbytes for spec in expected.values()
]
offsets: list[int] = []
end = 0
for size in sizes:
offsets.append(end)
end += size
try:
results = self.store.batch_get_into(
list(expected), [base + offset for offset in offsets], sizes
)
if not _batch_get_completed_safely(results, sizes):
raise EmbeddingStoreError("Unsafe batch GET results")
except BaseException as error:
self._poison([buffer])
raise EmbeddingStoreError(
"Mooncake I/O completion is unconfirmed; retaining read buffer"
) from error
valid: list[tuple[str, TensorSpec, int]] = []
for (key, spec), offset, size, result in zip(
expected.items(), offsets, sizes, results, strict=True
):
if result != size:
logger.warning(
"Store GET skipped for %s: result=%d, expected bytes=%d",
key,
result,
size,
)
continue
try:
_validate_mooncake_tensor_metadata(
key,
ctypes.string_at(base + offset, _MOONCAKE_TENSOR_HEADER.size),
spec,
)
except EmbeddingStoreOperationError as error:
logger.warning("Store GET skipped for %s: %s", key, error)
continue
valid.append((key, spec, offset + _MOONCAKE_TENSOR_HEADER.size))
if not valid:
return {}
# Allocate every destination before submitting any asynchronous copy.
targets = {
key: torch.empty(
spec.shape,
dtype=getattr(torch, spec.dtype.removeprefix("torch.")),
device=device,
)
for key, spec, _ in valid
}
ready = torch.Event() if device.type == "cuda" else None
try:
for key, spec, offset in valid:
targets[key].view(torch.uint8).view(-1).copy_(
buffer[offset : offset + spec.nbytes],
non_blocking=ready is not None,
)
if ready is not None:
ready.record(torch.accelerator.current_stream(device))
ready.synchronize()
except BaseException as error:
if ready is not None:
self._poison([buffer, *targets.values(), ready])
raise EmbeddingStoreError(
"Embedding copy completion is unconfirmed; retaining buffers"
) from error
raise
return targets
def put_tensor(self, pool_key: str, tensor: torch.Tensor) -> None:
self._check_healthy()
if not tensor.is_contiguous():
raise EmbeddingStoreOperationError("embedding tensor must be contiguous")
metadata = _encode_mooncake_tensor_metadata(tensor)
# The single publisher reuses this header only after PUT completion.
if self._put_header is None:
header = ctypes.create_string_buffer(len(metadata))
ret = self.store.register_buffer(ctypes.addressof(header), len(metadata))
if ret != 0:
raise EmbeddingStoreOperationError(
f"Failed to register embedding header: {ret}"
)
self._put_header = header
header_ptr = ctypes.addressof(self._put_header)
ctypes.memmove(header_ptr, metadata, len(metadata))
data_size = tensor.numel() * tensor.element_size()
buffers = [
(tensor, tensor.data_ptr(), data_size),
]
with self._registered_io(buffers) as confirm:
[result] = self.store.batch_put_from_multi_buffers(
[pool_key],
[[header_ptr, tensor.data_ptr()]],
[[len(metadata), data_size]],
self.replicate_config,
)
confirm(result)
if result < 0:
raise EmbeddingStoreOperationError(
f"failed to put embedding tensor for {pool_key}"
)
def _unregister_buffer(self, addr: int) -> None:
try:
ret = self.store.unregister_buffer(addr)
except Exception as error:
raise EmbeddingStoreError(
f"could not confirm buffer unregistration addr={addr:#x}"
) from error
if ret != 0:
raise EmbeddingStoreError(
f"failed to unregister embedding buffer addr={addr:#x}: {ret}"
)
def _validate_mooncake_tensor_metadata(
pool_key: str,
metadata: bytes,
expected: TensorSpec,
) -> None:
magic, dtype, num_tokens, hidden_dim = _MOONCAKE_TENSOR_HEADER.unpack(metadata)
if magic != _MOONCAKE_TENSOR_OBJECT_MAGIC:
raise EmbeddingStoreOperationError(
f"invalid Mooncake tensor metadata header for {pool_key}"
)
if _MOONCAKE_DTYPE_TO_TORCH_DTYPE.get(dtype) != expected.dtype:
raise EmbeddingStoreOperationError(
f"unexpected Mooncake tensor dtype for {pool_key}: {dtype}"
)
if (num_tokens, hidden_dim) != expected.shape:
raise EmbeddingStoreOperationError(
f"embedding tensor shape mismatch for {pool_key}: expected={expected.shape}"
)
def _encode_mooncake_tensor_metadata(tensor: torch.Tensor) -> bytes:
dtype = str(tensor.dtype)
if dtype not in _TORCH_DTYPE_TO_MOONCAKE_DTYPE:
raise EmbeddingStoreOperationError(
f"unsupported embedding tensor dtype: {dtype}"
)
if tensor.ndim != 2:
raise EmbeddingStoreOperationError(
f"embedding tensor must be 2D: shape={tuple(tensor.shape)}"
)
return _MOONCAKE_TENSOR_HEADER.pack(
_MOONCAKE_TENSOR_OBJECT_MAGIC,
_TORCH_DTYPE_TO_MOONCAKE_DTYPE[dtype],
*tensor.shape,
)
@@ -11,7 +11,6 @@ and MooncakeDistributedStore integration.
"""
import dataclasses
import json
import math
import os
import queue
@@ -20,10 +19,8 @@ import threading
import time
from collections.abc import Callable, Sequence
from concurrent.futures import Future, ThreadPoolExecutor
from dataclasses import dataclass
from typing import Any, Literal, TypeVar
from typing import Any, TypeVar
import regex as re
import torch
import zmq
@@ -67,6 +64,7 @@ from vllm.distributed.kv_transfer.kv_connector.v1.mooncake.store.protocol import
decode_lookup_response,
encode_lookup_response,
)
from vllm.distributed.mooncake_store import MooncakeStoreConfig, setup_mooncake_store
from vllm.logger import init_logger
from vllm.utils.math_utils import cdiv
from vllm.utils.network_utils import get_ip, make_zmq_socket
@@ -94,10 +92,6 @@ from .metrics import MooncakeStoreConnectorStats
logger = init_logger(__name__)
DEFAULT_GLOBAL_SEGMENT_SIZE = 4 * 1024 * 1024 * 1024 # 4 GiB
DEFAULT_LOCAL_BUFFER_SIZE = 4 * 1024 * 1024 * 1024 # 4 GiB
DEFAULT_TENANT_ID = "default"
MOONCAKE_NO_AVAILABLE_HANDLE = -200
_T = TypeVar("_T")
@@ -149,115 +143,6 @@ _DIRECT_IO_ALIGNMENT = 4096
_DIRECT_IO_PADDING_BYTES = 2 * _DIRECT_IO_ALIGNMENT
MooncakeMode = Literal["embedded", "standalone-store"]
@dataclass
class MooncakeStoreConfig:
"""Configuration for MooncakeDistributedStore.
``mode`` selects the topology: ``embedded`` (each rank contributes
``global_segment_size`` in-process) or ``standalone-store`` (rank
contributes 0; an external ``mooncake_client`` process owns the pool
and the SSD tier).
"""
metadata_server: str
master_server_address: str
protocol: str
device_name: str
mode: MooncakeMode = "embedded"
global_segment_size: int = DEFAULT_GLOBAL_SEGMENT_SIZE
local_buffer_size: int = DEFAULT_LOCAL_BUFFER_SIZE
enable_offload: bool = False
tenant_id: str = DEFAULT_TENANT_ID
def __post_init__(self) -> None:
if self.mode not in ("embedded", "standalone-store"):
raise ValueError(f"unknown Mooncake mode: {self.mode!r}")
if self.local_buffer_size <= 0:
raise ValueError("local_buffer_size must be > 0")
if self.mode == "embedded" and self.global_segment_size == 0:
raise ValueError("embedded mode requires global_segment_size > 0")
if self.mode == "standalone-store" and self.global_segment_size != 0:
raise ValueError("standalone-store mode requires global_segment_size == 0")
@staticmethod
def from_file(file_path: str) -> "MooncakeStoreConfig":
with open(file_path) as file:
config = json.load(file)
return MooncakeStoreConfig(
metadata_server=config.get("metadata_server", ""),
master_server_address=config.get("master_server_address", ""),
protocol=config.get("protocol", "rdma"),
device_name=config.get("device_name", ""),
mode=config.get("mode", "embedded"),
global_segment_size=_parse_size(
config.get("global_segment_size", DEFAULT_GLOBAL_SEGMENT_SIZE)
),
local_buffer_size=_parse_size(
config.get("local_buffer_size", DEFAULT_LOCAL_BUFFER_SIZE)
),
enable_offload=bool(config.get("enable_offload", False)),
tenant_id=_normalize_tenant_id(config.get("tenant_id", DEFAULT_TENANT_ID)),
)
@staticmethod
def load_from_config() -> "MooncakeStoreConfig":
config_path = os.getenv("MOONCAKE_CONFIG_PATH")
if not config_path:
raise ValueError(
"The environment variable 'MOONCAKE_CONFIG_PATH' is not set."
)
return MooncakeStoreConfig.from_file(config_path)
def _normalize_tenant_id(value: Any) -> str:
if value is None:
return DEFAULT_TENANT_ID
if not isinstance(value, str):
raise TypeError(
f"tenant_id must be a string or null, got {type(value).__name__}: {value!r}"
)
tenant_id = value.strip()
return tenant_id if tenant_id else DEFAULT_TENANT_ID
def _parse_size(value: Any) -> int:
"""Parse storage size strings with units: GB, MB, KB, B."""
if isinstance(value, int):
return value
if not isinstance(value, str):
try:
return int(value)
except (TypeError, ValueError) as e:
raise TypeError(f"Unsupported type for size: {type(value)}") from e
cleaned = value.strip().lower()
if not cleaned:
raise ValueError("Size cannot be empty.")
unit_multipliers = {
"gb": 1024**3,
"mb": 1024**2,
"kb": 1024,
"b": 1,
}
match = re.match(r"^\s*([\d.]+)\s*(gb|mb|kb|b)?\s*$", cleaned)
if not match:
raise ValueError(f"Invalid format: '{value}'")
number_str = match.group(1)
unit = match.group(2) or "b"
multiplier = unit_multipliers[unit]
try:
numeric_value = float(number_str)
except ValueError as exc:
raise ValueError(f"Invalid numeric value '{number_str}' in: '{value}'") from exc
return int(numeric_value * multiplier)
def _align_up(value: int, alignment: int) -> int:
return ((value + alignment - 1) // alignment) * alignment
@@ -1595,23 +1480,7 @@ class MooncakeStoreWorker:
self.store = MooncakeDistributedStore()
local_ip = get_ip()
local_hostname = rdma_utils.get_requester_local_hostname(local_ip)
setup_kwargs: dict[str, str] = {}
if store_config.tenant_id != DEFAULT_TENANT_ID:
setup_kwargs["tenant_id"] = store_config.tenant_id
ret = self.store.setup(
local_hostname,
store_config.metadata_server,
store_config.global_segment_size,
store_config.local_buffer_size,
store_config.protocol,
store_config.device_name,
store_config.master_server_address,
**setup_kwargs,
)
if ret != 0:
msg = "Initialize MooncakeDistributedStore failed."
logger.error(msg)
raise RuntimeError(msg)
setup_mooncake_store(self.store, store_config, local_hostname)
preferred_segment = rdma_utils.get_configured_preferred_segment(extra_config)
self.preferred_segment = preferred_segment
+144
View File
@@ -0,0 +1,144 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Shared configuration and setup for EC and KV Mooncake Store clients."""
import json
import os
from dataclasses import dataclass
from typing import Any, Literal
import regex as re
DEFAULT_GLOBAL_SEGMENT_SIZE = 4 * 1024 * 1024 * 1024 # 4 GiB
DEFAULT_LOCAL_BUFFER_SIZE = 4 * 1024 * 1024 * 1024 # 4 GiB
DEFAULT_TENANT_ID = "default"
MooncakeMode = Literal["embedded", "standalone-store"]
@dataclass
class MooncakeStoreConfig:
"""Configuration for MooncakeDistributedStore.
``mode`` selects the topology: ``embedded`` (each rank contributes
``global_segment_size`` in-process) or ``standalone-store`` (rank
contributes 0; an external ``mooncake_client`` process owns the pool
and the SSD tier).
"""
metadata_server: str
master_server_address: str
protocol: str
device_name: str
mode: MooncakeMode = "embedded"
global_segment_size: int = DEFAULT_GLOBAL_SEGMENT_SIZE
local_buffer_size: int = DEFAULT_LOCAL_BUFFER_SIZE
enable_offload: bool = False
tenant_id: str = DEFAULT_TENANT_ID
def __post_init__(self) -> None:
if self.mode not in ("embedded", "standalone-store"):
raise ValueError(f"unknown Mooncake mode: {self.mode!r}")
if self.local_buffer_size <= 0:
raise ValueError("local_buffer_size must be > 0")
if self.mode == "embedded" and self.global_segment_size == 0:
raise ValueError("embedded mode requires global_segment_size > 0")
if self.mode == "standalone-store" and self.global_segment_size != 0:
raise ValueError("standalone-store mode requires global_segment_size == 0")
@staticmethod
def from_file(file_path: str) -> "MooncakeStoreConfig":
with open(file_path) as file:
config = json.load(file)
return MooncakeStoreConfig(
metadata_server=config.get("metadata_server", ""),
master_server_address=config.get("master_server_address", ""),
protocol=config.get("protocol", "rdma"),
device_name=config.get("device_name", ""),
mode=config.get("mode", "embedded"),
global_segment_size=_parse_size(
config.get("global_segment_size", DEFAULT_GLOBAL_SEGMENT_SIZE)
),
local_buffer_size=_parse_size(
config.get("local_buffer_size", DEFAULT_LOCAL_BUFFER_SIZE)
),
enable_offload=bool(config.get("enable_offload", False)),
tenant_id=_normalize_tenant_id(config.get("tenant_id", DEFAULT_TENANT_ID)),
)
@staticmethod
def load_from_config() -> "MooncakeStoreConfig":
config_path = os.getenv("MOONCAKE_CONFIG_PATH")
if not config_path:
raise ValueError(
"The environment variable 'MOONCAKE_CONFIG_PATH' is not set."
)
return MooncakeStoreConfig.from_file(config_path)
def _normalize_tenant_id(value: Any) -> str:
if value is None:
return DEFAULT_TENANT_ID
if not isinstance(value, str):
raise TypeError(
f"tenant_id must be a string or null, got {type(value).__name__}: {value!r}"
)
tenant_id = value.strip()
return tenant_id if tenant_id else DEFAULT_TENANT_ID
def _parse_size(value: Any) -> int:
"""Parse storage size strings with units: GB, MB, KB, B."""
if isinstance(value, int):
return value
if not isinstance(value, str):
try:
return int(value)
except (TypeError, ValueError) as e:
raise TypeError(f"Unsupported type for size: {type(value)}") from e
cleaned = value.strip().lower()
if not cleaned:
raise ValueError("Size cannot be empty.")
unit_multipliers = {
"gb": 1024**3,
"mb": 1024**2,
"kb": 1024,
"b": 1,
}
match = re.match(r"^\s*([\d.]+)\s*(gb|mb|kb|b)?\s*$", cleaned)
if not match:
raise ValueError(f"Invalid format: '{value}'")
number_str = match.group(1)
unit = match.group(2) or "b"
multiplier = unit_multipliers[unit]
try:
numeric_value = float(number_str)
except ValueError as exc:
raise ValueError(f"Invalid numeric value '{number_str}' in: '{value}'") from exc
return int(numeric_value * multiplier)
def setup_mooncake_store(
store: Any, config: MooncakeStoreConfig, local_hostname: str
) -> None:
"""Initialize a native Store, preserving explicit tenant configuration."""
setup_kwargs: dict[str, str] = {}
if config.tenant_id != DEFAULT_TENANT_ID:
setup_kwargs["tenant_id"] = config.tenant_id
ret = store.setup(
local_hostname,
config.metadata_server,
config.global_segment_size,
config.local_buffer_size,
config.protocol,
config.device_name,
config.master_server_address,
**setup_kwargs,
)
if ret != 0:
raise RuntimeError("Initialize MooncakeDistributedStore failed.")