diff --git a/docs/deployment/integrations/production-stack.md b/docs/deployment/integrations/production-stack.md index 4db595164e..d93300a2b0 100644 --- a/docs/deployment/integrations/production-stack.md +++ b/docs/deployment/integrations/production-stack.md @@ -4,7 +4,7 @@ Deploying vLLM on Kubernetes is a scalable and efficient way to serve machine le * **Upstream vLLM compatibility** – It wraps around upstream vLLM without modifying its code. * **Ease of use** – Simplified deployment via Helm charts and observability through Grafana dashboards. -* **High performance** – Optimized for LLM workloads with features like multimodel support, model-aware and prefix-aware routing, fast vLLM bootstrapping, and KV cache offloading with [LMCache](https://github.com/LMCache/LMCache), among others. +* **High performance** – Optimized for LLM workloads with features like multimodel support, model-aware and prefix-aware routing, fast vLLM bootstrapping, and KV cache offloading with [LMCache](https://github.com/LMCache/LMCache) (wired up in vLLM via `--kv-offloading-backend lmcache`; see the [LMCache examples](https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/lmcache) and [docs.lmcache.ai](https://docs.lmcache.ai)), among others. If you are new to Kubernetes, don't worry: in the vLLM production stack [repo](https://github.com/vllm-project/production-stack), we provide a step-by-step [guide](https://github.com/vllm-project/production-stack/blob/main/tutorials/00-install-kubernetes-env.md) and a [short video](https://www.youtube.com/watch?v=EsTJbQtzj0g) to set up everything and get started in **4 minutes**! diff --git a/docs/features/disagg_prefill.md b/docs/features/disagg_prefill.md index 8352d2f20e..578343096d 100644 --- a/docs/features/disagg_prefill.md +++ b/docs/features/disagg_prefill.md @@ -20,7 +20,7 @@ Two main reasons: Now supports 9 types of connectors: - **ExampleConnector**: refer to [examples/disaggregated/example_connector/run.sh](../../examples/disaggregated/example_connector/run.sh) for the example usage of ExampleConnector disaggregated prefilling. -- **LMCacheConnectorV1**: refer to [examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh](../../examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh) for the example usage of LMCacheConnectorV1 disaggregated prefilling which uses NIXL as the underlying KV transmission. +- **LMCacheConnectorV1**: refer to [examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh](../../examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_example_nixl.sh) for the example usage of LMCacheConnectorV1 disaggregated prefilling which uses NIXL as the underlying KV transmission. LMCache also offers a multi-process (MP) mode via `LMCacheMPConnector`, where a standalone `lmcache server` holds the KV cache shared by one or more vLLM instances; see the [LMCache examples](../../examples/disaggregated/lmcache/README.md) and the [LMCache docs](https://docs.lmcache.ai) for setup. - **NixlConnector**: refer to [tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh](../../tests/v1/kv_connector/nixl_integration/run_accuracy_test.sh) for the example usage of NixlConnector disaggregated prefilling which support fully async send/recv. For detailed usage guide, see [NixlConnector Usage Guide](nixl_connector_usage.md). For feature compatibility details, see [NixlConnector Compatibility Matrix](nixl_connector_compatibility.md). You may specify one or multiple NIXL transfer backends, such as: ```bash diff --git a/examples/disaggregated/lmcache/README.md b/examples/disaggregated/lmcache/README.md index 759be55d6f..87fec82684 100644 --- a/examples/disaggregated/lmcache/README.md +++ b/examples/disaggregated/lmcache/README.md @@ -1,10 +1,38 @@ # LMCache Examples -This folder demonstrates how to use LMCache for disaggregated prefilling, CPU offloading and KV cache sharing. +This folder demonstrates how to use LMCache with vLLM v1 for KV cache +offloading, disaggregated prefilling, and KV cache sharing. -## 1. Disaggregated Prefill in vLLM v1 +## Integration modes -This example demonstrates how to run LMCache with disaggregated prefill using NIXL on a single node. +LMCache integrates with vLLM v1 in two ways: + +- **In-process mode** (`LMCacheConnectorV1`): LMCache runs inside the vLLM + process and is configured through environment variables or a YAML config + file (`LMCACHE_CONFIG_FILE`). This is the simplest way to add single-node + CPU/disk offloading. +- **Multi-process (MP) mode** (`LMCacheMPConnector`): LMCache runs as a + standalone server (`lmcache server`) that owns the KV cache storage; one or + more vLLM instances connect to it. This is the recommended mode for + distributed KV storage and for sharing KV cache across instances. See the + [LMCache docs](https://docs.lmcache.ai) for the full MP setup. + +## 1. CPU offload (in-process) + +- `python cpu_offload_lmcache.py` - CPU offloading with `LMCacheConnectorV1` + for vLLM v1. + +## 2. CPU offload (multi-process) + +- `bash cpu_offload_lmcache_mp.sh` - CPU offloading with `LMCacheMPConnector`, + using a standalone `lmcache server`. vLLM provides a built-in shortcut for + this setup via `--kv-offloading-backend lmcache` and + `--kv-offloading-size `. + +## 3. Disaggregated Prefill in vLLM v1 + +This example demonstrates how to run LMCache with disaggregated prefill using +NIXL on a single node. ### Prerequisites @@ -46,15 +74,7 @@ The main script generates several log files: - `decoder.log` - Logs from the decode server - `proxy.log` - Logs from the proxy server -## 2. CPU Offload Examples +## 4. KV Cache Sharing -- `python cpu_offload_lmcache.py -v v0` - CPU offloading implementation for vLLM v0 -- `python cpu_offload_lmcache.py -v v1` - CPU offloading implementation for vLLM v1 - -## 3. KV Cache Sharing - -The `kv_cache_sharing_lmcache_v1.py` example demonstrates how to share KV caches between vLLM v1 instances. - -## 4. Disaggregated Prefill in vLLM v0 - -The `disaggregated_prefill_lmcache_v0.py` provides an example of how to run disaggregated prefill in vLLM v0. +The `kv_cache_sharing_lmcache_v1.py` example demonstrates how to share KV +caches between vLLM v1 instances through a centralized LMCache server. diff --git a/examples/disaggregated/lmcache/cpu_offload_lmcache.py b/examples/disaggregated/lmcache/cpu_offload_lmcache.py index 53036b3eb0..b67a929e5d 100644 --- a/examples/disaggregated/lmcache/cpu_offload_lmcache.py +++ b/examples/disaggregated/lmcache/cpu_offload_lmcache.py @@ -1,20 +1,8 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project """ -This file demonstrates the example usage of cpu offloading -with LMCache in vLLM v1 or v0. - -Usage: - - Specify vLLM version - - -v v0 : Use LMCacheConnector - model = mistralai/Mistral-7B-Instruct-v0.2 - (Includes enable_chunked_prefill = True) - - -v v1 : Use LMCacheConnectorV1 (default) - model = meta-llama/Meta-Llama-3.1-8B-Instruct - (Without enable_chunked_prefill) +This file demonstrates the example usage of CPU offloading +with LMCache in vLLM v1. Note that `lmcache` is needed to run this example. Requirements: @@ -23,7 +11,6 @@ Learn more about LMCache environment setup, please refer to: https://docs.lmcache.ai/getting_started/installation.html """ -import argparse import contextlib import os import time @@ -39,8 +26,6 @@ from vllm.engine.arg_utils import EngineArgs def setup_environment_variables(): # LMCache-related environment variables - # Use experimental features in LMCache - os.environ["LMCACHE_USE_EXPERIMENTAL"] = "True" # LMCache is set to use 256 tokens per chunk os.environ["LMCACHE_CHUNK_SIZE"] = "256" # Enable local CPU backend in LMCache @@ -50,9 +35,9 @@ def setup_environment_variables(): @contextlib.contextmanager -def build_llm_with_lmcache(lmcache_connector: str, model: str): +def build_llm_with_lmcache(model: str): ktc = KVTransferConfig( - kv_connector=lmcache_connector, + kv_connector="LMCacheConnectorV1", kv_role="kv_both", ) # Set GPU memory utilization to 0.8 for an A40 GPU with 40GB @@ -92,23 +77,10 @@ def print_output( print("-" * 50) -def parse_args(): - parser = argparse.ArgumentParser() - parser.add_argument( - "-v", - "--version", - choices=["v0", "v1"], - default="v1", - help="Specify vLLM version (default: v1)", - ) - return parser.parse_args() - - def main(): - lmcache_connector = "LMCacheConnectorV1" model = "meta-llama/Meta-Llama-3.1-8B-Instruct" setup_environment_variables() - with build_llm_with_lmcache(lmcache_connector, model) as llm: + with build_llm_with_lmcache(model) as llm: # This example script runs two requests with a shared prefix. # Define the shared prompt and specific prompts shared_prompt = "Hello, how are you?" * 1000 diff --git a/examples/disaggregated/lmcache/cpu_offload_lmcache_mp.sh b/examples/disaggregated/lmcache/cpu_offload_lmcache_mp.sh new file mode 100755 index 0000000000..2372eabe1a --- /dev/null +++ b/examples/disaggregated/lmcache/cpu_offload_lmcache_mp.sh @@ -0,0 +1,43 @@ +#!/bin/bash +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +# +# CPU offloading with LMCache in multi-process (MP) mode. +# +# In MP mode, LMCache runs as a standalone server process (`lmcache server`) +# that owns the KV cache storage. One or more vLLM instances connect to it via +# the `LMCacheMPConnector`. This is the recommended way to run LMCache for +# distributed KV storage and for sharing KV cache across vLLM instances. +# +# vLLM ships a built-in shortcut for this setup: pass `--kv-offloading-backend +# lmcache` together with `--kv-offloading-size ` and vLLM wires up the +# `LMCacheMPConnector` for you (it defaults to the LMCache server at +# tcp://localhost:5555, matching the `lmcache server` default). +# +# Requires `lmcache` to be installed (`pip install lmcache`). +# Learn more: https://docs.lmcache.ai +set -euo pipefail + +MODEL=${MODEL:-meta-llama/Meta-Llama-3.1-8B-Instruct} + +# 1. Launch the standalone LMCache server (binds tcp://localhost:5555 by +# default). `--l1-size-gb` sets the CPU memory budget for the L1 cache. +echo "Starting LMCache server..." +lmcache server --host localhost --port 5555 --l1-size-gb 5 & +LMCACHE_SERVER_PID=$! +trap 'kill $LMCACHE_SERVER_PID 2>/dev/null || true' EXIT + +# 2. Launch vLLM and offload KV cache to the LMCache server. +# The MP connector currently requires the non-hybrid KV cache manager. +echo "Starting vLLM server with LMCache MP offloading..." +vllm serve "$MODEL" \ + --port 8000 \ + --kv-offloading-size 5 \ + --kv-offloading-backend lmcache \ + --disable-hybrid-kv-cache-manager + +# Equivalent explicit configuration (instead of the two flags above): +# --kv-transfer-config \ +# '{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both", +# "kv_connector_extra_config":{"lmcache.mp.host":"tcp://localhost", +# "lmcache.mp.port":5555}}' diff --git a/examples/disaggregated/lmcache/disagg_prefill_lmcache_v0.py b/examples/disaggregated/lmcache/disagg_prefill_lmcache_v0.py deleted file mode 100644 index 6669eb3fb3..0000000000 --- a/examples/disaggregated/lmcache/disagg_prefill_lmcache_v0.py +++ /dev/null @@ -1,144 +0,0 @@ -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright contributors to the vLLM project -""" -This file demonstrates the example usage of disaggregated prefilling -with LMCache. -We will launch 2 vllm instances (GPU 0 for prefill and GPU 1 for decode), -and launch an additional LMCache server. -KV cache is transferred in the following manner: -vLLM prefill node -> LMCache server -> vLLM decode node. - -Note that `pip install lmcache` is needed to run this example. -Learn more about LMCache in https://github.com/LMCache/LMCache. -""" - -import os -import subprocess -import time -from multiprocessing import Event, Process - -from lmcache.experimental.cache_engine import LMCacheEngineBuilder -from lmcache.integration.vllm.utils import ENGINE_NAME - -from vllm import LLM, SamplingParams -from vllm.config import KVTransferConfig - -# LMCache-related environment variables -# The port to start LMCache server -port = 8100 -# Use experimental features in LMCache -os.environ["LMCACHE_USE_EXPERIMENTAL"] = "True" -# LMCache is set to use 256 tokens per chunk -os.environ["LMCACHE_CHUNK_SIZE"] = "256" -# Disable local CPU backend in LMCache -os.environ["LMCACHE_LOCAL_CPU"] = "False" -# Set local CPU memory buffer limit to 5.0 GB -os.environ["LMCACHE_MAX_LOCAL_CPU_SIZE"] = "5.0" -# Set the remote URL for LMCache server -os.environ["LMCACHE_REMOTE_URL"] = f"lm://localhost:{port}" -# Set the serializer/deserializer between vllm and LMCache server -# `naive` indicates using raw bytes of the tensor without any compression -os.environ["LMCACHE_REMOTE_SERDE"] = "naive" - -prompts = [ - "Hello, how are you?" * 1000, -] - - -def run_prefill(prefill_done, prompts): - # We use GPU 0 for prefill node. - os.environ["CUDA_VISIBLE_DEVICES"] = "0" - - sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=1) - - ktc = KVTransferConfig( - kv_connector="LMCacheConnector", - kv_role="kv_producer", - kv_rank=0, - kv_parallel_size=2, - ) - # Set GPU memory utilization to 0.8 for an A40 GPU with 40GB - # memory. Reduce the value if your GPU has less memory. - llm = LLM( - model="mistralai/Mistral-7B-Instruct-v0.2", - kv_transfer_config=ktc, - max_model_len=8000, - gpu_memory_utilization=0.8, - enforce_eager=True, - ) - - # llm.generate(prompts, sampling_params) - outputs = llm.generate(prompts, sampling_params) - for output in outputs: - generated_text = output.outputs[0].text - print(f"Generated text: {generated_text!r}") - print("Prefill node is finished.") - prefill_done.set() - - # Clean up lmcache backend - LMCacheEngineBuilder.destroy(ENGINE_NAME) - - -def run_decode(prefill_done, prompts, timeout=1): - # We use GPU 1 for decode node. - os.environ["CUDA_VISIBLE_DEVICES"] = "1" - - sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=10) - - ktc = KVTransferConfig( - kv_connector="LMCacheConnector", - kv_role="kv_consumer", - kv_rank=1, - kv_parallel_size=2, - ) - # Set GPU memory utilization to 0.8 for an A40 GPU with 40GB - # of memory. Reduce the value if your GPU has less memory. - llm = LLM( - model="mistralai/Mistral-7B-Instruct-v0.2", - kv_transfer_config=ktc, - max_model_len=8000, - gpu_memory_utilization=0.8, - enforce_eager=True, - ) - - print("Waiting for prefill node to finish...") - prefill_done.wait() - time.sleep(timeout) - - outputs = llm.generate(prompts, sampling_params) - for output in outputs: - generated_text = output.outputs[0].text - print(f"Generated text: {generated_text!r}") - - # Clean up lmcache backend - LMCacheEngineBuilder.destroy(ENGINE_NAME) - - -def run_lmcache_server(port): - server_proc = subprocess.Popen( - ["python", "-m", "lmcache.experimental.server", "localhost", str(port)] - ) - return server_proc - - -def main(): - prefill_done = Event() - prefill_process = Process(target=run_prefill, args=(prefill_done, prompts)) - decode_process = Process(target=run_decode, args=(prefill_done, prompts)) - lmcache_server_process = run_lmcache_server(port) - - # Start prefill node - prefill_process.start() - - # Start decode node - decode_process.start() - - # Clean up the processes - decode_process.join() - prefill_process.terminate() - lmcache_server_process.terminate() - lmcache_server_process.wait() - - -if __name__ == "__main__": - main() diff --git a/examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh b/examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh index 363c35028a..61e578460c 100644 --- a/examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh +++ b/examples/disaggregated/lmcache/disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh @@ -30,7 +30,6 @@ if [[ $1 == "prefiller" ]]; then UCX_TLS=cuda_ipc,cuda_copy,tcp \ LMCACHE_CONFIG_FILE=$prefill_config_file \ - LMCACHE_USE_EXPERIMENTAL=True \ VLLM_ENABLE_V1_MULTIPROCESSING=1 \ VLLM_WORKER_MULTIPROC_METHOD=spawn \ CUDA_VISIBLE_DEVICES=0 \ @@ -47,7 +46,6 @@ elif [[ $1 == "decoder" ]]; then UCX_TLS=cuda_ipc,cuda_copy,tcp \ LMCACHE_CONFIG_FILE=$decode_config_file \ - LMCACHE_USE_EXPERIMENTAL=True \ VLLM_ENABLE_V1_MULTIPROCESSING=1 \ VLLM_WORKER_MULTIPROC_METHOD=spawn \ CUDA_VISIBLE_DEVICES=1 \ diff --git a/examples/disaggregated/lmcache/kv_cache_sharing_lmcache_v1.py b/examples/disaggregated/lmcache/kv_cache_sharing_lmcache_v1.py index 46e2d903d4..489ff13212 100644 --- a/examples/disaggregated/lmcache/kv_cache_sharing_lmcache_v1.py +++ b/examples/disaggregated/lmcache/kv_cache_sharing_lmcache_v1.py @@ -26,8 +26,6 @@ from vllm.config import KVTransferConfig # LMCache-related environment variables # The port to start LMCache server port = 8100 -# Use experimental features in LMCache -os.environ["LMCACHE_USE_EXPERIMENTAL"] = "True" # LMCache is set to use 256 tokens per chunk os.environ["LMCACHE_CHUNK_SIZE"] = "256" # Disable local CPU backend in LMCache