iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

[Docs] Fix VLLM_CPU_KVCACHE_SPACE default and units (#60053)

Browse Source 
Signed-off-by: Praneeth <praneethreddyarikatla@gmail.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
This commit is contained in:
Praneeth Reddy A
2026-10-09 19:43:55 +08:00
committed by GitHub
co-authored by Li, Jiang <jiang1.li@intel.com>
parent d095e61c0b
commit 1c47ebfc57
3 changed files with 3 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/configuration/conserving_memory.md
+1 -1
View File
@@ -80,7 +80,7 @@ llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct", enforce_eager=True)
If you run out of CPU RAM, try the following options:
- (Multi-modal models only) you can set the size of multi-modal cache by setting `mm_processor_cache_gb` engine argument (default 4 GiB).
- (CPU backend only) you can set the size of KV cache using `VLLM_CPU_KVCACHE_SPACE` environment variable (default 4 GiB).
- (CPU backend only) you can set the size of KV cache using `VLLM_CPU_KVCACHE_SPACE` environment variable, in GiB (default 0, meaning vLLM doesn't override the KV cache size through this variable).
## Multi-modal input limits
docs/getting_started/installation/cpu.md
+1 -1
View File
@@ -293,7 +293,7 @@ See [AMD Zen optimizations](#amd-zen-optimizations) for detection rules, support
### How to decide `VLLM_CPU_KVCACHE_SPACE`?
This value is 4GB by default. Larger space can support more concurrent requests, longer context length. However, users should take care of memory capacity of each NUMA node. The memory usage of each TP rank is the sum of `weight shard size` and `VLLM_CPU_KVCACHE_SPACE`, if it exceeds the capacity of a single NUMA node, the TP worker will be killed with `exitcode 9` due to out-of-memory.
This value is 0 by default, which means vLLM doesn't override the KV cache size through this variable. When set, the value is in GiB (e.g, `VLLM_CPU_KVCACHE_SPACE`=40 means 40 GiB). Larger space can support more concurrent requests, longer context length. However, users should take care of memory capacity of each NUMA node. The memory usage of each TP rank is the sum of `weight shard size` and `VLLM_CPU_KVCACHE_SPACE`, if it exceeds the capacity of a single NUMA node, the TP worker will be killed with `exitcode 9` due to out-of-memory. This is a legacy setting; use `--kv-cache-memory-bytes` instead.
### How to do performance tuning for vLLM CPU?
vllm/platforms/cpu.py
+1 -1
View File
@@ -323,7 +323,7 @@ class CpuPlatform(Platform):
mamba_backend,
)
# Lagecy setting
# Legacy setting
env_key = "VLLM_CPU_KVCACHE_SPACE"
if env_key in os.environ and os.environ[env_key] != "":
kv_cache_space = int(os.environ[env_key])