[Feature] Support manually enabling the cumem allocator (#33648)

Signed-off-by: Kebe <mail@kebe7jun.com>
This commit is contained in:
Kebe
2026-05-20 08:58:30 -04:00
committed by GitHub
parent 87e31455b0
commit 19cf334207
6 changed files with 60 additions and 17 deletions
+4
View File
@@ -375,6 +375,10 @@ The `kv_load_failure_policy` setting controls how the system handles failures wh
!!! warning
Using `kv_load_failure_policy="recompute"` can lead to performance degradation in production deployments. When KV loads fail, the decode instance will execute prefill work with decode-optimized configurations, which is inefficient and defeats the purpose of disaggregated prefilling. This also increases tail latency for other ongoing decode requests.
### For NVIDIA GB-series GPUs
GB-series GPUs support multi-node NVLink. NIXL supports this capability, but KVCache must be registered as VMM during KVCache registration. To enable this feature, you need to set `--enable-cumem-allocator` or `--enable-sleep-mode` flags, and set `UCX_CUDA_IPC_ENABLE_MNNVL: 'y'` env. Otherwise, NIXL can only use RDMA/TCP for cross-node KVCache transfers.
## Experimental Feature
### Heterogeneous KV Layout support