[Feature] Support manually enabling the cumem allocator (#33648)
Signed-off-by: Kebe <mail@kebe7jun.com>
This commit is contained in:
@@ -375,6 +375,10 @@ The `kv_load_failure_policy` setting controls how the system handles failures wh
|
||||
!!! warning
|
||||
Using `kv_load_failure_policy="recompute"` can lead to performance degradation in production deployments. When KV loads fail, the decode instance will execute prefill work with decode-optimized configurations, which is inefficient and defeats the purpose of disaggregated prefilling. This also increases tail latency for other ongoing decode requests.
|
||||
|
||||
### For NVIDIA GB-series GPUs
|
||||
|
||||
GB-series GPUs support multi-node NVLink. NIXL supports this capability, but KVCache must be registered as VMM during KVCache registration. To enable this feature, you need to set `--enable-cumem-allocator` or `--enable-sleep-mode` flags, and set `UCX_CUDA_IPC_ENABLE_MNNVL: 'y'` env. Otherwise, NIXL can only use RDMA/TCP for cross-node KVCache transfers.
|
||||
|
||||
## Experimental Feature
|
||||
|
||||
### Heterogeneous KV Layout support
|
||||
|
||||
Reference in New Issue
Block a user