[ROCm] Enable the cuMem CUDA-graph pool offload on ROCm (#59523)
Signed-off-by: aoshen02 <aoshen@inferact.ai> Signed-off-by: indianspeedster <cspandey016@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: indianspeedster <cspandey016@gmail.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
indianspeedster
parent
bacbbe1878
commit
a6f43e57d6
@@ -107,7 +107,7 @@ By default, CUDA graph memory stays on the GPU while asleep. With
|
||||
`sleep_mode_offload_cudagraph=True` (off by default), CUDA graphs are captured
|
||||
into a cuMem pool that sleep backs up to CPU memory at both levels and any wake
|
||||
restores in place, so graphs are replayed, not recaptured. It needs the default
|
||||
`cumem` backend, CUDA and CUDA graphs; otherwise it has no effect.
|
||||
`cumem` backend, CUDA or ROCm, and CUDA graphs; otherwise it has no effect.
|
||||
|
||||
```python
|
||||
llm = LLM("Qwen/Qwen3-8B", enable_sleep_mode=True, sleep_mode_offload_cudagraph=True)
|
||||
|
||||
Reference in New Issue
Block a user