Merge remote-tracking branch 'upstream/main' into patches
This commit is contained in:
@@ -130,7 +130,8 @@ Batch invariance has been tested and verified on the following models:
|
||||
- **Llama 3**: Llama3.1 and 3.2 series, `meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.2-3B-Instruct` for example
|
||||
- **GPT-OSS**: `openai/gpt-oss-20b`, `openai/gpt-oss-120b`
|
||||
- **Mistral**: `mistralai/Mistral-7B-v0.3`
|
||||
- **Phi series**: `microsoft/Phi-3.5-mini-instruct`
|
||||
- **GLM-4 series**: `zai-org/glm-4-9b-chat`
|
||||
- **Phi series**: `microsoft/Phi-3.5-mini-instruct`, `microsoft/phi-4`
|
||||
- **Granite 3.1 (MoE)**: `ibm-granite/granite-3.1-1b-a400m-instruct`, `ibm-granite/granite-3.1-3b-a800m-instruct`
|
||||
- **Granite 3.1 (Dense)**: `ibm-granite/granite-3.1-2b-instruct`, `ibm-granite/granite-3.1-8b-instruct`
|
||||
- **EXAONE 4.0 series**: `LGAI-EXAONE/EXAONE-4.0-1.2B`, `LGAI-EXAONE/EXAONE-4.0.1-32B`, `LGAI-EXAONE/EXAONE-4.0-32B`
|
||||
@@ -152,6 +153,7 @@ When batch invariance is enabled, vLLM:
|
||||
5. On CUDA devices with tuned matmul table entries for the model's bf16 unquantized forward linear layers (Ada, Hopper, Blackwell),
|
||||
runs without `torch.compile` using breakable CUDA graphs so tile configs follow the runtime batch size; set `VLLM_USE_BREAKABLE_CUDAGRAPH=0` to opt out
|
||||
(not applied when sequence parallelism / async TP are enabled, since those are `torch.compile` passes).
|
||||
6. Splits the LoRA shrink reduction into 8 fixed K chunks that are summed in a fixed order instead of with atomics
|
||||
|
||||
!!! warning
|
||||
Batch invariance under tensor parallelism is not yet supported for all-reduces whose size isn't a multiple of 16 bytes, for example a hidden size that isn't a multiple of 8 in fp16/bf16. Such tensors can switch between NCCL and custom all-reduce depending on batch size. All validated models meet this requirement.
|
||||
|
||||
@@ -101,6 +101,41 @@ llm.release_kv_cache_memory()
|
||||
llm.wake_up(tags=["kv_cache"]) # Reallocate KV cache and resume scheduling.
|
||||
```
|
||||
|
||||
#### Offloading CUDA graph memory
|
||||
|
||||
By default, CUDA graph memory stays on the GPU while asleep. With
|
||||
`sleep_mode_offload_cudagraph=True` (off by default), CUDA graphs are captured
|
||||
into a cuMem pool that sleep backs up to CPU memory at both levels and any wake
|
||||
restores in place, so graphs are replayed, not recaptured. It needs the default
|
||||
`cumem` backend, CUDA and CUDA graphs; otherwise it has no effect.
|
||||
|
||||
```python
|
||||
llm = LLM("Qwen/Qwen3-8B", enable_sleep_mode=True, sleep_mode_offload_cudagraph=True)
|
||||
```
|
||||
|
||||
or `vllm serve <model> --enable-sleep-mode --sleep-mode-offload-cudagraph`.
|
||||
|
||||
The cost is pinned host memory for the pool's backup, also at level 2, and one
|
||||
extra copy per captured custom allreduce (about 3% decode latency at batch
|
||||
size 1). `NCCL_GRAPH_REGISTER` defaults to `0`, since NCCL graph registration
|
||||
would pin the pool; an explicit value is kept with a warning. Graph executables
|
||||
outside PyTorch pools stay resident.
|
||||
|
||||
#### Releasing NCCL communicator memory
|
||||
|
||||
By default, NCCL communicators keep their GPU buffers while asleep. With
|
||||
`enable_nccl_comm_suspend=True` (off by default, experimental), sleep releases
|
||||
them with `ncclCommSuspend` and wake restores them with `ncclCommResume`. The
|
||||
communicators keep their topology, so they are not re-created. It needs NCCL
|
||||
2.29.7 or newer; with an older library, a warning is logged and the memory stays
|
||||
on the GPU.
|
||||
|
||||
```python
|
||||
llm = LLM("Qwen/Qwen3-8B", enable_sleep_mode=True, enable_nccl_comm_suspend=True)
|
||||
```
|
||||
|
||||
or `vllm serve <model> --enable-sleep-mode --enable-nccl-comm-suspend`.
|
||||
|
||||
### Online Serving
|
||||
|
||||
To enable sleep mode in a vLLM server you need to initialize it with the flag `VLLM_SERVER_DEV_MODE=1` and pass `--enable-sleep-mode` to the vLLM server.
|
||||
|
||||
Reference in New Issue
Block a user