[Sleep] Retain draft-private weights with existing parameter selector (#61046)

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
aoshen02
2026-10-11 23:30:56 +08:00
committed by GitHub
co-authored by Codex
parent 3e8e1364ff
commit 239a3fa3a8
4 changed files with 97 additions and 6 deletions
+36 -3
View File
@@ -77,8 +77,15 @@ llm.wake_up(tags=["kv_cache"])
#### Retaining frozen weights during RLHF updates
Set `sleep_preserve_parameter_names` (CLI: `--sleep-preserve-parameter-names`)
to runtime parameter-name glob
patterns for weights that stay frozen during training. Each pattern must match
to runtime parameter-name glob patterns for weights that stay frozen during
training. It covers two use cases:
- **Frozen Engram tables (original use case):** retain table weights and scales
omitted by the trainer, without retransmitting them after every sleep.
- **Frozen draft weights:** retain draft-private parameters when the trainer
updates only the target model.
Target patterns match
`model.named_parameters()`; checkpoint names and `requires_grad` are not used.
Level-2 sleep backs up selected GPU parameters to pageable CPU memory; weights
@@ -87,7 +94,33 @@ copy. Allow enough host memory for backups. Level-1 behavior is unchanged.
The trainer must omit these parameters from updates, and reload/post-processing
must preserve their values and storage (including avoiding replacement with meta
tensors). This option does not filter updates and applies only to the target model.
tensors). This option does not filter updates.
For frozen Engram tables omitted by the trainer, select the target table weights
and scales. For example, DeepSeek V4.1 uses these runtime parameter names:
```bash
VLLM_SERVER_DEV_MODE=1 vllm serve deepseek-ai/DeepSeek-V4.1-Flash --enable-sleep-mode \
--sleep-preserve-parameter-names '*.engram.embed_tokens.weight*'
```
CPU-offloaded tables remain on CPU without an extra backup copy. Engram's
[inference-time CPU offload](engram.md#cpu-offload) is configured separately;
this option preserves the frozen tables across level-2 sleep and weight updates.
To retain a frozen drafter while updating only target weights, use `draft.*`:
```bash
VLLM_SERVER_DEV_MODE=1 vllm serve Qwen/Qwen3.8-27B --enable-sleep-mode \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--sleep-preserve-parameter-names 'draft.*'
```
vLLM selects all draft-private parameters internally; you do not need to list
them individually. Use `draft.<name>` to select specific draft parameters.
Parameters shared with the target are excluded from the draft selection, so
target updates remain authoritative. An empty `draft.*` selection is valid.
The Engram and draft patterns can be supplied together.
#### Release only KV cache memory