[Qwen4Exp] Serve PLE rows from the loader's mapped checkpoint shards (PR #58815)
Adds an opt-in backend, --engram-config '{"host_file_gather": true}', that
serves Qwen4Exp PLE rows from the checkpoint files through the page cache
instead of allocating the 47.7 GiB table on GPU or in pinned host memory.
- Qwen4ExpPLEFileGatherEmbedding allocates no table rows; load_weights keeps
the default loader's zero-copy safetensors views of each shard. At bind
(post-load hook) every view must be file-backed per /proc/self/maps (HF
cache symlinks work; heap/pinned/anonymous memory is rejected) and the
shards must cover the whole table.
- Each step, Qwen4ExpModelState.prepare_inputs computes the PLE n-gram ids,
copies them to pinned host memory, syncs the stream once, issues
POSIX_FADV_WILLNEED over the coalesced page ranges of the step's rows,
gathers the rows on the host, and copies them into a fixed device staging
buffer. The forward reads only the staging buffer, so FULL CUDA graph
capture is unaffected.
- Rows stay in the checkpoint dtype (FP8 table stays FP8, dequantized on the
GPU); dummy loads serve zeros; weight reload is rejected; takes precedence
over cpu_offload; rejected on ROCm and with elastic EP.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
+37
-7
@@ -83,10 +83,7 @@ directly from the Triton kernel through unified virtual addressing (UVA),
|
||||
prefetching on a side CUDA stream so transfers overlap with the compute of
|
||||
preceding layers. The freed GPU memory goes to the KV cache.
|
||||
|
||||
`cpu_offload` defaults to `true` (on): it follows the
|
||||
`VLLM_PLE_CPU_OFFLOAD` environment variable, which also defaults to `true`.
|
||||
An explicit `--engram-config` value takes precedence over the environment
|
||||
variable.
|
||||
`cpu_offload` defaults to `true` and is set only through `--engram-config`.
|
||||
|
||||
To keep the tables resident on GPU instead (e.g. when host memory is
|
||||
scarce):
|
||||
@@ -96,8 +93,40 @@ vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
|
||||
--engram-config.cpu_offload false
|
||||
```
|
||||
|
||||
or equivalently `VLLM_PLE_CPU_OFFLOAD=0`. CPU offload requires a GPU with UVA
|
||||
support; vLLM fails fast when it is unavailable.
|
||||
CPU offload requires a GPU with UVA support; vLLM fails fast when it is
|
||||
unavailable.
|
||||
|
||||
## Host file gather (Qwen4Exp)
|
||||
|
||||
`host_file_gather` serves Qwen4Exp PLE rows from the loader's own file
|
||||
mapping of the safetensors shards instead of allocating the table. Set
|
||||
`--engram-config '{"host_file_gather": true}'`. It applies only to Qwen4Exp
|
||||
and takes precedence over `cpu_offload`.
|
||||
|
||||
Rows stay in the checkpoint dtype: an FP8 table stays FP8 and is dequantized
|
||||
on the GPU, and an unquantized table must match `--dtype`. CPU offload
|
||||
converts on load; this mode rejects the mismatch at load.
|
||||
|
||||
Faulted pages are reclaimable page cache, shared across co-located processes.
|
||||
They appear in each process's RSS as file-backed (`RssFile`), and PSS is
|
||||
where they are split across processes. A write to the mapping copies that
|
||||
page into anonymous RSS. Each step pays a host sync, and every TP rank
|
||||
gathers all rows. Co-located ranks share the cache; ranks on different
|
||||
machines do not.
|
||||
|
||||
Serving depends on the checkpoint files staying unchanged while the server
|
||||
runs: an in-place overwrite serves the new bytes, truncation kills the worker
|
||||
with SIGBUS, and deleting or renaming the files is safe.
|
||||
|
||||
The first touch of a row costs a disk read, and a page evicted under memory
|
||||
pressure is read again on the next miss. By default no prewarm is needed; for
|
||||
a fast first request, read the PLE files once after the server has started
|
||||
and allocated its KV cache.
|
||||
|
||||
Reload is unsupported and fails closed. ROCm and elastic EP reject the key.
|
||||
Load paths that do not hand over file-backed safetensors tensors fail at bind
|
||||
(`eager`, `.bin`, runai, `fastsafetensors`, `instanttensor`). The multithread
|
||||
loader is file-backed and is admitted. `--load-format dummy` serves zeros.
|
||||
|
||||
## Data-parallel topologies (DeepSeek V4.1)
|
||||
|
||||
@@ -118,7 +147,8 @@ vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
|
||||
```
|
||||
|
||||
`embedding_across_dp` is not supported with elastic expert parallelism yet.
|
||||
Qwen4Exp PLE tables are ETP-sharded instead and honor `cpu_offload` only.
|
||||
Qwen4Exp PLE tables are ETP-sharded instead and honor only `cpu_offload` and
|
||||
`host_file_gather`.
|
||||
|
||||
## Sharing host tables across DP replicas
|
||||
|
||||
|
||||
Reference in New Issue
Block a user