[Bugfix][Spec Decode] Cache the Mamba state at the block-grid position of EAGLE resume (#53945)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Signed-off-by: Adam Shaver <ashaver@nvidia.com> Signed-off-by: akshaver <168006157+akshaver@users.noreply.github.com> Co-authored-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: roikoren755 <26850796+roikoren755@users.noreply.github.com>
This commit is contained in:
co-authored by
wzhao18
Claude Opus 5
roikoren755
parent
414057a3d3
commit
263c4ff95f
@@ -13,6 +13,28 @@ Set `enable_prefix_caching=True` in vLLM engine to enable APC. Here is an exampl
|
||||
|
||||
[examples/features/automatic_prefix_caching/automatic_prefix_caching_offline.py](../../examples/features/automatic_prefix_caching/automatic_prefix_caching_offline.py)
|
||||
|
||||
## Hybrid Mamba models
|
||||
|
||||
Under `--mamba-cache-mode align`, Mamba state is stored only on the Mamba block grid, so a prefix-cache hit can resume only at a block boundary. `--enable-mamba-fine-grained-prefix-cache` also stores a checkpoint at the shared-prefix junction, the point where an earlier request with the same prefix stopped. Requests whose shared prefix ends inside a block can then reuse it.
|
||||
|
||||
This helps when many requests share a long system prompt and then diverge. It is off by default, and takes effect only when all of the following hold:
|
||||
|
||||
- `--mamba-cache-mode align`
|
||||
- EAGLE/MTP speculative decoding on the Mamba group
|
||||
- `--prefix-match-unit` smaller than the Mamba block size
|
||||
- the model does not use multi-module MTP
|
||||
|
||||
```bash
|
||||
vllm serve <hybrid-model> \
|
||||
--mamba-cache-mode align \
|
||||
--prefix-match-unit 64 \
|
||||
--enable-mamba-fine-grained-prefix-cache
|
||||
```
|
||||
|
||||
`--prefix-match-unit` is required. It sets the granularity at which prefix-cache keys are computed. When unset it defaults to the greatest common divisor of the prefix-cacheable KV cache group block sizes. Under `align` that is the block size itself, so no sub-block boundary exists and the flag has no effect.
|
||||
|
||||
Choose a value that divides the block size of every prefix-cacheable KV cache group, and that is a multiple of the per-state compression ratio for models that use one, such as sparse MLA. vLLM validates both at startup and names the offending sizes in the error. Read the served block size from the startup log. 64 is a reasonable starting point.
|
||||
|
||||
## Example workloads
|
||||
|
||||
We describe two example workloads, where APC can provide huge performance benefit:
|
||||
|
||||
Reference in New Issue
Block a user