[Core][Mamba] Remove leftovers of mamba_cache_mode="all" (#60043)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> Signed-off-by: Jiangyun Zhu <zhujiangyun@inferact.ai> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent
bc3ddb198a
commit
2cc2e68d1c
@@ -15,23 +15,22 @@ Set `enable_prefix_caching=True` in vLLM engine to enable APC. Here is an exampl
|
||||
|
||||
## Hybrid Mamba models
|
||||
|
||||
Under `--mamba-cache-mode align`, Mamba state is stored only on the Mamba block grid, so a prefix-cache hit can resume only at a block boundary. `--enable-mamba-shared-prefix-checkpoint` also stores a checkpoint at the shared-prefix junction, the point where an earlier request with the same prefix stopped. Requests whose shared prefix ends inside a block can then reuse it.
|
||||
With prefix caching enabled, Mamba state is stored only on the Mamba block grid, so a prefix-cache hit can resume only at a block boundary. `--enable-mamba-shared-prefix-checkpoint` also stores a checkpoint at the shared-prefix junction, the point where an earlier request with the same prefix stopped. Requests whose shared prefix ends inside a block can then reuse it.
|
||||
|
||||
This helps when many requests share a long system prompt and then diverge. It is off by default, and takes effect only when all of the following hold:
|
||||
|
||||
- `--mamba-cache-mode align`
|
||||
- prefix caching enabled
|
||||
- EAGLE/MTP speculative decoding on the Mamba group
|
||||
- `--prefix-match-unit` smaller than the Mamba block size
|
||||
- the model does not use multi-module MTP
|
||||
|
||||
```bash
|
||||
vllm serve <hybrid-model> \
|
||||
--mamba-cache-mode align \
|
||||
--prefix-match-unit 64 \
|
||||
--enable-mamba-shared-prefix-checkpoint
|
||||
```
|
||||
|
||||
`--prefix-match-unit` is required. It sets the granularity at which prefix-cache keys are computed. When unset it defaults to the greatest common divisor of the prefix-cacheable KV cache group block sizes. Under `align` that is the block size itself, so no sub-block boundary exists and the flag has no effect.
|
||||
`--prefix-match-unit` is required. It sets the granularity at which prefix-cache keys are computed. When unset it defaults to the greatest common divisor of the prefix-cacheable KV cache group block sizes. For hybrid Mamba models that is the block size itself, so no sub-block boundary exists and the flag has no effect.
|
||||
|
||||
Choose a value that divides the block size of every prefix-cacheable KV cache group, and that is a multiple of the per-state compression ratio for models that use one, such as sparse MLA. vLLM validates both at startup and names the offending sizes in the error. Read the served block size from the startup log. 64 is a reasonable starting point.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user