[Misc] Rename --enable-mamba-fine-grained-prefix-cache (#53945 follow-up) (#57382)

Signed-off-by: Roi Koren <roik@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
roikoren755
2026-09-20 09:50:34 -07:00
committed by GitHub
co-authored by mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent 7d99c2c4fd
commit 49ee12d742
10 changed files with 38 additions and 38 deletions
+2 -2
View File
@@ -15,7 +15,7 @@ Set `enable_prefix_caching=True` in vLLM engine to enable APC. Here is an exampl
## Hybrid Mamba models
Under `--mamba-cache-mode align`, Mamba state is stored only on the Mamba block grid, so a prefix-cache hit can resume only at a block boundary. `--enable-mamba-fine-grained-prefix-cache` also stores a checkpoint at the shared-prefix junction, the point where an earlier request with the same prefix stopped. Requests whose shared prefix ends inside a block can then reuse it.
Under `--mamba-cache-mode align`, Mamba state is stored only on the Mamba block grid, so a prefix-cache hit can resume only at a block boundary. `--enable-mamba-shared-prefix-checkpoint` also stores a checkpoint at the shared-prefix junction, the point where an earlier request with the same prefix stopped. Requests whose shared prefix ends inside a block can then reuse it.
This helps when many requests share a long system prompt and then diverge. It is off by default, and takes effect only when all of the following hold:
@@ -28,7 +28,7 @@ This helps when many requests share a long system prompt and then diverge. It is
vllm serve <hybrid-model> \
--mamba-cache-mode align \
--prefix-match-unit 64 \
--enable-mamba-fine-grained-prefix-cache
--enable-mamba-shared-prefix-checkpoint
```
`--prefix-match-unit` is required. It sets the granularity at which prefix-cache keys are computed. When unset it defaults to the greatest common divisor of the prefix-cacheable KV cache group block sizes. Under `align` that is the block size itself, so no sub-block boundary exists and the flag has no effect.