diff --git a/docs/design/hybrid_kv_cache_manager.md b/docs/design/hybrid_kv_cache_manager.md index 82d54e9b5c..9891c82f97 100644 --- a/docs/design/hybrid_kv_cache_manager.md +++ b/docs/design/hybrid_kv_cache_manager.md @@ -115,9 +115,10 @@ Unfortunately, not all models have such a beautiful ratio, and approach in Case - Group 0: 10 full attention layers (full.0 - full.9) - Group 1: 10 sliding window attention layers (sw.0 - sw.9) - Group 2: 10 sliding window attention layers (sw.10 - sw.19) -- ... -- Group 6: 10 sliding window attention layers (sw.40 - sw.49) -- Group 7: 2 sliding window attention layers (sw.50 - sw.51) and 8 padding layers +- Group 3: 10 sliding window attention layers (sw.20 - sw.29) +- Group 4: 10 sliding window attention layers (sw.30 - sw.39) +- Group 5: 10 sliding window attention layers (sw.40 - sw.49) +- Group 6: 2 sliding window attention layers (sw.50 - sw.51) and 8 padding layers We will update this algorithm if this heuristic leads to a bad result when a new model comes out (e.g., 20 full + 30 sw, the group size should be 10 instead of 20).