diff --git a/docs/design/hybrid_kv_cache_manager.md b/docs/design/hybrid_kv_cache_manager.md index 8f17b473ad..82d54e9b5c 100644 --- a/docs/design/hybrid_kv_cache_manager.md +++ b/docs/design/hybrid_kv_cache_manager.md @@ -159,7 +159,7 @@ For simplicity, we assume `block_size=1` in this section. ### High level idea -The block pool uses a dict similar to `tuple(block_hash, group_id) -> block` to catch the full blocks. That means the same tokens of different groups are cached and evicted independently. +The block pool uses a dict similar to `tuple(block_hash, group_id) -> block` to cache the full blocks. That means the same tokens of different groups are cached and evicted independently. When a new request comes in, we check the cache hit prefix of each group, and return the intersection of these groups as the cached prefix of the request. See below for the detailed algorithm for checking the cache hit of one group & performing the intersection. diff --git a/docs/design/paged_attention.md b/docs/design/paged_attention.md index 7c0132cd2a..f4742c7faa 100644 --- a/docs/design/paged_attention.md +++ b/docs/design/paged_attention.md @@ -52,7 +52,7 @@ __device__ void paged_attention_kernel( ) ``` -There are also a list of template arguments above the function +There is also a list of template arguments above the function signature that are determined during compilation time. `scalar_t` represents the data type of the query, key, and value data elements, such as FP16. `HEAD_SIZE` indicates the number of elements in each @@ -178,7 +178,7 @@ const scalar_t* k_ptr = k_cache + physical_block_number * kv_block_stride + physical_block_offset * x; ``` -Unlike to `q_ptr`, `k_ptr` in each thread will point to different +Unlike `q_ptr`, `k_ptr` in each thread will point to different key token at different iterations. As shown above, that `k_ptr` points to key token data based on `k_cache` at assigned block, assigned head and assigned token. diff --git a/vllm/v1/engine/__init__.py b/vllm/v1/engine/__init__.py index a04f080ea6..38ca8dc6da 100644 --- a/vllm/v1/engine/__init__.py +++ b/vllm/v1/engine/__init__.py @@ -156,7 +156,7 @@ class EngineCoreEventType(enum.IntEnum): class EngineCoreEvent(msgspec.Struct): """A timestamped engine core event associated with a request. - The timestamp is a monotonic timestamps and is used for by the engine + The timestamp is a monotonic timestamp and is used by the engine frontend to calculate intervals between engine core events. These timestamps should not be compared with timestamps from other processes. """