Chore: Fix minor doc sentence, grammar, quote errors (#40469)
Signed-off-by: Ashwin Phadke <23502062+ashwin-phadke@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
This commit is contained in:
co-authored by
Harry Mellor
parent
3c43237233
commit
d511b5bae9
@@ -159,7 +159,7 @@ For simplicity, we assume `block_size=1` in this section.
|
||||
|
||||
### High level idea
|
||||
|
||||
The block pool uses a dict similar to `tuple(block_hash, group_id) -> block` to catch the full blocks. That means the same tokens of different groups are cached and evicted independently.
|
||||
The block pool uses a dict similar to `tuple(block_hash, group_id) -> block` to cache the full blocks. That means the same tokens of different groups are cached and evicted independently.
|
||||
|
||||
When a new request comes in, we check the cache hit prefix of each group, and return the intersection of these groups as the cached prefix of the request. See below for the detailed algorithm for checking the cache hit of one group & performing the intersection.
|
||||
|
||||
|
||||
@@ -52,7 +52,7 @@ __device__ void paged_attention_kernel(
|
||||
)
|
||||
```
|
||||
|
||||
There are also a list of template arguments above the function
|
||||
There is also a list of template arguments above the function
|
||||
signature that are determined during compilation time. `scalar_t`
|
||||
represents the data type of the query, key, and value data elements,
|
||||
such as FP16. `HEAD_SIZE` indicates the number of elements in each
|
||||
@@ -178,7 +178,7 @@ const scalar_t* k_ptr = k_cache + physical_block_number * kv_block_stride
|
||||
+ physical_block_offset * x;
|
||||
```
|
||||
|
||||
Unlike to `q_ptr`, `k_ptr` in each thread will point to different
|
||||
Unlike `q_ptr`, `k_ptr` in each thread will point to different
|
||||
key token at different iterations. As shown above, that `k_ptr`
|
||||
points to key token data based on `k_cache` at assigned block,
|
||||
assigned head and assigned token.
|
||||
|
||||
@@ -156,7 +156,7 @@ class EngineCoreEventType(enum.IntEnum):
|
||||
class EngineCoreEvent(msgspec.Struct):
|
||||
"""A timestamped engine core event associated with a request.
|
||||
|
||||
The timestamp is a monotonic timestamps and is used for by the engine
|
||||
The timestamp is a monotonic timestamp and is used by the engine
|
||||
frontend to calculate intervals between engine core events. These
|
||||
timestamps should not be compared with timestamps from other processes.
|
||||
"""
|
||||
|
||||
Reference in New Issue
Block a user