[FIX] Fix Alibi implementation in PagedAttention kernel (#945)

* [FIX] Fix Alibi implementation in PagedAttention kernel

* Fix test_attention

* Fix

---------

Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Oliver-ss <yuansongwx@outlook.com>
This commit is contained in:
Zhuohan Li
2023-09-07 15:53:14 -07:00
committed by GitHub
co-authored by Woosuk Kwon Oliver-ss
parent c957c741d9
commit db09d4ad83
2 changed files with 4 additions and 3 deletions
+1 -1
View File
@@ -178,7 +178,7 @@ __global__ void single_query_cached_kv_attention_kernel(
// This includes a reduction across the threads in the same thread group.
float qk = scale * Qk_dot<scalar_t, THREAD_GROUP_SIZE>::dot(q_vecs[thread_group_offset], k_vecs);
// Add the ALiBi bias if slopes are given.
qk += (alibi_slope != 0) ? alibi_slope * (token_idx - context_len) : 0;
qk += (alibi_slope != 0) ? alibi_slope * (token_idx - context_len + 1) : 0;
if (thread_group_offset == 0) {
// Store the partial reductions to shared memory.