[Bugfix][KV Offload] Bound unaligned SWA loads by physical GPU blocks (#49052)
Signed-off-by: Colton Ottley <colton@ottleyengineering.com> Co-authored-by: Colton Ottley <colton@ottleyengineering.com> Co-authored-by: jasl <jasl9187@hotmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Or Ozeri <oro@il.ibm.com>
This commit is contained in:
co-authored by
Colton Ottley
jasl
Claude Opus 4.8
Or Ozeri
parent
da3a252fd1
commit
5559679229
@@ -860,6 +860,7 @@ class OffloadingConnectorScheduler:
|
||||
num_pending_gpu_blocks
|
||||
<= group_config.sliding_window_size_in_chunks
|
||||
* self.config.blocks_per_chunk
|
||||
+ 1
|
||||
)
|
||||
|
||||
num_chunks = cdiv(num_cached_tokens, tokens_per_chunk)
|
||||
|
||||
Reference in New Issue
Block a user