[Bugfix] OffloadingConnector: respect skip_reading_prefix_cache flag (#44592)
Signed-off-by: Hsiao-Yuan Chen <hy.c@Hsiao-YuandeMacBook-Pro.local> Signed-off-by: littlecircle0730 <littlecircle0730@gmail.com> Signed-off-by: littlecircle0730 <43994952+littlecircle0730@users.noreply.github.com> Co-authored-by: Hsiao-Yuan Chen <hy.c@Hsiao-YuandeMacBook-Pro.local> Co-authored-by: Or Ozeri <or@ozery.com>
This commit is contained in:
co-authored by
Hsiao-Yuan Chen
Or Ozeri
parent
e0871ad225
commit
4bc83323f2
@@ -571,7 +571,11 @@ class OffloadingConnectorScheduler:
|
||||
req_status.update_offload_keys()
|
||||
req_status.num_locally_computed_tokens = num_computed_tokens
|
||||
|
||||
num_hit_tokens = self._lookup(req_status)
|
||||
num_hit_tokens: int | None
|
||||
if request.skip_reading_prefix_cache:
|
||||
num_hit_tokens = 0
|
||||
else:
|
||||
num_hit_tokens = self._lookup(req_status)
|
||||
req_status.update_num_hit_blocks(num_computed_tokens + (num_hit_tokens or 0))
|
||||
|
||||
self._touch(req_status)
|
||||
|
||||
Reference in New Issue
Block a user