[BugFix][Model] Jamba - Handle aborted requests, Add tests and fix cleanup bug (#6425)

Co-authored-by: Mor Zusman <morz@ai21.com>
This commit is contained in:
Mor Zusman
2024-07-16 01:32:55 +00:00
committed by GitHub
co-authored by Mor Zusman
parent d6f3b3d5c4
commit 9ad32dacd9
5 changed files with 176 additions and 24 deletions
+1
View File
@@ -374,6 +374,7 @@ class Scheduler:
for aborted_group in aborted_groups:
# Remove the sequence group from the state queue.
state_queue.remove(aborted_group)
self._finished_requests_ids.append(aborted_group.request_id)
for seq in aborted_group.get_seqs():
if seq.is_finished():
continue