[Bugfix][V1] Free out-of-window blocks on the processed-token basis under async scheduling (#47728)
Signed-off-by: Saddss <28726669061@qq.com> Signed-off-by: Nick Hill <nickhill123@gmail.com> Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: Saddss <28726669061@qq.com> Co-authored-by: Nick Hill <nickhill123@gmail.com>
This commit is contained in:
co-authored by
Yifan Qiao
Saddss
Nick Hill
parent
934eeaecfb
commit
cd0de48d08
@@ -81,7 +81,7 @@ def assert_encoder_kv_cache_spec(engine: LLM) -> None:
|
||||
assert spec.sliding_window == cdiv(750, 4) + 1 == 189
|
||||
assert (
|
||||
spec.max_admission_blocks_per_request(
|
||||
max_num_batched_tokens=1,
|
||||
max_in_flight_tokens=1,
|
||||
max_model_len=vllm_config.model_config.max_model_len,
|
||||
)
|
||||
== 13
|
||||
|
||||
Reference in New Issue
Block a user