[BugFix] Eagerly abort cancelled final-step requests (#29987)
Currently, when requests are cancelled while executing their final step, "completion" is handled based on normal stop processing (e.g. length or stop token), so the abort has no effect. This is typically not a problem, but when a kv connector is involved it thinks the request completed successfully rather than being aborted. This is problematic for disaggregated prefill which will free kv cache blocks if the request was aborted but not if it completed successfully—since the cancelled request will never be sent to the decode side, kv cache blocks remain pinned until the fall-back timeout expires. The problem is exacerbated when many requests are cancelled and/or there are large prefills whose forward pass takes a long time (since the window is bigger). This PR fixes the problem by processing pending aborts immediately prior to processing model output each step; we process only aborts, not new requests, since it's preferable for latency to process model outputs before new incoming requests. Fixes #26400. Signed-off-by: Nick Hill <nhill@redhat.com>
This commit is contained in:
@@ -51,7 +51,6 @@ from vllm.v1.outputs import (
|
||||
ModelRunnerOutput,
|
||||
)
|
||||
from vllm.v1.utils import report_usage_stats
|
||||
from vllm.v1.worker.gpu_model_runner import GPUModelRunner
|
||||
from vllm.v1.worker.utils import is_residual_scattered_for_sp
|
||||
from vllm.v1.worker.worker_base import WorkerBase
|
||||
|
||||
@@ -59,6 +58,7 @@ logger = init_logger(__name__)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from vllm.model_executor.model_loader.tensorizer import TensorizerConfig
|
||||
from vllm.v1.worker.gpu_model_runner import GPUModelRunner
|
||||
|
||||
|
||||
class Worker(WorkerBase):
|
||||
@@ -259,7 +259,11 @@ class Worker(WorkerBase):
|
||||
self.vllm_config, self.device
|
||||
)
|
||||
else:
|
||||
self.model_runner = GPUModelRunner(self.vllm_config, self.device)
|
||||
from vllm.v1.worker.gpu_model_runner import (
|
||||
GPUModelRunner as GPUModelRunnerV1,
|
||||
)
|
||||
|
||||
self.model_runner = GPUModelRunnerV1(self.vllm_config, self.device)
|
||||
|
||||
if self.rank == 0:
|
||||
# If usage stat is enabled, collect relevant info.
|
||||
@@ -556,7 +560,7 @@ class Worker(WorkerBase):
|
||||
and forward_pass
|
||||
):
|
||||
# currently only supported by V1 GPUModelRunner
|
||||
assert isinstance(self.model_runner, GPUModelRunner)
|
||||
assert not self.use_v2_model_runner
|
||||
num_scheduled_tokens_np = np.array(
|
||||
list(scheduler_output.num_scheduled_tokens.values()),
|
||||
dtype=np.int32,
|
||||
|
||||
Reference in New Issue
Block a user