MOVZXandHinata
50dabc9989
attach_deferred_state raised 'deferred model state for a step without
sampled outputs' when a non-last PP rank produced a PLE deferred state but
receive() left the step slot None (no row samples). With num_running_reqs
> 1 a prefill-only step appears (a 2048-token chunk for one request,
num_output_tokens=0, scheduled_spec_decode_tokens={}), and the raise
killed the engine every time. Confirmed in the container log: the same
PLE fatal at sample_tokens on sequence 101, 1766, 3150, 3264.
The deferred state has no consumer for a None slot (get_prev_sampled_outputs
returns None) and references the KV cache it was built for, so drop it to
release the reference. This is a prefill-only step under concurrent load,
not a fault.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
46 lines
2.1 KiB
Diff
46 lines
2.1 KiB
Diff
From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
|
|
From: MOVZX <movzx@yahoo.com>
|
|
Date: Sat, 17 Oct 2026 21:58:00 +0000
|
|
Subject: [PATCH 106/106] ple: drop the deferred state on a prefill-only step
|
|
(no crash)
|
|
|
|
attach_deferred_state raised 'deferred model state for a step without
|
|
sampled outputs' when a non-last PP rank produced a PLE deferred state but
|
|
receive() left the step slot None (no row samples). With num_running_reqs
|
|
> 1 a prefill-only step appears (a 2048-token chunk for one request,
|
|
num_output_tokens=0, scheduled_spec_decode_tokens={}), and the raise
|
|
killed the engine every time. Confirmed in the container log: the same
|
|
PLE fatal at sample_tokens on sequence 101, 1766, 3150, 3264.
|
|
|
|
The deferred state has no consumer for a None slot (get_prev_sampled_outputs
|
|
returns None) and references the KV cache it was built for, so drop it to
|
|
release the reference. This is a prefill-only step under concurrent load,
|
|
not a fault.
|
|
|
|
Co-Authored-By: Hinata <hinata@ihoree.local>
|
|
---
|
|
vllm/v1/worker/gpu/pp_utils.py | 10 +++++++---
|
|
1 file changed, 7 insertions(+), 3 deletions(-)
|
|
|
|
diff --git a/vllm/v1/worker/gpu/pp_utils.py b/vllm/v1/worker/gpu/pp_utils.py
|
|
index 0a54342..5b9cb07 100644
|
|
--- a/vllm/v1/worker/gpu/pp_utils.py
|
|
+++ b/vllm/v1/worker/gpu/pp_utils.py
|
|
@@ -173,10 +173,12 @@ class PPHandler:
|
|
slot = self.queue[-1]
|
|
if slot is None:
|
|
# receive() skipped the broadcast: no row of this step samples, so
|
|
- # no row can have a speculative window to commit.
|
|
- raise RuntimeError(
|
|
- "deferred model state for a step without sampled outputs"
|
|
- )
|
|
+ # no row can have a speculative window to commit. The deferred state
|
|
+ # (PLE model state for the next step) has no consumer this step and
|
|
+ # references the KV cache it was built for, so drop it to release
|
|
+ # the reference. This happens on a prefill-only step under
|
|
+ # concurrent load (num_running_reqs > 1); it is not a fault.
|
|
+ return
|
|
self.queue[-1] = replace(slot, deferred_state=state)
|
|
|
|
def drop_deferred_states(self) -> int:
|