[KV] Restore PP fix #54793 -- no KV tensors for another stage's layers
A KV cache group with no layer on a pipeline stage was still given tensors there, and allocate_kv_cache raised StopIteration on the second stage. The fix was lost in the v0.30.0 rebase. Test drives the real get_kv_cache_configs with the production layout. Co-Authored-By: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
@@ -1806,7 +1806,14 @@ def get_kv_cache_config_from_groups(
|
||||
layers_by_spec: defaultdict[KVCacheSpec, list[str]] = defaultdict(list)
|
||||
if isinstance(group_spec, UniformTypeKVCacheSpecs):
|
||||
for layer_name, spec in group_spec.kv_cache_specs.items():
|
||||
layers_by_spec[spec].append(layer_name)
|
||||
# Fork (E2 PP fix #54793, dropped in the v0.30.0 rebase, restored
|
||||
# the vLLM 0.30 rebase): _project_kv_cache_groups_to_worker keeps the GLOBAL
|
||||
# UniformTypeKVCacheSpecs for a group with no layer on this PP
|
||||
# stage, so without this filter the worker gets a tensor for
|
||||
# another stage's layer and allocate_kv_cache's group lookup
|
||||
# raises StopIteration (Flash-Next PP1: the layer-2 PLE state).
|
||||
if layer_name in group.layer_names:
|
||||
layers_by_spec[spec].append(layer_name)
|
||||
elif group.layer_names:
|
||||
layers_by_spec[group_spec].extend(group.layer_names)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user