[KV] Restore PP fix #54795 -- PP stages need only share a KV layout
Co-Authored-By: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
@@ -264,8 +264,9 @@ def resolve_kv_cache_layout(
|
||||
"""Resolve one KV cache layout for the whole model.
|
||||
|
||||
Runs once in the engine core. Every worker reports the layouts its backends
|
||||
support, most preferred first (``get_supported_kv_cache_layouts``); all
|
||||
ranks run the same backends, so their lists must agree. Specs mixing HNC
|
||||
support, most preferred first (``get_supported_kv_cache_layouts``); ranks
|
||||
may expose different sets under pipeline parallelism, but their lists must
|
||||
share at least one layout. Specs mixing HNC
|
||||
shapes narrow the candidates to block-compact layouts. An explicit
|
||||
``VLLM_KV_CACHE_LAYOUT`` must be one of the candidates or resolution fails,
|
||||
with the legacy ``NHD``/``HND`` names as aliases for ``LBNHC``/``LBHNC``; the
|
||||
@@ -282,10 +283,20 @@ def resolve_kv_cache_layout(
|
||||
assert supported_layouts and all(supported_layouts), (
|
||||
"No worker reported supported KV cache layouts."
|
||||
)
|
||||
assert all(names == supported_layouts[0] for names in supported_layouts[1:]), (
|
||||
f"Workers disagree on supported KV cache layouts: {supported_layouts}."
|
||||
)
|
||||
candidates = [_layout_from_name(name) for name in supported_layouts[0]]
|
||||
# Fork: restore PP fix #54795 from the E2 base, dropped
|
||||
# in the v0.30.0 rebase. Under pipeline parallelism the stages run
|
||||
# different backends (PP0 carries the PLE short-conv, PP1 the drafter), so
|
||||
# their lists need only share a layout, not be identical.
|
||||
common_layout_names = [
|
||||
name
|
||||
for name in supported_layouts[0]
|
||||
if all(name in names for names in supported_layouts[1:])
|
||||
]
|
||||
if not common_layout_names:
|
||||
raise ValueError(
|
||||
f"Workers share no supported KV cache layout: {supported_layouts}."
|
||||
)
|
||||
candidates = [_layout_from_name(name) for name in common_layout_names]
|
||||
|
||||
# A block-compact layout means the block is densely packed in memory, so any mix of
|
||||
# specs can re-interpret HNC with different sizes as long as the total number of
|
||||
|
||||
Reference in New Issue
Block a user