[Bugfix] Disable sequence parallelism / async TP under batch invariance and add a TP regression test (#56377)
Signed-off-by: LioEinaudi <zhao3024667639@gmail.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent
7a877ae34d
commit
0c87a197b8
@@ -144,7 +144,7 @@ When batch invariance is enabled, vLLM:
|
||||
|
||||
1. Uses deterministic kernel implementations for attention and other operations
|
||||
2. Ensures consistent numerical behavior across different batch sizes
|
||||
3. Disables certain optimizations that may introduce non-determinism (such as custom all-reduce operations in tensor parallel mode)
|
||||
3. Disables certain optimizations that may introduce non-determinism (such as custom all-reduce operations in tensor parallel mode, and sequence parallelism / async TP, whose reduce-scatter path is not batch-invariant)
|
||||
4. On CUDA devices with tuned matmul table entries for the model's bf16 unquantized forward linear layers (Ada, Hopper, Blackwell),
|
||||
runs without `torch.compile` using breakable CUDA graphs so tile configs follow the runtime batch size; set `VLLM_USE_BREAKABLE_CUDAGRAPH=0` to opt out
|
||||
(not applied when sequence parallelism / async TP are enabled, since those are `torch.compile` passes).
|
||||
|
||||
@@ -1718,6 +1718,26 @@ class VllmConfig:
|
||||
)
|
||||
self.compilation_config.mode = CompilationMode.NONE
|
||||
|
||||
# TODO: This is a stopgap that turns sequence parallelism / async TP
|
||||
# off under batch invariance. Make the sequence-parallel reduce-scatter
|
||||
# path batch-invariant instead so they can stay enabled (#56370).
|
||||
# Sequence parallelism / async TP rewrite all_reduce + rms_norm into a
|
||||
# reduce-scatter whose reduction order depends on the batch, so they
|
||||
# are not batch-invariant. Decide this before the breakable-CUDA-graph
|
||||
# default below, which declines to auto-enable when they are on.
|
||||
pass_config = self.compilation_config.pass_config
|
||||
if envs.VLLM_BATCH_INVARIANT and (
|
||||
pass_config.enable_sp or pass_config.fuse_gemm_comms
|
||||
):
|
||||
logger.warning_once(
|
||||
"Disabling sequence parallelism and async TP "
|
||||
"(pass_config.enable_sp / fuse_gemm_comms) when "
|
||||
"VLLM_BATCH_INVARIANT is enabled: the reduce-scatter path "
|
||||
"is not batch-invariant (see vllm-project/vllm#56370)."
|
||||
)
|
||||
pass_config.enable_sp = False
|
||||
pass_config.fuse_gemm_comms = False
|
||||
|
||||
breakable_cudagraph_enabled = self._maybe_enable_breakable_cudagraph()
|
||||
|
||||
if not breakable_cudagraph_enabled and (
|
||||
|
||||
Reference in New Issue
Block a user