[Compile] Remove torch.compile-based sequence parallelism and AsyncTP (#60517)
This commit is contained in:
@@ -225,9 +225,9 @@ outputs = model.generate(
|
||||
)
|
||||
```
|
||||
|
||||
### Piecewise compilation and full graph custom passes (attention fusion, sequence parallelism)
|
||||
### Piecewise compilation and full graph custom passes (attention fusion)
|
||||
|
||||
Unfortunately, some custom compile passes have to see the whole graph to be effective and hence aren't compatible with piecewise compilation. This includes `AttnQuantFusionPass` and `SequenceParallelismPass`. As a short-term solution, we automatically disable piecewise compilation (by setting `splitting_ops=[]`) when attention fusion is enabled. We use CUDA Graph modes `FULL` or `FULL_DECODE_ONLY` (depending on backend support). However, this leads to another optimization incompatibility and confusing performance tradeoffs.
|
||||
Unfortunately, some custom compile passes have to see the whole graph to be effective and hence aren't compatible with piecewise compilation. This includes `AttnQuantFusionPass`. As a short-term solution, we automatically disable piecewise compilation (by setting `splitting_ops=[]`) when attention fusion is enabled. We use CUDA Graph modes `FULL` or `FULL_DECODE_ONLY` (depending on backend support). However, this leads to another optimization incompatibility and confusing performance tradeoffs.
|
||||
|
||||
Long term, we've added the ability to partition the graph in Inductor instead of right after Dynamo. It can be enabled with `CompilationConfig.use_inductor_graph_partition=True` but is currently experimental and only available with `torch>=2.9`. This also increases compilation time as it has to compile the whole graph and cannot reuse piecewise compilation artifacts. Once vLLM supports 2.9, we plan to make this the default approach as it will also speed up piecewise cudagraph capture.
|
||||
|
||||
|
||||
+1
-75
@@ -25,8 +25,6 @@ or just on the low or high end.
|
||||
| [MLA Attention + Quant](#attention--quantization-fuse_attn_quant) | `fuse_attn_quant` | MLA Attention output → FP8/NVFP4 quant | Off by default | TBD | Yes | Always |
|
||||
| [RoPE + KV-Cache Update](#rope--kv-cache-update-fuse_rope_kvcache) | `fuse_rope_kvcache` | Rotary embedding → KV cache write | O2 (ROCm/AITER only) | 2-4% | No | Low |
|
||||
| [QK Norm + RoPE](#qk-norm--rope-enable_qk_norm_rope_fusion) | `enable_qk_norm_rope_fusion` | Q/K RMSNorm → rotary embedding | Off by default | 2-3% | No | Low |
|
||||
| [Sequence Parallelism](#sequence-parallelism-enable_sp) | `enable_sp` | AllReduce → ReduceScatter + AllGather | Off by default | Prereq for AsyncTP | Yes | High |
|
||||
| [AsyncTP GEMM + collective](#asynctp-gemm--collective-overlap-fuse_gemm_comms) | `fuse_gemm_comms` | GEMM → reduce-scatter / all-gather → GEMM | Off by default | 7-10% | Yes | High |
|
||||
| [RMSNorm + Quant](#rmsnorm--quantization-fuse_norm_quant) | `fuse_norm_quant` | RMSNorm (+residual add) → FP8/FP4 quant | O1 (conditional) | 1-4% | No | Always |
|
||||
| [SiLU+Mul + Quant](#silumul--quantization-fuse_act_quant) | `fuse_act_quant` | SiLU+Mul activation → FP8/FP4 quant | O1 (conditional) | 1-4% | No | Always |
|
||||
| [RMSNorm + Padding](#rmsnorm--padding-fuse_act_padding) | `fuse_act_padding` | Residual add + RMSNorm → padding | O1 (ROCm/AITER only) | TBD | No | Always |
|
||||
@@ -44,9 +42,7 @@ The table below lists the quantization schemes supported by each fusion on each
|
||||
| `fuse_attn_quant`\* | FP8 static\*, NVFP4\* | FP8 static\* | FP8 static\* | — | FP8 static\* |
|
||||
| `fuse_attn_quant` (MLA)\* | FP8 static\*, FP8 per-group\*, NVFP4\* | FP8 static\*, FP8 per-group\* | FP8 static\*, FP8 per-group\* | — | FP8 static\* (untested) |
|
||||
| `fuse_rope_kvcache` | — | — | — | — | FP16/BF16 |
|
||||
| `enable_qk_norm_rope_fusion` | FP16/BF16 | FP16/BF16 | FP16/BF16† | FP16/BF16† | — |
|
||||
| `enable_sp` | FP16/BF16, FP8 static† | FP16/BF16, FP8 static | FP16/BF16† | FP16/BF16† | — |
|
||||
| `fuse_gemm_comms` | FP16/BF16, FP8 static† | FP16/BF16, FP8 static | FP16/BF16† | FP16/BF16† | — |
|
||||
| `enable_qk_norm_rope_fusion` | FP16/BF16 | FP16/BF16 | FP16/BF16 | FP16/BF16 | — |
|
||||
| `fuse_norm_quant` | FP8 static, FP8 per-token, FP8 per-group | FP8 static, FP8 per-token, FP8 per-group | FP8 static, FP8 per-token, FP8 per-group | — | FP8 static, FP8 per-token, FP8 per-group |
|
||||
| `fuse_act_quant` | FP8 static, NVFP4 | FP8 static, FP8 per-group (128/64) | FP8 static, FP8 per-group (128/64) | — | FP8 per-group |
|
||||
| `fuse_act_padding` | — | — | — | — | FP16/BF16 |
|
||||
@@ -56,10 +52,6 @@ The table below lists the quantization schemes supported by each fusion on each
|
||||
fused quantization output. See the [`fuse_attn_quant` section](#attention--quantization-fuse_attn_quant)
|
||||
for per-backend details.
|
||||
|
||||
† `enable_sp` and `fuse_gemm_comms` are only autoconfigured for SM90 today;
|
||||
other architectures support requires setting `PassConfig.sp_min_token_num` explicitly.
|
||||
SM100 support also requires setting `VLLM_DISABLED_KERNELS=FlashInferFP8ScaledMMLinearKernel`.
|
||||
|
||||
## Enabling / Disabling Fusions
|
||||
|
||||
Fusions are exposed through `PassConfig`, which is nested inside `CompilationConfig`:
|
||||
@@ -186,72 +178,6 @@ If these conditions are set, the fusion is enabled automatically for optimizatio
|
||||
|
||||
- Pass: [`vllm/compilation/passes/fusion/rope_kvcache_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rope_kvcache_fusion.py)
|
||||
|
||||
### Sequence Parallelism (`enable_sp`)
|
||||
|
||||
**What it fuses.** Replaces all-reduce collectives with reduce-scatter + local RMSNorm + all-gather,
|
||||
splitting the sequence dimension across TP ranks. This restructures the graph so the subsequent AsyncTP
|
||||
pass can fuse the reduce-scatter / all-gather with the surrounding GEMMs.
|
||||
|
||||
Sequence Parallelism itself does not directly improve performance; it is a prerequisite for the
|
||||
AsyncTP pass (`fuse_gemm_comms`). SP is only applied above a minimum token threshold that is
|
||||
autoconfigured based on device capability and model `hidden_size`. Currently only active on
|
||||
H100/SM90 for models with `hidden_size >= 8192`. The threshold is configurable via
|
||||
`PassConfig.sp_min_token_num`.
|
||||
|
||||
The general transformation:
|
||||
|
||||
```text
|
||||
Input → AllReduce → RMSNorm → Output
|
||||
becomes:
|
||||
Input → ReduceScatter → local RMSNorm → AllGather → Output
|
||||
```
|
||||
|
||||
Patterns covered:
|
||||
|
||||
- First block: `AllReduce → RMSNorm` → `ReduceScatter → RMSNorm → AllGather`
|
||||
- Middle blocks: `AllReduce → fused_add_RMSNorm` → `ReduceScatter → fused_add_RMSNorm → AllGather`
|
||||
- Both with optional `→ FP8 static quant` suffix
|
||||
|
||||
Requires: `use_inductor_graph_partition=True` **or** piecewise compilation with static sizes
|
||||
divisible by `tensor_parallel_size`.
|
||||
|
||||
Supported hardware: Only tested on NVIDIA CUDA, possibly works on ROCm. FP8 all-gather requires sm90+.
|
||||
|
||||
**Code locations.**
|
||||
|
||||
- Pass: [`vllm/compilation/passes/fusion/sequence_parallelism.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/sequence_parallelism.py)
|
||||
|
||||
### AsyncTP GEMM + Collective Overlap (`fuse_gemm_comms`)
|
||||
|
||||
!!! info
|
||||
Requires `enable_sp=True` (enabled automatically). This pass is a no-op if Sequence Parallelism has not been applied.
|
||||
|
||||
**What it fuses.** After Sequence Parallelism transforms the graph, fuses GEMM kernels with the
|
||||
surrounding reduce-scatter (output projection) and all-gather (input projection) using
|
||||
`torch.ops.symm_mem` symmetric-memory primitives, overlapping communication and computation.
|
||||
This overlap is only profitable for large `num_tokens`, so the fusion (and preceding SP)
|
||||
is only performed in the higher compiled range above `PassConfig.sp_min_token_num`.
|
||||
|
||||
Patterns covered:
|
||||
|
||||
- `GEMM → reduce-scatter` → `fused_matmul_reduce_scatter`
|
||||
- `all-gather → GEMM` → `all_gather_matmul`
|
||||
- FP8 scaled variants of both patterns
|
||||
|
||||
Supported hardware: NVIDIA CUDA with symmetric-memory (`torch.distributed._symmetric_memory`) support.
|
||||
|
||||
On B200, pattern-matching fp8 FlashInfer scaled MM is not supported, so it must be disabled
|
||||
([#27893](https://github.com/vllm-project/vllm/issues/27893))
|
||||
|
||||
```shell
|
||||
VLLM_DISABLED_KERNELS=FlashInferFP8ScaledMMLinearKernel ...
|
||||
```
|
||||
|
||||
**Code locations.**
|
||||
|
||||
- Pass: [`vllm/compilation/passes/fusion/collective_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/collective_fusion.py)
|
||||
- Sequence parallelism pass: [`vllm/compilation/passes/fusion/sequence_parallelism.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/sequence_parallelism.py)
|
||||
|
||||
### QK Norm + RoPE (`enable_qk_norm_rope_fusion`)
|
||||
|
||||
!!! info
|
||||
|
||||
@@ -233,19 +233,17 @@ storing this information in vLLM's `PassContext` for later use in clone eliminat
|
||||
### 3. IR Fusion and Transformation Passes
|
||||
|
||||
After functionalization, custom vLLM passes operate on the functional FX graph containing high-level IR operations.
|
||||
These passes can perform fusion, distribute operations for sequence parallelism, and other transformations:
|
||||
These passes can perform fusion and other transformations:
|
||||
|
||||
```python
|
||||
# Example: Sequence Parallelism (see SequenceParallelismPass)
|
||||
# Before SP pass
|
||||
# Example: AllReduce + RMSNorm fusion (see AllReduceFusionPass)
|
||||
# Before
|
||||
|
||||
all_reduce = torch.ops.vllm.all_reduce(x, "tp:0")
|
||||
rms_norm = torch.ops.vllm_ir.rms_norm(all_reduce, weight, 1e-5)
|
||||
|
||||
# after SP pass
|
||||
reduce_scatter = torch.ops.vllm.reduce_scatter(x, "tp:0")
|
||||
rms_norm = torch.ops.vllm_ir.rms_norm(all_reduce, weight, 1e-5)
|
||||
all_gather = torch.ops.vllm.all_gather(x, "tp:0")
|
||||
# After
|
||||
rms_norm = torch.ops.vllm.flashinfer_trtllm_fused_allreduce_norm(x, weight, ...)
|
||||
```
|
||||
|
||||
Fusion passes benefit from the high-level representation: they don't need to match against low-level PyTorch operations,
|
||||
|
||||
Reference in New Issue
Block a user