[vLLM IR] 2/N fused_add_rms_norm and maybe_inplace overload (#36823)
Signed-off-by: Luka Govedič <lgovedic@redhat.com> Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
This commit is contained in:
@@ -5,12 +5,14 @@ TL;DR:
|
||||
- use tlparse to acquire torch.compile logs. Include these logs in bug reports and/or support asks.
|
||||
- The vLLM-torch.compile integration is multiple pieces. vLLM exposes flags to turn off each piece:
|
||||
|
||||
| Online Flag | Offline Flag | Result |
|
||||
| ----------- | ------------ | ------ |
|
||||
| --enforce-eager | enforce_eager=True | Turn off torch.compile and CUDAGraphs |
|
||||
| -cc.mode=0 | mode=CompilationMode.NONE | Turn off torch.compile only |
|
||||
| -cc.cudagraph_mode=NONE | compilation_config=CompilationConfig(cudagraph_mode=CUDAGraphMode.NONE) | Turn off CUDAGraphs only |
|
||||
| -cc.backend=eager | compilation_config=CompilationConfig(backend='eager') | Turn off TorchInductor |
|
||||
| Online Flag | Offline Flag | Result |
|
||||
|--------------------------------|--------------------------------------------------------------------------------|------------------------------------------------------|
|
||||
| --enforce-eager | enforce_eager=True | Turn off torch.compile and CUDAGraphs |
|
||||
| -cc.mode=0 | compilation_config=CompilationConfig(mode=CompilationMode.NONE) | Turn off torch.compile only |
|
||||
| -cc.mode=1 | compilation_config=CompilationConfig(mode=CompilationMode.STOCK_TORCH_COMPILE) | Turn off vLLM-compile modifications to torch.compile |
|
||||
| -cc.cudagraph_mode=NONE | compilation_config=CompilationConfig(cudagraph_mode=CUDAGraphMode.NONE) | Turn off CUDAGraphs only |
|
||||
| -cc.backend=eager | compilation_config=CompilationConfig(backend='eager') | Turn off TorchInductor |
|
||||
| -cc.ir_enable_torch_wrap=False | compilation_config=CompilationConfig(ir_enable_torch_wrap=False) | Turn off vLLM IR wrapping |
|
||||
|
||||
## vLLM-torch.compile overview
|
||||
|
||||
@@ -22,7 +24,7 @@ Most notably, vLLM-compile is NOT torch.compile, it is a custom compiler built u
|
||||
|
||||
- Given a model, we do a full graph capture via TorchDynamo that is dynamic on the batch size (number of tokens)
|
||||
- vLLM then optionally splits and/or specializes this graph and then uses TorchInductor to compile each graph into a compiled artifact.
|
||||
This step may use vLLM custom Inductor passes to further optimize the graph.
|
||||
This step may use vLLM custom Inductor passes to further optimize the graph. This includes vLLM IR lowering to remove dispatch overhead.
|
||||
- The compiled artifact is saved to vLLM's compile cache so that it can be loaded in the future.
|
||||
- vLLM applies CUDAGraphs to reduce CPU overheads.
|
||||
|
||||
@@ -34,6 +36,7 @@ For more details on the design, please see the following resources:
|
||||
|
||||
- [Introduction to vLLM-torch.compile blogpost](https://blog.vllm.ai/2025/08/20/torch-compile.html)
|
||||
- [vLLM-torch.compile integration design](./torch_compile.md)
|
||||
- [vLLM IR design](./vllm_ir.md)
|
||||
- [vLLM Office Hours #26](https://www.youtube.com/live/xLyxc7hxCJc?si=Xulo9pe53C6ywf0V&t=561)
|
||||
- [Talk at PyTorch Conference 2025](https://youtu.be/1wV1ESbGrVQ?si=s1GqymUfwiwOrDTg&t=725)
|
||||
|
||||
@@ -117,6 +120,21 @@ from vllm.config.compilation import CompilationConfig, CUDAGraphMode
|
||||
LLM(model, compilation_config=CompilationConfig(cudagraph_mode=CUDAGraphMode.NONE))
|
||||
```
|
||||
|
||||
vLLM IR makes heavy use of the compilation pipeline, from functionalization, custom fusions, and lowering.
|
||||
To turn that off and capture eager-mode dispatching behavior of vLLM IR, run with `ir_enable_torch_wrap=False`.
|
||||
IR torch wrap is only enabled by default when using `mode=VLLM_COMPILE` and `backend="inductor"` (default).
|
||||
|
||||
```sh
|
||||
# Online
|
||||
vllm serve -cc.ir_enable_torch_wrap=False
|
||||
```
|
||||
|
||||
```py
|
||||
# Offline
|
||||
from vllm.config.compilation import CompilationConfig
|
||||
LLM(model, compilation_config=CompilationConfig(ir_enable_torch_wrap=False))
|
||||
```
|
||||
|
||||
## Debugging TorchDynamo
|
||||
|
||||
vLLM requires model code be capturable into a full graph via TorchDynamo (torch.compile's frontend).
|
||||
|
||||
Reference in New Issue
Block a user