[Profiler] Add Proton CUDA graph attribution for MRV2 (#51084)
Signed-off-by: Luossu <zhluosuu@outlook.com> Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Nick Hill <nickhill123@gmail.com>
This commit is contained in:
@@ -90,18 +90,18 @@ $ curl -X POST http://localhost:8000/stop_profile
|
||||
is Triton's GPU profiler. It can collect a low-overhead aggregate tree or a
|
||||
Chrome trace and works through the same vLLM profiling controls as the PyTorch
|
||||
and CUDA profilers. Proton currently supports NVIDIA GPUs through CUPTI and
|
||||
requires eager execution.
|
||||
supports CUDA graph attribution.
|
||||
|
||||
Start a server with a local output directory:
|
||||
Start a server with a local output directory and graph attribution:
|
||||
|
||||
```bash
|
||||
vllm serve meta-llama/Llama-3.1-8B-Instruct \
|
||||
--enforce-eager \
|
||||
--profiler-config '{
|
||||
"profiler": "proton",
|
||||
"proton_profiler_dir": "./proton_profile",
|
||||
"proton_output_format": "hatchet",
|
||||
"proton_hook": "triton"
|
||||
"proton_hook": "triton",
|
||||
"proton_graph_attribution": true
|
||||
}'
|
||||
```
|
||||
|
||||
@@ -111,7 +111,7 @@ rank-qualified output name, such as
|
||||
`proton_dp0_pp0_tp0_dcp0_ep0_rank0_pid1234_0123456789abcdef0123456789abcdef_run0.hatchet`,
|
||||
so distributed workers, restarted servers, and repeated profiling runs do not
|
||||
overwrite one another. A `profile_prefix` is included when supplied. Each
|
||||
profile is finalized by `/stop_profile` and is ready to inspect immediately.
|
||||
profile is written by `/stop_profile` and is ready to inspect immediately.
|
||||
|
||||
The Proton-specific options are:
|
||||
|
||||
@@ -121,21 +121,39 @@ The Proton-specific options are:
|
||||
- `proton_mode`: an optional backend mode string
|
||||
- `proton_hook`: `triton` to record Triton launch metadata, or unset
|
||||
- `proton_output_format`: `hatchet`, `hatchet_msgpack`, `chrome_trace`, or unset
|
||||
- `proton_graph_attribution`: observe CUDA graph capture for replay attribution;
|
||||
disabled by default and requires `proton_data: "tree"`
|
||||
|
||||
`hatchet` and `hatchet_msgpack` require `proton_data: "tree"`, while
|
||||
`chrome_trace` requires `proton_data: "trace"`. CUDA graph profiling is not yet
|
||||
supported, so `--enforce-eager` is required for every Proton mode.
|
||||
`chrome_trace` requires `proton_data: "trace"`.
|
||||
|
||||
Automatic backend selection is recommended. vLLM currently supports Proton's
|
||||
`cupti` backend on NVIDIA GPUs. ROCm support is not yet available. vLLM does not
|
||||
expose Proton's experimental instrumentation backend because current upstream
|
||||
Triton builds can produce profiles without timing metrics. Backend-specific
|
||||
modes, including `pcsampling`, can be selected with `proton_mode`.
|
||||
can produce profiles without timing metrics. When `proton_graph_attribution` is
|
||||
enabled, Proton observes vLLM's CUDA graph capture with the configured profiling
|
||||
session active, then deactivates that same session until profiling starts. This
|
||||
lets later profiles attribute replayed kernels without retaining model-startup
|
||||
activity. Backend-specific modes can be selected with `proton_mode`;
|
||||
`pcsampling` synchronizes the CUDA context and therefore requires
|
||||
`--enforce-eager`. When CUDA graphs are enabled (including encoder graphs),
|
||||
Proton requires `proton_graph_attribution: true` to collect replayed kernels.
|
||||
For Chrome traces, disable CUDA graphs with `--enforce-eager`.
|
||||
|
||||
Triton 3.6 supports explicit `hatchet` and `chrome_trace` output. The
|
||||
`hatchet_msgpack` format and `periodic_flushing` mode require Triton 3.7 or
|
||||
newer; vLLM rejects these options with the detected version before starting a
|
||||
session.
|
||||
CUDA graph-attributed profiles support repeated `start_profile`/`stop_profile`
|
||||
runs. Each stop flushes and writes one tree-data phase while preserving the
|
||||
graph-aware session. Without `proton_graph_attribution`, each `stop_profile`
|
||||
instead finalizes and writes an independent Proton session.
|
||||
|
||||
CUDA graph attribution requires Triton 3.7 or newer. It uses the phase data API
|
||||
to discard graph-capture activity and separate profiling runs. The `hatchet_msgpack`
|
||||
output format and `periodic_flushing` mode also require Triton 3.7 or newer.
|
||||
`periodic_flushing` cannot be combined with graph attribution because both
|
||||
manage the session's data phases. Ordinary Proton profiling remains available
|
||||
with Triton 3.6.
|
||||
|
||||
Graph attribution retains capture metadata for the worker lifetime. With eager
|
||||
execution or no graphs to capture, profiling uses ordinary independent sessions.
|
||||
|
||||
Inspect tree profiles with:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user