[1/N] Unify multiple-path encoder cuda graph support (#49934)

Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
Isotr0py
2026-08-02 10:33:23 +08:00
committed by GitHub
co-authored by mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent c67fe497a2
commit e2fa28594f
7 changed files with 193 additions and 388 deletions
+16 -25
View File
@@ -41,13 +41,13 @@ class BudgetGraphMetadata:
Budgets are auto-generated as power-of-2 levels from a model-provided range via `get_encoder_cudagraph_budget_range()`, with the maximum budget always included even if it does not fall on a power-of-2 boundary. Budgets can also be explicitly specified by the user via `encoder_cudagraph_token_budgets` in `CompilationConfig`. Budgets are auto-generated as power-of-2 levels from a model-provided range via `get_encoder_cudagraph_budget_range()`, with the maximum budget always included even if it does not fall on a power-of-2 boundary. Budgets can also be explicitly specified by the user via `encoder_cudagraph_token_budgets` in `CompilationConfig`.
When `EncoderCudaGraphConfig.enable_dual_path_graph` is `True`, the manager generates two independent budget lists — `global_token_budgets` (multiples of `global_token_per_image`) and `local_token_budgets` (multiples of `local_token_per_patch`) — and stores captured graphs under `budget_graphs["global"]` and `budget_graphs["local"]` respectively. Each entry in `EncoderCudaGraphConfig.paths` defines an independently captured encoder path. A path can provide its own minimum token budget and opt into zero-token batches; the manager generates and stores a separate budget graph set for every configured path.
### Greedy bin-packing at runtime ### Greedy bin-packing at runtime
When a batch of images arrives, the manager sorts images by output token count (smallest first) and greedily packs as many images as possible into each sub-batch while staying within the **largest** token budget and the maximum batch size. Once a sub-batch is finalized (the next image would overflow either constraint), the manager finds the **smallest** budget that fits the sub-batch's total tokens and replays the corresponding CUDA Graph. This repeats until the batch is exhausted. Images that exceed all budgets fall back to eager execution. When a batch of images arrives, the manager sorts images by output token count (smallest first) and greedily packs as many images as possible into each sub-batch while staying within the **largest** token budget and the maximum batch size. Once a sub-batch is finalized (the next image would overflow either constraint), the manager finds the **smallest** budget that fits the sub-batch's total tokens and replays the corresponding CUDA Graph. This repeats until the batch is exhausted. Images that exceed all budgets fall back to eager execution.
For dual-path models, the manager routes to `_execute_local_dual_path()`, which constrains both global and local token budgets simultaneously during packing (see [Dual-Path graph capture](#dual-path-graph-capture)). For multi-path models, the same greedy packing loop constrains every configured path simultaneously (see [Multi-Path graph capture](#multi-path-graph-capture)).
For each graph replay: For each graph replay:
@@ -56,38 +56,29 @@ For each graph replay:
3. Replay the CUDA Graph. 3. Replay the CUDA Graph.
4. Clone outputs from `output_buffer` (cloning is necessary since the buffer is reused across replays). 4. Clone outputs from `output_buffer` (cloning is necessary since the buffer is reused across replays).
### Dual-Path graph capture ### Multi-Path graph capture
For two-tower vision encoders (e.g., DeepSeek-OCR), the `EncoderCudaGraphConfig` sets `enable_dual_path_graph=True` and provides `global_token_per_image` / `local_token_per_patch`. The manager captures two independent sets of CUDA graphs — one for the **global** image path and one for the **local** patch path — stored under `budget_graphs["global"]` and `budget_graphs["local"]` respectively. `EncoderCudaGraphConfig.paths` maps path names to `EncoderCudaGraphPathConfig` capture policies. For example, DeepSeek-OCR configures a **global** image path and a **local** patch path, which are captured independently under `budget_graphs["global"]` and `budget_graphs["local"]`.
**Budget generation.** Two separate budget lists are generated: **Budget generation.** Each path gets a separate budget list. For DeepSeek-OCR:
* `global_token_budgets` — power-of-2 multiples of `global_token_per_image` (e.g., `[272, 544, 1088, 2176, 4352, 8704, 13824]` for DeepSeek-OCR). * the `global` path — power-of-2 budgets starting at the global path minimum (e.g., `[272, 544, 1088, 2176, 4352, 8704, 13824]` for DeepSeek-OCR).
* `local_token_budgets` — power-of-2 multiples of `local_token_per_patch` (e.g., `[0, 100, 200, 400, 800, 1600, 3200, 6400, 12800]` for DeepSeek-OCR). A budget of `0` is always included to handle images with no local patches (images ≤ 640×640 that produce only global features). * the `local` path — power-of-2 budgets starting at the local path minimum (e.g., `[0, 100, 200, 400, 800, 1600, 3200, 6400, 12800, 13824]` for DeepSeek-OCR). A budget of `0` is included when `allow_zero_tokens=True` to handle images with no local patches (images ≤ 640×640 that produce only global features).
Both lists are capped at the same `max_budget`. Both lists are capped at the same `max_budget`.
**Dual-path greedy packing.** Each `EncoderItemSpec` provides both `global_output_tokens` (constant per image) and `local_output_tokens` (proportional to the patch count). The dual-path packing algorithm constrains both budgets simultaneously: **Multi-path greedy packing.** Each `EncoderItemSpec` provides `path_output_tokens`, mapping each path to its contribution for that item. The packing algorithm constrains every path simultaneously:
* Sort images by total output tokens (global + local), smallest first. * Sort images by total output tokens (global + local), smallest first.
* Greedily pack images: an image is added to the current sub-batch only if both the accumulated global tokens ≤ `max_global_budget` **and** the accumulated local tokens ≤ `max_local_budget`, with the image count ≤ `max_batch_size`. * Greedily pack images: an image is added to the current sub-batch only if every accumulated path token count is within that path's maximum budget, with the image count ≤ `max_batch_size`.
* Once either constraint would overflow, finalize the sub-batch and find the smallest fitting budget **independently** for each path. * Once any path constraint would overflow, finalize the sub-batch and find the smallest fitting budget **independently** for each path.
* Repeat until all images are packed. * Repeat until all images are packed.
**Partial graph fallback.** After packing, each sub-batch falls into one of four execution scenarios: **Partial graph fallback.** For each non-empty path, the manager replays the smallest fitting graph or runs only that path eagerly when no graph fits. Paths with zero tokens are skipped; a `0`-budget graph is never captured or replayed.
| Global budget | Local budget | Execution |
| :---: | :---: | --- |
| Found | Found | Both paths use CUDA graph replay |
| Found | `None` | Global graph replay + local path skipped (no patches) |
| `None` | Found | Global eager fallback + local graph replay |
| `None` | `None` | Both paths fall back to eager execution |
Note that the `0`-budget graph is never actually replayed for local — it signals that local patch processing should be skipped entirely.
**Buffer keys per path.** Global and local paths use different buffer keys. For DeepSeek-OCR, the global path uses `pixel_values` (full images, shape `[B, 3, 1280, 1280]`) while the local path uses `images_crop` (patches, shape `[P, 3, 1024, 1024]`). The manager iterates over each captured graph's own `input_buffers.keys()` rather than a shared `buffer_keys` list, so both paths can use different buffers. **Buffer keys per path.** Global and local paths use different buffer keys. For DeepSeek-OCR, the global path uses `pixel_values` (full images, shape `[B, 3, 1280, 1280]`) while the local path uses `images_crop` (patches, shape `[P, 3, 1024, 1024]`). The manager iterates over each captured graph's own `input_buffers.keys()` rather than a shared `buffer_keys` list, so both paths can use different buffers.
**Post-processing.** The `postprocess_encoder_output` method receives a `local_output` parameter (a tensor or `None`) containing the local-path encoder output. The model is responsible for assembling global and local features into the final per-image embedding. For DeepSeek-OCR, this means reshaping the global output into `[B, 272, n_embed]`, the local output into `[P, 100, n_embed]`, assembling patch grids with newline tokens, and concatenating `[patches_grid, global, view_separator]` for each image. **Post-processing.** The `postprocess_encoder_output` method receives an `outputs` dictionary keyed by path name. The model is responsible for assembling global and local features into the final per-image embedding. For DeepSeek-OCR, it reshapes the global output into `[B, 272, n_embed]`, reshapes the local output into `[P, 100, n_embed]`, assembles patch grids with newline tokens, and concatenating `[patches_grid, global, view_separator]` for each image.
!!! note !!! note
The dual-path design enables partial CUDA graph coverage — one path can hit while the other falls back to eager. This avoids wasted compute on zero-padded patch buffers for untiled images and avoids graph invalidation caused by variable `crop_shape` per image. The dual-path design enables partial CUDA graph coverage — one path can hit while the other falls back to eager. This avoids wasted compute on zero-padded patch buffers for untiled images and avoids graph invalidation caused by variable `crop_shape` per image.
@@ -111,13 +102,13 @@ Models opt-in to encoder CUDA Graphs by implementing the [SupportsEncoderCudaGra
* `get_encoder_cudagraph_config()` — returns static configuration (supported modalities, buffer keys, output hidden size, padding logics, max frames per video). * `get_encoder_cudagraph_config()` — returns static configuration (supported modalities, buffer keys, output hidden size, padding logics, max frames per video).
* `get_encoder_cudagraph_budget_range(vllm_config)` — returns `(min_budget, max_budget)` for auto-inference of token budgets. * `get_encoder_cudagraph_budget_range(vllm_config)` — returns `(min_budget, max_budget)` for auto-inference of token budgets.
* `get_encoder_cudagraph_item_specs(mm_kwargs)` — returns `list[EncoderItemSpec]` describing each item with its input size, total output token count (`output_tokens`), and optionally per-path token counts (`global_output_tokens`, `local_output_tokens`) for dual-path models. * `get_encoder_cudagraph_item_specs(mm_kwargs)` — returns `list[EncoderItemSpec]` describing each item with its input size, total output token count (`output_tokens`), and per-path token counts (`path_output_tokens`) for multi-path models.
* `select_encoder_cudagraph_items(mm_kwargs, indices)` — extracts a sub-batch of items by index, used during greedy packing and DP sharding. * `select_encoder_cudagraph_items(mm_kwargs, indices)` — extracts a sub-batch of items by index, used during greedy packing and DP sharding.
* `prepare_encoder_cudagraph_capture_inputs(..., path="default")` — creates dummy inputs for graph capture. The `path` parameter (`"global"` or `"local"`) tells the model which path to generate dummy inputs for. Returns `EncoderCudaGraphCaptureInputs` with a single `values: dict[str, torch.Tensor]` that contains all buffers to be recorded into the graph. * `prepare_encoder_cudagraph_capture_inputs(..., path="default")` — creates dummy inputs for graph capture. The `path` parameter (`"global"` or `"local"`) tells the model which path to generate dummy inputs for. Returns `EncoderCudaGraphCaptureInputs` with a single `values: dict[str, torch.Tensor]` that contains all buffers to be recorded into the graph.
* `prepare_encoder_cudagraph_replay_buffers(mm_kwargs, max_batch_size, max_frames_per_batch, path="default")` — computes buffer values from actual batch inputs. The `path` parameter selects which modality keys to extract from `mm_kwargs`. Returns `EncoderCudaGraphReplayBuffers` with a `values` dict whose keys match the captured graph's `input_buffers.keys()`. * `prepare_encoder_cudagraph_replay_buffers(mm_kwargs, max_batch_size, max_frames_per_batch, path="default")` — computes buffer values from actual batch inputs. The `path` parameter selects which modality keys to extract from `mm_kwargs`. Returns `EncoderCudaGraphReplayBuffers` with a `values` dict whose keys match the captured graph's `input_buffers.keys()`.
* `encoder_cudagraph_forward(inputs: dict[str, torch.Tensor], path="default")` — forward pass accepting only fixed-shaped input tensors (the captured `values` dict). Called during both capture and replay. The `path` parameter dispatches to the correct encoder sub-module (e.g., global vs. local path for DeepSeek-OCR). * `encoder_cudagraph_forward(inputs: dict[str, torch.Tensor], path="default")` — forward pass accepting only fixed-shaped input tensors (the captured `values` dict). Called during both capture and replay. The `path` parameter dispatches to the correct encoder sub-module (e.g., global vs. local path for DeepSeek-OCR).
* `encoder_eager_forward(mm_kwargs, path="default")` — fallback eager forward when no graph fits. When `path` is `"global"` or `"local"`, runs only that encoder path without graph capture. * `encoder_eager_forward(mm_kwargs, path="default")` — fallback eager forward when no graph fits. When `path` is `"global"` or `"local"`, runs only that encoder path without graph capture.
* `postprocess_encoder_output(..., local_output=None)` — post-process encoder output. The `local_output` parameter receives the local-path encoder output tensor (or `None`), enabling dual-path models to assemble global and local features into the final per-image embedding. * `postprocess_encoder_output(outputs, ...)` — post-process encoder outputs keyed by path name, enabling multi-path models to assemble their path-specific features into the final per-item embedding.
!!! note !!! note
The `SupportsEncoderCudaGraph` protocol is designed to be model-agnostic. New vision encoder models can opt-in by implementing the protocol methods without modifying the manager. The `SupportsEncoderCudaGraph` protocol is designed to be model-agnostic. New vision encoder models can opt-in by implementing the protocol methods without modifying the manager.
@@ -146,14 +137,14 @@ Models opt-in to encoder CUDA Graphs by implementing the [SupportsEncoderCudaGra
## Configuration ## Configuration
Three fields in `CompilationConfig` control encoder CUDA Graphs: Four fields in `CompilationConfig` control encoder CUDA Graphs:
* `cudagraph_mm_encoder` (`bool`, default `False`) — enable CUDA Graph capture for multimodal encoder. When enabled, captures the full encoder forward as a CUDA Graph for each token budget level. * `cudagraph_mm_encoder` (`bool`, default `False`) — enable CUDA Graph capture for multimodal encoder. When enabled, captures the full encoder forward as a CUDA Graph for each token budget level.
* `encoder_cudagraph_token_budgets` (`list[int]`, default `[]`) — token budget levels for capture. If empty (default), auto-inferred from model architecture as power-of-2 levels. User-provided values override auto-inference. * `encoder_cudagraph_token_budgets` (`list[int]`, default `[]`) — token budget levels for capture. If empty (default), auto-inferred from model architecture as power-of-2 levels. User-provided values override auto-inference.
* `encoder_cudagraph_max_vision_items_per_batch` (`int`, default `0`) — maximum number of images/videos per batch during capture. If 0 (default), auto-inferred as `max_budget // min_budget`. * `encoder_cudagraph_max_vision_items_per_batch` (`int`, default `0`) — maximum number of images/videos per batch during capture. If 0 (default), auto-inferred as `max_budget // min_budget`.
* `encoder_cudagraph_max_frames_per_batch` (`int`, default `None`) — maximum number of video frames per batch during capture. If `None` (default), auto-inferred as `encoder_cudagraph_max_vision_items_per_batch * max_frames_per_video` (`max_frames_per_video` is a model-specific value from `EncoderCudaGraphConfig`, computed by `get_max_frames_per_video()` on the model). If we limit the video count per prompt to `0`, it will also be set to `0` (i.e., fall back to image-only mode). * `encoder_cudagraph_max_frames_per_batch` (`int`, default `None`) — maximum number of video frames per batch during capture. If `None` (default), auto-inferred as `encoder_cudagraph_max_vision_items_per_batch * max_frames_per_video` (`max_frames_per_video` is a model-specific value from `EncoderCudaGraphConfig`, computed by `get_max_frames_per_video()` on the model). If we limit the video count per prompt to `0`, it will also be set to `0` (i.e., fall back to image-only mode).
Dual-path mode is configured at the model level via `EncoderCudaGraphConfig` fields (`enable_dual_path_graph`, `global_token_per_image`, `local_token_per_patch`) — no additional user configuration is required. The manager automatically generates separate budget lists and routes to dual-path execution when the model opts in. Multi-path mode is configured at the model level through `EncoderCudaGraphConfig.paths`. Each `EncoderCudaGraphPathConfig` can set a path-specific minimum budget and whether zero-token batches are allowed. The manager automatically generates separate budget lists and uses the same execution loop for single- and multi-path models.
## Usage guide ## Usage guide
@@ -106,6 +106,7 @@ def _make_manager_with_budgets(budgets: list[int]) -> EncoderCudaGraphManager:
""" """
mgr = object.__new__(EncoderCudaGraphManager) mgr = object.__new__(EncoderCudaGraphManager)
mgr.token_budgets = sorted(budgets) mgr.token_budgets = sorted(budgets)
mgr.path_token_budgets = {"default": mgr.token_budgets}
mgr.max_batch_size = 16 mgr.max_batch_size = 16
mgr.use_dp = False mgr.use_dp = False
mgr.config = EncoderCudaGraphConfig( mgr.config = EncoderCudaGraphConfig(
@@ -417,6 +418,7 @@ def _make_manager_for_gpu(
"""Create EncoderCudaGraphManager bypassing VllmConfig for GPU tests.""" """Create EncoderCudaGraphManager bypassing VllmConfig for GPU tests."""
mgr = object.__new__(EncoderCudaGraphManager) mgr = object.__new__(EncoderCudaGraphManager)
mgr.token_budgets = sorted(token_budgets) mgr.token_budgets = sorted(token_budgets)
mgr.path_token_budgets = {"default": mgr.token_budgets}
mgr.max_batch_size = max_batch_size mgr.max_batch_size = max_batch_size
mgr.max_frames_per_batch = ( mgr.max_frames_per_batch = (
max_frames_per_batch if max_frames_per_batch is not None else max_batch_size * 2 max_frames_per_batch if max_frames_per_batch is not None else max_batch_size * 2
+22 -11
View File
@@ -65,6 +65,7 @@ from vllm.v1.sample.logits_processor import (
from vllm.v1.worker.encoder_cudagraph_defs import ( from vllm.v1.worker.encoder_cudagraph_defs import (
EncoderCudaGraphCaptureInputs, EncoderCudaGraphCaptureInputs,
EncoderCudaGraphConfig, EncoderCudaGraphConfig,
EncoderCudaGraphPathConfig,
EncoderCudaGraphReplayBuffers, EncoderCudaGraphReplayBuffers,
EncoderItemSpec, EncoderItemSpec,
) )
@@ -697,9 +698,15 @@ class DeepseekOCRForCausalLM(
modalities=["image"], modalities=["image"],
buffer_keys=["pixel_values"], buffer_keys=["pixel_values"],
out_hidden_size=self.projector_config.n_embed, out_hidden_size=self.projector_config.n_embed,
enable_dual_path_graph=True, paths={
global_token_per_image=self.global_image_output_token, "global": EncoderCudaGraphPathConfig(
local_token_per_patch=self.single_patch_output_token, min_token_budget=self.global_image_output_token
),
"local": EncoderCudaGraphPathConfig(
min_token_budget=self.single_patch_output_token,
allow_zero_tokens=True,
),
},
) )
def get_encoder_cudagraph_budget_range( def get_encoder_cudagraph_budget_range(
@@ -730,8 +737,10 @@ class DeepseekOCRForCausalLM(
EncoderItemSpec( EncoderItemSpec(
input_size=num_input_tokens, input_size=num_input_tokens,
output_tokens=num_output_tokens, output_tokens=num_output_tokens,
global_output_tokens=global_output_token, path_output_tokens={
local_output_tokens=local_output_token, "global": global_output_token,
"local": local_output_token,
},
) )
) )
return item_specs return item_specs
@@ -917,32 +926,34 @@ class DeepseekOCRForCausalLM(
def postprocess_encoder_output( def postprocess_encoder_output(
self, self,
output: torch.Tensor, outputs: dict[str, torch.Tensor],
indices: list[int], indices: list[int],
per_item_out_tokens: list[int], per_item_out_tokens: list[int],
dest: dict[int, torch.Tensor] | list[torch.Tensor | None], dest: dict[int, torch.Tensor] | list[torch.Tensor | None],
clone: bool = False, clone: bool = False,
batch_mm_kwargs: dict[str, Any] | None = None, batch_mm_kwargs: dict[str, Any] | None = None,
local_output: torch.Tensor | None = None,
) -> None: ) -> None:
""" """
Assemble per-image embeddings from global and local encoder outputs. Assemble per-image embeddings from global and local encoder outputs.
``output`` contains global-image features with newlines already ``output['global']`` contains global-image features with newlines already
inserted (from CUDA graph replay or eager fallback): inserted (from CUDA graph replay or eager fallback):
``[B * 272, n_embed]``. ``[B * 272, n_embed]``.
``local_output`` contains local-patch features without ``output['local']`` contains local-patch features without
newlines (from CUDA graph replay or eager fallback): newlines (from CUDA graph replay or eager fallback):
``[P * 100, n_embed]``. May be ``None`` if no patches in batch. ``[P * 100, n_embed]``. May be ``None`` if no patches in batch.
This method: This method:
1. Splits ``output`` into per-image global portions. 1. Splits ``output['global']`` into per-image global portions.
2. Splits ``local_output`` into per-image patch groups. 2. Splits ``output['local']`` into per-image patch groups.
3. For each image: assembles patch grid with newlines via 3. For each image: assembles patch grid with newlines via
``_assemble_patch_grid``, then concatenates ``_assemble_patch_grid``, then concatenates
``[local_tiled, global, view_seperator]``. ``[local_tiled, global, view_seperator]``.
""" """
output = outputs["global"]
local_output = outputs.get("local")
assert batch_mm_kwargs is not None
bsz = len(indices) bsz = len(indices)
n_embed = output.shape[-1] n_embed = output.shape[-1]
+4 -3
View File
@@ -1696,13 +1696,12 @@ class SupportsEncoderCudaGraph(Protocol):
def postprocess_encoder_output( def postprocess_encoder_output(
self, self,
output: torch.Tensor, outputs: dict[str, torch.Tensor],
indices: list[int], indices: list[int],
per_item_out_tokens: list[int], per_item_out_tokens: list[int],
dest: dict[int, torch.Tensor] | list[torch.Tensor | None], dest: dict[int, torch.Tensor] | list[torch.Tensor | None],
clone: bool = False, clone: bool = False,
batch_mm_kwargs: dict[str, Any] | None = None, batch_mm_kwargs: dict[str, Any] | None = None,
local_output: torch.Tensor | None = None,
) -> None: ) -> None:
""" """
Post-process encoder output, directly call scatter_output_slices by default. Post-process encoder output, directly call scatter_output_slices by default.
@@ -1714,7 +1713,9 @@ class SupportsEncoderCudaGraph(Protocol):
""" """
from vllm.model_executor.models.utils import scatter_output_slices from vllm.model_executor.models.utils import scatter_output_slices
scatter_output_slices(output, indices, per_item_out_tokens, dest, clone) scatter_output_slices(
outputs["default"], indices, per_item_out_tokens, dest, clone
)
def prepare_encoder_cudagraph_capture_inputs( def prepare_encoder_cudagraph_capture_inputs(
self, self,
+20 -9
View File
@@ -725,6 +725,7 @@ class Step3VLForConditionalGeneration(
def get_encoder_cudagraph_config(self): def get_encoder_cudagraph_config(self):
from vllm.v1.worker.encoder_cudagraph_defs import ( from vllm.v1.worker.encoder_cudagraph_defs import (
EncoderCudaGraphConfig, EncoderCudaGraphConfig,
EncoderCudaGraphPathConfig,
) )
return EncoderCudaGraphConfig( return EncoderCudaGraphConfig(
@@ -734,9 +735,15 @@ class Step3VLForConditionalGeneration(
"patch_pixel_values", "patch_pixel_values",
], ],
out_hidden_size=self.config.hidden_size, out_hidden_size=self.config.hidden_size,
enable_dual_path_graph=True, paths={
global_token_per_image=self.img_output_tokens, "global": EncoderCudaGraphPathConfig(
local_token_per_patch=self.patch_output_tokens, min_token_budget=self.img_output_tokens
),
"local": EncoderCudaGraphPathConfig(
min_token_budget=self.patch_output_tokens,
allow_zero_tokens=True,
),
},
) )
def get_encoder_cudagraph_budget_range( def get_encoder_cudagraph_budget_range(
@@ -771,8 +778,10 @@ class Step3VLForConditionalGeneration(
output_tokens=( output_tokens=(
self.img_output_tokens + num_patch * self.patch_output_tokens self.img_output_tokens + num_patch * self.patch_output_tokens
), ),
global_output_tokens=self.img_output_tokens, path_output_tokens={
local_output_tokens=num_patch * self.patch_output_tokens, "global": self.img_output_tokens,
"local": num_patch * self.patch_output_tokens,
},
) )
for num_patch in num_patches for num_patch in num_patches
] ]
@@ -876,19 +885,21 @@ class Step3VLForConditionalGeneration(
def postprocess_encoder_output( def postprocess_encoder_output(
self, self,
output: torch.Tensor, outputs: dict[str, torch.Tensor],
indices: list[int], indices: list[int],
per_item_out_tokens: list[int], per_item_out_tokens: list[int],
dest: dict[int, torch.Tensor] | list[torch.Tensor | None], dest: dict[int, torch.Tensor] | list[torch.Tensor | None],
clone: bool = False, clone: bool = False,
batch_mm_kwargs: dict[str, Any] | None = None, batch_mm_kwargs: dict[str, Any] | None = None,
local_output: torch.Tensor | None = None,
): ):
"""CPU-side per-item merge after dual-path graph replay. """CPU-side per-item merge after dual-path graph replay.
``output`` contains global-image features and ``local_output`` ``outputs['global']`` contains global-image features and ``outputs['local']``
contains local-patch features (or ``None`` when there are no patches). contains local-patch features (or ``None`` when there are no patches).
""" """
output = outputs["global"]
local_output = outputs.get("local")
assert batch_mm_kwargs is not None
num_patches = batch_mm_kwargs["num_patches"] num_patches = batch_mm_kwargs["num_patches"]
hidden = output.shape[-1] hidden = output.shape[-1]
bsz = len(indices) bsz = len(indices)
@@ -899,7 +910,7 @@ class Step3VLForConditionalGeneration(
patch_tokens = total_patches * self.patch_output_tokens patch_tokens = total_patches * self.patch_output_tokens
global_part = output[:img_tokens].reshape(bsz, self.img_output_tokens, hidden) global_part = output[:img_tokens].reshape(bsz, self.img_output_tokens, hidden)
if total_patches > 0: if total_patches > 0 and local_output is not None:
patch_part = local_output[:patch_tokens].reshape( patch_part = local_output[:patch_tokens].reshape(
-1, self.patch_output_tokens, hidden -1, self.patch_output_tokens, hidden
) )
+104 -323
View File
@@ -158,34 +158,33 @@ class EncoderCudaGraphManager:
self.graph_misses = 0 self.graph_misses = 0
self.log_stats_interval = 100 self.log_stats_interval = 100
if self.config.enable_dual_path_graph: max_budget = self.token_budgets[-1]
max_budget = self.token_budgets[-1] if not self.config.paths:
self.global_token_budgets = self._generate_budgets( raise ValueError("Encoder CUDA graph config must define at least one path")
self.config.global_token_per_image, self.path_token_budgets: dict[str, list[int]] = {}
max_budget, for path, path_config in self.config.paths.items():
) min_path_budget = path_config.min_token_budget
self.local_token_budgets = self._generate_budgets( if min_path_budget is None:
self.config.local_token_per_patch, budgets = list(self.token_budgets)
max_budget, else:
) if min_path_budget <= 0 or min_path_budget > max_budget:
# When `image_width <= 640 and image_height <= 640`, the mm inputs raise ValueError(
# will only contain global image, without generating local patches. f"Invalid minimum budget {min_path_budget} for encoder "
self.local_token_budgets.insert(0, 0) f"CUDA graph path {path!r}; max budget is {max_budget}"
logger.info( )
"EncoderCudaGraphManager dual-path mode: " budgets = self._generate_budgets(min_path_budget, max_budget)
"global_budgets=%s, local_budgets=%s", if path_config.allow_zero_tokens:
self.global_token_budgets, budgets.insert(0, 0)
self.local_token_budgets, self.path_token_budgets[path] = budgets
)
else: logger.info(
logger.info( "EncoderCudaGraphManager initialized with paths=%s, "
"EncoderCudaGraphManager initialized with " "max_batch_size=%d, max_frames_per_batch=%s, use_dp=%s",
"budgets=%s, max_batch_size=%d, max_frames_per_batch=%s, use_dp=%s", self.path_token_budgets,
self.token_budgets, self.max_batch_size,
self.max_batch_size, self.max_frames_per_batch,
self.max_frames_per_batch, self.use_dp,
self.use_dp, )
)
@staticmethod @staticmethod
def _generate_budgets(min_budget: int, max_budget: int) -> list[int]: def _generate_budgets(min_budget: int, max_budget: int) -> list[int]:
@@ -211,40 +210,29 @@ class EncoderCudaGraphManager:
self.graph_pool = None self.graph_pool = None
def capture(self, graph_pool: Any): def capture(self, graph_pool: Any):
"""Capture CUDA graphs for all token budgets.""" """Capture CUDA graphs for every configured path and token budget."""
self.graph_pool = graph_pool self.graph_pool = graph_pool
if self.config.enable_dual_path_graph: for path, budgets in self.path_token_budgets.items():
for token_budget in sorted(self.global_token_budgets, reverse=True): for token_budget in sorted(budgets, reverse=True):
self._capture_budget_graph(token_budget, path="global") if token_budget > 0:
for token_budget in sorted(self.local_token_budgets, reverse=True): self._capture_budget_graph(token_budget, path=path)
if token_budget == 0:
continue
self._capture_budget_graph(token_budget, path="local")
logger.info(
"Encoder CUDA graph capture complete. "
"Captured %d global + %d local budget graphs.",
len(self.budget_graphs["global"]),
len(self.budget_graphs["local"]),
)
return
for token_budget in sorted(self.token_budgets, reverse=True):
self._capture_budget_graph(token_budget)
logger.info( logger.info(
"Encoder CUDA graph capture complete. Captured %d budget graphs.", "Encoder CUDA graph capture complete. Captured %d graphs across %d paths.",
len(self.budget_graphs["default"]), self.get_num_graphs_to_capture(),
len(self.path_token_budgets),
) )
def get_num_graphs_to_capture(self) -> int: def get_num_graphs_to_capture(self) -> int:
if self.config.enable_dual_path_graph: return sum(
return len(self.global_token_budgets) + len(self.local_token_budgets) token_budget > 0
return len(self.token_budgets) for budgets in self.path_token_budgets.values()
for token_budget in budgets
)
def _get_graph_set(self, path: str = "default") -> dict[int, BudgetGraphMetadata]: def _get_graph_set(self, path: str = "default") -> dict[int, BudgetGraphMetadata]:
# Lazy init global/local graph sets for dual-path models, or default graph """Return the captured graphs for one encoder path."""
# set for single-path models.
if path not in self.budget_graphs: if path not in self.budget_graphs:
self.budget_graphs[path] = {} self.budget_graphs[path] = {}
return self.budget_graphs[path] return self.budget_graphs[path]
@@ -331,7 +319,7 @@ class EncoderCudaGraphManager:
Args: Args:
mm_kwargs: Multimodal inputs for the batch. mm_kwargs: Multimodal inputs for the batch.
token_budget: Token budget to use. token_budget: Token budget to use.
path: Path for the graph. Should be one of ["default", "global", "local"]. path: Configured encoder path.
Returns: Returns:
Encoder outputs, or None if graph not captured. Encoder outputs, or None if graph not captured.
""" """
@@ -351,9 +339,7 @@ class EncoderCudaGraphManager:
path, path,
) )
# Copy replay buffers into graph input buffers. Iterate over the # Copy replay values into the buffers recorded for this path.
# graph's own buffer keys (which may differ per path for dual-path
# models) rather than the global config.buffer_keys.
for key, buf in graph_meta.input_buffers.items(): for key, buf in graph_meta.input_buffers.items():
src = replay.values.get(key) src = replay.values.get(key)
if src is None: if src is None:
@@ -375,296 +361,91 @@ class EncoderCudaGraphManager:
self, self,
mm_kwargs: dict[str, Any], mm_kwargs: dict[str, Any],
) -> list[torch.Tensor]: ) -> list[torch.Tensor]:
"""Execute encoder on local inputs using greedy-packed CUDA graphs. """Execute locally using greedy packing across all configured paths."""
Sort images by output token count (smallest first), then greedily pack
as many images as possible into each batch while staying within
max_budget tokens and max_batch_size. Once a batch is finalised (next
image would overflow either constraint), find the smallest fitting
budget once for that batch.
For dual-path models (``enable_dual_path_graph=True``), two independent
graph sets are used: one for global images, one for local patches.
Budgets are found independently per path; if only one path fits, the
other falls back to eager via partial fallback.
By exchange argument, greedy smallest-first packing minimises eager
fallbacks -- any other ordering yields a higher token sum in some batch,
making that batch more likely to exceed the budget.
Stats note:
graph_hits -- counted inside _run_budget_graph after successful replay.
graph_misses -- counted here for single-image batches where the image
exceeds max_budget. Batches split due to max_batch_size
always satisfy total_tokens <= max_budget and therefore
always find a valid budget (no miss).
"""
if self.config.enable_dual_path_graph:
return self._execute_local_dual_path(mm_kwargs)
return self._execute_local_single_path(mm_kwargs)
def _execute_local_single_path(
self,
mm_kwargs: dict[str, Any],
) -> list[torch.Tensor]:
"""Single-path greedy-packing execution (original behaviour)."""
item_specs = self._get_item_specs(mm_kwargs) item_specs = self._get_item_specs(mm_kwargs)
num_items = len(item_specs) num_items = len(item_specs)
max_budget = self.token_budgets[-1] paths = tuple(self.path_token_budgets)
per_item_path_tokens = {
path: [spec.get_path_output_tokens(path) for spec in item_specs]
for path in paths
}
per_item_out_tokens = [spec.output_tokens for spec in item_specs] per_item_out_tokens = [spec.output_tokens for spec in item_specs]
max_path_budgets = {path: max(self.path_token_budgets[path]) for path in paths}
# Sort ascending by output token count (smallest first)
sorted_indices = sorted(range(num_items), key=lambda i: per_item_out_tokens[i]) sorted_indices = sorted(range(num_items), key=lambda i: per_item_out_tokens[i])
batches: list[tuple[list[int], dict[str, int | None]]] = []
# Greedy pack against max_budget and max_batch_size.
# _find_smallest_fitting_budget_given_tokens is called once per
# finalised batch, not per image.
batches: list[tuple[list[int], int | None]] = []
current_batch: list[int] = [] current_batch: list[int] = []
current_batch_tokens = 0 current_tokens = dict.fromkeys(paths, 0)
def append_current_batch() -> None:
if not current_batch:
return
path_budgets = {
path: (
0
if current_tokens[path] == 0
else self._find_smallest_fitting_budget_given_tokens(
current_tokens[path], self.path_token_budgets[path]
)
)
for path in paths
}
batches.append((list(current_batch), path_budgets))
for orig_idx in sorted_indices: for orig_idx in sorted_indices:
item_tokens = per_item_out_tokens[orig_idx] item_tokens = {path: per_item_path_tokens[path][orig_idx] for path in paths}
if ( fits = len(current_batch) < self.max_batch_size and all(
current_batch_tokens + item_tokens <= max_budget current_tokens[path] + item_tokens[path] <= max_path_budgets[path]
and len(current_batch) < self.max_batch_size for path in paths
):
current_batch.append(orig_idx)
current_batch_tokens += item_tokens
else:
if current_batch:
batches.append(
(
current_batch,
self._find_smallest_fitting_budget_given_tokens(
current_batch_tokens
),
)
)
current_batch = [orig_idx]
current_batch_tokens = item_tokens
if current_batch:
batches.append(
(
current_batch,
self._find_smallest_fitting_budget_given_tokens(
current_batch_tokens
),
)
) )
if current_batch and not fits:
append_current_batch()
current_batch = []
current_tokens = dict.fromkeys(paths, 0)
# outputs_by_orig_idx maps each original image index to its output current_batch.append(orig_idx)
# tensor. Needed because greedy packing reorders images; we restore for path in paths:
# the original order before returning. current_tokens[path] += item_tokens[path]
outputs_by_orig_idx: dict[int, torch.Tensor] = {}
for batch_orig_indices, token_budget in batches: append_current_batch()
batch_mm_kwargs = self.model.select_encoder_cudagraph_items(
mm_kwargs, batch_orig_indices
)
batch_out_tokens = sum(per_item_out_tokens[i] for i in batch_orig_indices)
if token_budget is None:
# Single oversized image: item_tokens > max_budget.
# graph_misses counted here for this eager fallback.
logger.debug(
"Encoder CUDA graph fallback to eager: no budget for "
"%d tokens from %d images",
batch_out_tokens,
len(batch_orig_indices),
)
self.graph_misses += len(batch_orig_indices)
with torch.inference_mode():
raw = self.model.encoder_eager_forward(batch_mm_kwargs)
scatter_output_slices(
raw,
batch_orig_indices,
per_item_out_tokens,
outputs_by_orig_idx,
)
else:
logger.debug(
"Encoder CUDA graph: batch_size=%d, tokens=%d, "
"budget=%d, waste=%.1f%%",
len(batch_orig_indices),
batch_out_tokens,
token_budget,
(token_budget - batch_out_tokens) / token_budget * 100,
)
# graph_hits counted inside _run_budget_graph after replay.
output = self._run_budget_graph(batch_mm_kwargs, token_budget)
assert output is not None
self.model.postprocess_encoder_output(
output,
batch_orig_indices,
per_item_out_tokens,
outputs_by_orig_idx,
clone=True,
batch_mm_kwargs=batch_mm_kwargs,
)
# Return in original batch order (caller maps outputs to token positions)
return [outputs_by_orig_idx[i] for i in range(num_items)]
def _execute_local_dual_path(
self,
mm_kwargs: dict[str, Any],
) -> list[torch.Tensor]:
"""Dual-path greedy-packing execution.
Each image contributes both global tokens (constant per image)
and local tokens (patches * patch_tokens). Greedy packing
respects both budgets independently, then selects the smallest
fitting budget per path with partial eager fallback.
"""
item_specs = self._get_item_specs(mm_kwargs)
num_items = len(item_specs)
max_global_budget = self.global_token_budgets[-1]
max_local_budget = self.local_token_budgets[-1]
per_item_global_tokens = [spec.global_output_tokens for spec in item_specs]
per_item_local_tokens = [spec.local_output_tokens for spec in item_specs]
per_item_total_tokens = [spec.output_tokens for spec in item_specs]
# Sort ascending by total output tokens
sorted_indices = sorted(
range(num_items), key=lambda i: per_item_total_tokens[i]
)
# Each batch is a tuple of (indices, global_budget, local_budget).
batches: list[tuple[list[int], int | None, int | None]] = []
current_batch: list[int] = []
current_global_tokens = 0
current_local_tokens = 0
for orig_idx in sorted_indices:
global_token = per_item_global_tokens[orig_idx]
local_token = per_item_local_tokens[orig_idx]
if (
current_global_tokens + global_token <= max_global_budget
and current_local_tokens + local_token <= max_local_budget
and len(current_batch) < self.max_batch_size
):
current_batch.append(orig_idx)
current_global_tokens += global_token
current_local_tokens += local_token
else:
if current_batch:
batches.append(
(
current_batch,
self._find_smallest_fitting_budget_given_tokens(
current_global_tokens, self.global_token_budgets
),
self._find_smallest_fitting_budget_given_tokens(
current_local_tokens, self.local_token_budgets
),
)
)
current_batch = [orig_idx]
current_global_tokens = global_token
current_local_tokens = local_token
if current_batch:
batches.append(
(
current_batch,
self._find_smallest_fitting_budget_given_tokens(
current_global_tokens, self.global_token_budgets
),
self._find_smallest_fitting_budget_given_tokens(
current_local_tokens, self.local_token_budgets
),
)
)
outputs_by_orig_idx: dict[int, torch.Tensor] = {} outputs_by_orig_idx: dict[int, torch.Tensor] = {}
for batch_indices, path_budgets in batches:
for batch_orig_indices, global_budget, local_budget in batches:
batch_mm_kwargs = self.model.select_encoder_cudagraph_items( batch_mm_kwargs = self.model.select_encoder_cudagraph_items(
mm_kwargs, batch_orig_indices mm_kwargs, batch_indices
)
batch_global_tokens = sum(
per_item_global_tokens[i] for i in batch_orig_indices
)
batch_local_tokens = sum(
per_item_local_tokens[i] for i in batch_orig_indices
) )
graph_outputs: dict[str, torch.Tensor] = {}
all_eager = True
both_eager = global_budget is None and local_budget is None for path in paths:
path_tokens = sum(per_item_path_tokens[path][i] for i in batch_indices)
if path_tokens == 0:
continue
if both_eager: token_budget = path_budgets[path]
logger.debug( if token_budget is None:
"Encoder CUDA graph dual-path full eager fallback: " with torch.inference_mode():
"%d global + %d local tokens from %d images", output = self.model.encoder_eager_forward(
batch_global_tokens, batch_mm_kwargs, path=path
batch_local_tokens, )
len(batch_orig_indices), else:
) all_eager = False
self.graph_misses += len(batch_orig_indices) output = self._run_budget_graph(
with torch.inference_mode(): batch_mm_kwargs, token_budget, path=path
raw = self.model.encoder_eager_forward(batch_mm_kwargs)
per_item_total = [
per_item_global_tokens[i] + per_item_local_tokens[i] + 1
for i in batch_orig_indices
]
scatter_output_slices(
raw, batch_orig_indices, per_item_total, outputs_by_orig_idx
)
continue
logger.debug(
"Encoder CUDA graph dual-path: batch_size=%d, "
"global=%d (budget=%s), local=%d (budget=%s)",
len(batch_orig_indices),
batch_global_tokens,
global_budget,
batch_local_tokens,
local_budget,
)
# Execute global path: graph or eager fallback
if global_budget is not None:
global_output = self._run_budget_graph(
batch_mm_kwargs,
global_budget,
path="global",
)
assert global_output is not None
else:
with torch.inference_mode():
global_output = self.model.encoder_eager_forward(
batch_mm_kwargs, path="global"
) )
assert output is not None
graph_outputs[path] = output
# Execute local path: graph or eager fallback if all_eager:
if local_budget is not None and batch_local_tokens > 0: self.graph_misses += len(batch_indices)
local_output = self._run_budget_graph(
batch_mm_kwargs,
local_budget,
path="local",
)
assert local_output is not None
elif batch_local_tokens > 0:
with torch.inference_mode():
local_output = self.model.encoder_eager_forward(
batch_mm_kwargs, path="local"
)
else:
local_output = None
self.model.postprocess_encoder_output( self.model.postprocess_encoder_output(
global_output, graph_outputs,
batch_orig_indices, batch_indices,
per_item_global_tokens, per_item_out_tokens,
outputs_by_orig_idx, outputs_by_orig_idx,
clone=True, clone=True,
batch_mm_kwargs=batch_mm_kwargs, batch_mm_kwargs=batch_mm_kwargs,
local_output=local_output,
) )
return [outputs_by_orig_idx[i] for i in range(num_items)] return [outputs_by_orig_idx[i] for i in range(num_items)]
+25 -17
View File
@@ -26,13 +26,28 @@ class EncoderItemSpec:
"""Number of output tokens after encoder processing (e.g. after """Number of output tokens after encoder processing (e.g. after
spatial merge).""" spatial merge)."""
global_output_tokens: int = 0 path_output_tokens: dict[str, int] = field(default_factory=dict)
"""Number of output tokens from the global image path. """Per-path output token counts for multi-path encoders.
Only used when ``EncoderCudaGraphConfig.enable_dual_path_graph`` is True."""
local_output_tokens: int = 0 Single-path encoders leave this empty and use ``output_tokens`` for the
"""Number of output tokens from the local patch path. default path.
Only used when ``EncoderCudaGraphConfig.enable_dual_path_graph`` is True.""" """
def get_path_output_tokens(self, path: str) -> int:
if path == "default" and not self.path_output_tokens:
return self.output_tokens
return self.path_output_tokens.get(path, 0)
@dataclass(frozen=True)
class EncoderCudaGraphPathConfig:
"""Capture policy for one independently replayable encoder path."""
min_token_budget: int | None = None
"""Smallest capture budget, or the model default minimum when unset."""
allow_zero_tokens: bool = False
"""Whether a batch may omit this path entirely."""
@dataclass @dataclass
@@ -68,17 +83,10 @@ class EncoderCudaGraphConfig:
Only relevant when "video" is in ``modalities``. Only relevant when "video" is in ``modalities``.
Image-only models can use the default of 1.""" Image-only models can use the default of 1."""
enable_dual_path_graph: bool = False paths: dict[str, EncoderCudaGraphPathConfig] = field(
"""If True, the manager captures two independent graph sets default_factory=lambda: {"default": EncoderCudaGraphPathConfig()}
(global + local) and runs dual-path graph selection during inference.""" )
"""Independently captured encoder paths keyed by their forward name."""
global_token_per_image: int = 0
"""Tokens per global image (e.g. 272 for DeepSeek-OCR).
Only used when ``enable_dual_path_graph`` is True."""
local_token_per_patch: int = 0
"""Tokens per local patch (e.g. 100 for DeepSeek-OCR).
Only used when ``enable_dual_path_graph`` is True."""
@dataclass @dataclass