[MM][Perf][CG] Support dual-path ViT full CUDA graph for DeepSeek-OCR (#43586)
Signed-off-by: shen-shanshan <467638484@qq.com> Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
This commit is contained in:
co-authored by
Roger Wang
Isotr0py
parent
b2cfae777d
commit
ad32608e24
@@ -2,6 +2,8 @@
|
||||
|
||||
The [CUDA Graphs](cuda_graphs.md) infrastructure in vLLM primarily targets the **decoder** (language model) forward pass. vLLM also supports capturing the **encoder** (vision transformer) forward pass as CUDA Graphs, independently from the decoder. This is based on <https://github.com/vllm-project/vllm/pull/35963>.
|
||||
|
||||
For two-tower vision encoders (e.g., DeepSeek-OCR's SAM + CLIP with dynamic tiling), a **dual-path graph** mode captures two independent sets of CUDA graphs — one for the global image path and one for the local patch path — enabling independent budget selection and partial eager fallback per path. This is based on <https://github.com/vllm-project/vllm/pull/43586>.
|
||||
|
||||
!!! note
|
||||
Encoder CUDA Graphs are orthogonal to decoder CUDA Graphs — both can be enabled simultaneously. Encoder graphs capture the vision encoder execution (e.g., ViT in Qwen3-VL), while decoder graphs capture the language model execution as described in the [CUDA Graphs design document](cuda_graphs.md).
|
||||
|
||||
@@ -11,6 +13,8 @@ Vision encoder inference incurs CUDA kernel launch overhead on the host side. Th
|
||||
|
||||
Encoder CUDA Graphs eliminate this overhead by pre-capturing the full encoder forward pass at multiple token budget levels during model initialization, then replaying the appropriate graph at runtime.
|
||||
|
||||
For two-tower vision encoders such as DeepSeek-OCR (SAM + CLIP with dynamic tiling), the global image path and local patch path have independent token profiles (272 tokens per global image vs. 100 tokens per local patch). Capturing a single monolithic graph for both paths would significantly reduce packing efficiency. The dual-path graph mode captures each path as a separate set of budgets, allowing the manager to pack and replay each path independently.
|
||||
|
||||
## Design
|
||||
|
||||
The encoder CUDA Graph system uses a **budget-based capture/replay** strategy, managed by [EncoderCudaGraphManager][vllm.v1.worker.encoder_cudagraph.EncoderCudaGraphManager]. The system contains the following core components:
|
||||
@@ -37,10 +41,14 @@ class BudgetGraphMetadata:
|
||||
|
||||
Budgets are auto-generated as power-of-2 levels from a model-provided range via `get_encoder_cudagraph_budget_range()`, with the maximum budget always included even if it does not fall on a power-of-2 boundary. Budgets can also be explicitly specified by the user via `encoder_cudagraph_token_budgets` in `CompilationConfig`.
|
||||
|
||||
When `EncoderCudaGraphConfig.enable_dual_path_graph` is `True`, the manager generates two independent budget lists — `global_token_budgets` (multiples of `global_token_per_image`) and `local_token_budgets` (multiples of `local_token_per_patch`) — and stores captured graphs under `budget_graphs["global"]` and `budget_graphs["local"]` respectively.
|
||||
|
||||
### Greedy bin-packing at runtime
|
||||
|
||||
When a batch of images arrives, the manager sorts images by output token count (smallest first) and greedily packs as many images as possible into each sub-batch while staying within the **largest** token budget and the maximum batch size. Once a sub-batch is finalized (the next image would overflow either constraint), the manager finds the **smallest** budget that fits the sub-batch's total tokens and replays the corresponding CUDA Graph. This repeats until the batch is exhausted. Images that exceed all budgets fall back to eager execution.
|
||||
|
||||
For dual-path models, the manager routes to `_execute_local_dual_path()`, which constrains both global and local token budgets simultaneously during packing (see [Dual-Path graph capture](#dual-path-graph-capture)).
|
||||
|
||||
For each graph replay:
|
||||
|
||||
1. Call `prepare_encoder_cudagraph_replay_buffers()` to compute buffer values (including `pixel_values` and precomputed metadata) from actual batch inputs.
|
||||
@@ -48,6 +56,42 @@ For each graph replay:
|
||||
3. Replay the CUDA Graph.
|
||||
4. Clone outputs from `output_buffer` (cloning is necessary since the buffer is reused across replays).
|
||||
|
||||
### Dual-Path graph capture
|
||||
|
||||
For two-tower vision encoders (e.g., DeepSeek-OCR), the `EncoderCudaGraphConfig` sets `enable_dual_path_graph=True` and provides `global_token_per_image` / `local_token_per_patch`. The manager captures two independent sets of CUDA graphs — one for the **global** image path and one for the **local** patch path — stored under `budget_graphs["global"]` and `budget_graphs["local"]` respectively.
|
||||
|
||||
**Budget generation.** Two separate budget lists are generated:
|
||||
|
||||
* `global_token_budgets` — power-of-2 multiples of `global_token_per_image` (e.g., `[272, 544, 1088, 2176, 4352, 8704, 13824]` for DeepSeek-OCR).
|
||||
* `local_token_budgets` — power-of-2 multiples of `local_token_per_patch` (e.g., `[0, 100, 200, 400, 800, 1600, 3200, 6400, 12800]` for DeepSeek-OCR). A budget of `0` is always included to handle images with no local patches (images ≤ 640×640 that produce only global features).
|
||||
|
||||
Both lists are capped at the same `max_budget`.
|
||||
|
||||
**Dual-path greedy packing.** Each `EncoderItemSpec` provides both `global_output_tokens` (constant per image) and `local_output_tokens` (proportional to the patch count). The dual-path packing algorithm constrains both budgets simultaneously:
|
||||
|
||||
* Sort images by total output tokens (global + local), smallest first.
|
||||
* Greedily pack images: an image is added to the current sub-batch only if both the accumulated global tokens ≤ `max_global_budget` **and** the accumulated local tokens ≤ `max_local_budget`, with the image count ≤ `max_batch_size`.
|
||||
* Once either constraint would overflow, finalize the sub-batch and find the smallest fitting budget **independently** for each path.
|
||||
* Repeat until all images are packed.
|
||||
|
||||
**Partial graph fallback.** After packing, each sub-batch falls into one of four execution scenarios:
|
||||
|
||||
| Global budget | Local budget | Execution |
|
||||
| :---: | :---: | --- |
|
||||
| Found | Found | Both paths use CUDA graph replay |
|
||||
| Found | `None` | Global graph replay + local path skipped (no patches) |
|
||||
| `None` | Found | Global eager fallback + local graph replay |
|
||||
| `None` | `None` | Both paths fall back to eager execution |
|
||||
|
||||
Note that the `0`-budget graph is never actually replayed for local — it signals that local patch processing should be skipped entirely.
|
||||
|
||||
**Buffer keys per path.** Global and local paths use different buffer keys. For DeepSeek-OCR, the global path uses `pixel_values` (full images, shape `[B, 3, 1280, 1280]`) while the local path uses `images_crop` (patches, shape `[P, 3, 1024, 1024]`). The manager iterates over each captured graph's own `input_buffers.keys()` rather than a shared `buffer_keys` list, so both paths can use different buffers.
|
||||
|
||||
**Post-processing.** The `postprocess_encoder_output` method receives a `local_output` parameter (a tensor or `None`) containing the local-path encoder output. The model is responsible for assembling global and local features into the final per-image embedding. For DeepSeek-OCR, this means reshaping the global output into `[B, 272, n_embed]`, the local output into `[P, 100, n_embed]`, assembling patch grids with newline tokens, and concatenating `[patches_grid, global, view_separator]` for each image.
|
||||
|
||||
!!! note
|
||||
The dual-path design enables partial CUDA graph coverage — one path can hit while the other falls back to eager. This avoids wasted compute on zero-padded patch buffers for untiled images and avoids graph invalidation caused by variable `crop_shape` per image.
|
||||
|
||||
### Data-parallel support
|
||||
|
||||
When `mm_encoder_tp_mode="data"`, the manager distributes images across TP ranks using load-balanced assignment via `get_load_balance_assignment`, executes locally on each rank, then gathers results back in the original order via `tensor_model_parallel_all_gather`.
|
||||
@@ -67,29 +111,30 @@ Models opt-in to encoder CUDA Graphs by implementing the [SupportsEncoderCudaGra
|
||||
|
||||
* `get_encoder_cudagraph_config()` — returns static configuration (supported modalities, buffer keys, output hidden size, padding logics, max frames per video).
|
||||
* `get_encoder_cudagraph_budget_range(vllm_config)` — returns `(min_budget, max_budget)` for auto-inference of token budgets.
|
||||
* `get_encoder_cudagraph_item_specs(mm_kwargs)` — returns `list[EncoderItemSpec]` describing each item with its input size and output token count. Replaces the former three separate methods (`get_num_items`, `get_per_item_output_tokens`, `get_per_item_input_sizes`).
|
||||
* `get_encoder_cudagraph_item_specs(mm_kwargs)` — returns `list[EncoderItemSpec]` describing each item with its input size, total output token count (`output_tokens`), and optionally per-path token counts (`global_output_tokens`, `local_output_tokens`) for dual-path models.
|
||||
* `select_encoder_cudagraph_items(mm_kwargs, indices)` — extracts a sub-batch of items by index, used during greedy packing and DP sharding.
|
||||
* `prepare_encoder_cudagraph_capture_inputs(...)` — creates dummy inputs for graph capture. Returns `EncoderCudaGraphCaptureInputs` with a single `values: dict[str, torch.Tensor]` that contains all buffers to be recorded into the graph.
|
||||
* `prepare_encoder_cudagraph_replay_buffers(mm_kwargs, max_batch_size, max_frames_per_batch)` — computes buffer values from actual batch inputs. Returns `EncoderCudaGraphReplayBuffers` with a `values` dict whose keys match `buffer_keys` in the config.
|
||||
* `encoder_cudagraph_forward(inputs: dict[str, torch.Tensor])` — forward pass accepting only fixed-shaped input tensors (the captured `values` dict). Called during both capture and replay. The `pixel_values` tensor is included in `inputs` alongside metadata buffers.
|
||||
* `encoder_eager_forward(mm_kwargs)` — fallback eager forward when no graph fits.
|
||||
* `postprocess_encoder_output(...)` — post-process encoder output, delegates to `scatter_output_slices` by default.
|
||||
* `prepare_encoder_cudagraph_capture_inputs(..., path="default")` — creates dummy inputs for graph capture. The `path` parameter (`"global"` or `"local"`) tells the model which path to generate dummy inputs for. Returns `EncoderCudaGraphCaptureInputs` with a single `values: dict[str, torch.Tensor]` that contains all buffers to be recorded into the graph.
|
||||
* `prepare_encoder_cudagraph_replay_buffers(mm_kwargs, max_batch_size, max_frames_per_batch, path="default")` — computes buffer values from actual batch inputs. The `path` parameter selects which modality keys to extract from `mm_kwargs`. Returns `EncoderCudaGraphReplayBuffers` with a `values` dict whose keys match the captured graph's `input_buffers.keys()`.
|
||||
* `encoder_cudagraph_forward(inputs: dict[str, torch.Tensor], path="default")` — forward pass accepting only fixed-shaped input tensors (the captured `values` dict). Called during both capture and replay. The `path` parameter dispatches to the correct encoder sub-module (e.g., global vs. local path for DeepSeek-OCR).
|
||||
* `encoder_eager_forward(mm_kwargs, path="default")` — fallback eager forward when no graph fits. When `path` is `"global"` or `"local"`, runs only that encoder path without graph capture.
|
||||
* `postprocess_encoder_output(..., local_output=None)` — post-process encoder output. The `local_output` parameter receives the local-path encoder output tensor (or `None`), enabling dual-path models to assemble global and local features into the final per-image embedding.
|
||||
|
||||
!!! note
|
||||
The `SupportsEncoderCudaGraph` protocol is designed to be model-agnostic. New vision encoder models can opt-in by implementing the protocol methods without modifying the manager.
|
||||
|
||||
**Supported models:**
|
||||
|
||||
| Architecture | Models | CG for Image | CG for Video |
|
||||
| ------------ | ------ | ------------ | ------------ |
|
||||
| `Llama4ForConditionalGeneration` | `Llama 4` | ✅︎ | - |
|
||||
| `InternVLChatModel` | `InternVL3.5`, `InternVL3`, `InternVL2.5`, `InternVL2` | ✅︎ | ✅︎ |
|
||||
| `Qwen2VLForConditionalGeneration` | `Qwen2-VL` | ✅︎ | ✅︎ |
|
||||
| `Qwen2_5_VLForConditionalGeneration` | `Qwen2.5-VL` | ✅︎ | ✅︎ |
|
||||
| `Qwen3VLForConditionalGeneration` | `Qwen3-VL` | ✅︎ | ✅︎ |
|
||||
| `Qwen3_5ForConditionalGeneration` | `Qwen3.5` | ✅︎ | ✅︎ |
|
||||
| `Step3VLForConditionalGeneration` | `Step3-VL` | ✅︎ | ❌︎ |
|
||||
| `Glm4vForConditionalGeneration` | `GLM-4.1V, GLM-4.6V-Flash` | ✅︎ | ✅︎ |
|
||||
| Architecture | Models | CG for Image | CG for Video | Dual-Path Graph |
|
||||
| ------------ | ------ | ------------ | ------------ | --------------- |
|
||||
| `DeepseekOCRForCausalLM` | `DeepSeek-OCR` | ✅︎ | ❌︎ | ✅︎ |
|
||||
| `Glm4vForConditionalGeneration` | `GLM-4.1V, GLM-4.6V-Flash` | ✅︎ | ✅︎ | ❌︎ |
|
||||
| `InternVLChatModel` | `InternVL3.5`, `InternVL3`, `InternVL2.5`, `InternVL2` | ✅︎ | ✅︎ | ❌︎ |
|
||||
| `Llama4ForConditionalGeneration` | `Llama 4` | ✅︎ | ❌︎ | ❌︎ |
|
||||
| `Qwen2VLForConditionalGeneration` | `Qwen2-VL` | ✅︎ | ✅︎ | ❌︎ |
|
||||
| `Qwen2_5_VLForConditionalGeneration` | `Qwen2.5-VL` | ✅︎ | ✅︎ | ❌︎ |
|
||||
| `Qwen3VLForConditionalGeneration` | `Qwen3-VL` | ✅︎ | ✅︎ | ❌︎ |
|
||||
| `Qwen3_5ForConditionalGeneration` | `Qwen3.5` | ✅︎ | ✅︎ | ❌︎ |
|
||||
| `Step3VLForConditionalGeneration` | `Step3-VL` | ✅︎ | ❌︎ | ❌︎ |
|
||||
|
||||
!!! note
|
||||
Encoder CUDA Graphs have currently been tested with `--mm-encoder-attn-backend=FLASH_ATTN` and `--mm-encoder-attn-backend=FLASHINFER` on Blackwell GPUs.
|
||||
@@ -104,6 +149,8 @@ Three fields in `CompilationConfig` control encoder CUDA Graphs:
|
||||
* `encoder_cudagraph_max_vision_items_per_batch` (`int`, default `0`) — maximum number of images/videos per batch during capture. If 0 (default), auto-inferred as `max_budget // min_budget`.
|
||||
* `encoder_cudagraph_max_frames_per_batch` (`int`, default `None`) — maximum number of video frames per batch during capture. If `None` (default), auto-inferred as `encoder_cudagraph_max_vision_items_per_batch * max_frames_per_video` (`max_frames_per_video` is a model-specific value from `EncoderCudaGraphConfig`, computed by `get_max_frames_per_video()` on the model). If we limit the video count per prompt to `0`, it will also be set to `0` (i.e., fall back to image-only mode).
|
||||
|
||||
Dual-path mode is configured at the model level via `EncoderCudaGraphConfig` fields (`enable_dual_path_graph`, `global_token_per_image`, `local_token_per_patch`) — no additional user configuration is required. The manager automatically generates separate budget lists and routes to dual-path execution when the model opts in.
|
||||
|
||||
## Usage guide
|
||||
|
||||
### Image inference
|
||||
|
||||
@@ -2533,15 +2533,16 @@ MODELS_NEED_VIDEO_METADATA = [
|
||||
|
||||
MODELS_SUPPORT_VIT_CUDA_GRAPH = [
|
||||
"llama4",
|
||||
"internvl_chat",
|
||||
"qwen2_vl",
|
||||
"qwen2_5_vl",
|
||||
"qwen3_vl",
|
||||
"qwen3_vl_moe",
|
||||
"qwen2_vl",
|
||||
"qwen3_5",
|
||||
"qwen3_5_moe",
|
||||
"internvl_chat",
|
||||
"stepvl",
|
||||
"glm4_1v",
|
||||
"deepseek_ocr",
|
||||
]
|
||||
|
||||
|
||||
|
||||
@@ -29,6 +29,7 @@ class VitCudagraphTestConfig:
|
||||
vllm_runner_kwargs: dict = field(default_factory=dict)
|
||||
compilation_config_overrides: dict = field(default_factory=dict)
|
||||
marks: list = field(default_factory=list)
|
||||
skip: bool = False
|
||||
|
||||
|
||||
def params_with_marks(
|
||||
@@ -75,15 +76,16 @@ MODEL_CONFIGS: dict[str, VitCudagraphTestConfig] = {
|
||||
},
|
||||
marks=[pytest.mark.core_model],
|
||||
),
|
||||
"internvl": VitCudagraphTestConfig(
|
||||
model="OpenGVLab/InternVL3-1B",
|
||||
num_video_frames=8,
|
||||
image_prompt=internvl_chat_template("<image>\nWhat is in this image?"),
|
||||
video_prompt=internvl_chat_template(
|
||||
"<video>\nDescribe this video in one sentence."
|
||||
"qwen2_vl": VitCudagraphTestConfig(
|
||||
model="Qwen/Qwen2-VL-2B-Instruct",
|
||||
image_prompt=qwen_vl_chat_template(
|
||||
"<|vision_start|><|image_pad|><|vision_end|>What is in this image?"
|
||||
),
|
||||
video_prompt=qwen_vl_chat_template(
|
||||
"<|vision_start|><|video_pad|><|vision_end|>"
|
||||
"Describe this video in one sentence."
|
||||
),
|
||||
needs_video_metadata=False,
|
||||
vllm_runner_kwargs={"trust_remote_code": True},
|
||||
marks=[pytest.mark.core_model],
|
||||
),
|
||||
"qwen2_5_vl": VitCudagraphTestConfig(
|
||||
@@ -122,16 +124,15 @@ MODEL_CONFIGS: dict[str, VitCudagraphTestConfig] = {
|
||||
needs_video_metadata=True,
|
||||
marks=[pytest.mark.core_model],
|
||||
),
|
||||
"qwen2_vl": VitCudagraphTestConfig(
|
||||
model="Qwen/Qwen2-VL-2B-Instruct",
|
||||
image_prompt=qwen_vl_chat_template(
|
||||
"<|vision_start|><|image_pad|><|vision_end|>What is in this image?"
|
||||
),
|
||||
video_prompt=qwen_vl_chat_template(
|
||||
"<|vision_start|><|video_pad|><|vision_end|>"
|
||||
"Describe this video in one sentence."
|
||||
"internvl": VitCudagraphTestConfig(
|
||||
model="OpenGVLab/InternVL3-1B",
|
||||
num_video_frames=8,
|
||||
image_prompt=internvl_chat_template("<image>\nWhat is in this image?"),
|
||||
video_prompt=internvl_chat_template(
|
||||
"<video>\nDescribe this video in one sentence."
|
||||
),
|
||||
needs_video_metadata=False,
|
||||
vllm_runner_kwargs={"trust_remote_code": True},
|
||||
marks=[pytest.mark.core_model],
|
||||
),
|
||||
"step3_vl": VitCudagraphTestConfig(
|
||||
@@ -179,6 +180,25 @@ MODEL_CONFIGS: dict[str, VitCudagraphTestConfig] = {
|
||||
),
|
||||
},
|
||||
),
|
||||
"deepseek_ocr": VitCudagraphTestConfig(
|
||||
model="deepseek-ai/DeepSeek-OCR",
|
||||
modalities=["image"],
|
||||
image_prompt="<image>\nWhat is in this image?",
|
||||
marks=[pytest.mark.core_model],
|
||||
compilation_config_overrides={
|
||||
"encoder_cudagraph_token_budgets": [272],
|
||||
"mode": 0,
|
||||
"cudagraph_mode": 2,
|
||||
},
|
||||
vllm_runner_kwargs={
|
||||
"load_format": "dummy",
|
||||
"hf_overrides": partial(
|
||||
dummy_hf_overrides,
|
||||
model_arch="DeepseekOCRForCausalLM",
|
||||
),
|
||||
},
|
||||
skip=True, # TODO: Re-enable this once OOM issues are resolved on CI.
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
@@ -201,6 +221,9 @@ def get_compilation_config(config: VitCudagraphTestConfig):
|
||||
def test_vit_cudagraph_image(model_id, vllm_runner, image_assets):
|
||||
config = MODEL_CONFIGS[model_id]
|
||||
|
||||
if config.skip:
|
||||
pytest.skip(f"{model_id} is marked to be skipped.")
|
||||
|
||||
if "image" not in config.modalities:
|
||||
pytest.skip(f"{model_id} does not support the image modality.")
|
||||
|
||||
@@ -242,6 +265,9 @@ def test_vit_cudagraph_image(model_id, vllm_runner, image_assets):
|
||||
def test_vit_cudagraph_video(model_id, vllm_runner, video_assets):
|
||||
config = MODEL_CONFIGS[model_id]
|
||||
|
||||
if config.skip:
|
||||
pytest.skip(f"{model_id} is marked to be skipped.")
|
||||
|
||||
if "video" not in config.modalities:
|
||||
pytest.skip(f"{model_id} does not support the video modality")
|
||||
|
||||
|
||||
@@ -108,7 +108,7 @@ def _make_manager_with_budgets(budgets: list[int]) -> EncoderCudaGraphManager:
|
||||
mgr.token_budgets = sorted(budgets)
|
||||
mgr.max_batch_size = 16
|
||||
mgr.use_dp = False
|
||||
mgr.budget_graphs = {}
|
||||
mgr.budget_graphs = {"default": {}}
|
||||
mgr.graph_pool = None
|
||||
mgr.graph_hits = 0
|
||||
mgr.graph_misses = 0
|
||||
@@ -341,6 +341,7 @@ class SimpleMockViTModel(torch.nn.Module, SupportsEncoderCudaGraph):
|
||||
max_frames_per_batch: int,
|
||||
device: torch.device,
|
||||
dtype: torch.dtype,
|
||||
path: str = "default",
|
||||
) -> EncoderCudaGraphCaptureInputs:
|
||||
per_image_output = token_budget // max_batch_size
|
||||
grid_config = [
|
||||
@@ -365,6 +366,7 @@ class SimpleMockViTModel(torch.nn.Module, SupportsEncoderCudaGraph):
|
||||
mm_kwargs: dict[str, Any],
|
||||
max_batch_size: int,
|
||||
max_frames_per_batch: int,
|
||||
path: str = "default",
|
||||
) -> EncoderCudaGraphReplayBuffers:
|
||||
grid_thw = mm_kwargs["image_grid_thw"]
|
||||
n_out = _count_output_tokens(grid_thw, _SPATIAL_MERGE)
|
||||
@@ -380,12 +382,14 @@ class SimpleMockViTModel(torch.nn.Module, SupportsEncoderCudaGraph):
|
||||
def encoder_cudagraph_forward(
|
||||
self,
|
||||
values: dict[str, torch.Tensor],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
return self._forward(values["pixel_values"])
|
||||
|
||||
def encoder_eager_forward(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
return self._forward(mm_kwargs["pixel_values"])
|
||||
|
||||
@@ -413,7 +417,7 @@ def _make_manager_for_gpu(
|
||||
max_frames_per_batch if max_frames_per_batch is not None else max_batch_size * 2
|
||||
)
|
||||
mgr.use_dp = False
|
||||
mgr.budget_graphs = {}
|
||||
mgr.budget_graphs = {"default": {}}
|
||||
mgr.graph_pool = None
|
||||
mgr.graph_hits = 0
|
||||
mgr.graph_misses = 0
|
||||
@@ -479,15 +483,15 @@ class TestEncoderCudaGraphCaptureReplay:
|
||||
# --- capture ---
|
||||
|
||||
def test_capture_creates_one_graph_per_budget(self):
|
||||
assert len(self.mgr.budget_graphs) == len(_BUDGETS)
|
||||
assert set(self.mgr.budget_graphs.keys()) == set(_BUDGETS)
|
||||
assert len(self.mgr.budget_graphs["default"]) == len(_BUDGETS)
|
||||
assert set(self.mgr.budget_graphs["default"].keys()) == set(_BUDGETS)
|
||||
|
||||
def test_capture_uses_supplied_graph_pool(self):
|
||||
assert self.mgr.graph_pool is self.graph_pool
|
||||
|
||||
def test_clear_releases_graphs_and_pool(self):
|
||||
self.mgr.clear()
|
||||
assert self.mgr.budget_graphs == {}
|
||||
assert self.mgr.budget_graphs == {"default": {}}
|
||||
assert self.mgr.graph_pool is None
|
||||
|
||||
# --- output shape ---
|
||||
@@ -642,6 +646,7 @@ class SimpleMockViTVideoModel(SimpleMockViTModel):
|
||||
max_frames_per_batch: int,
|
||||
device: torch.device,
|
||||
dtype: torch.dtype,
|
||||
path: str = "default",
|
||||
) -> EncoderCudaGraphCaptureInputs:
|
||||
per_item_output = token_budget // max_batch_size
|
||||
frames_per_item = max_frames_per_batch // max_batch_size
|
||||
@@ -678,6 +683,7 @@ class SimpleMockViTVideoModel(SimpleMockViTModel):
|
||||
mm_kwargs: dict[str, Any],
|
||||
max_batch_size: int,
|
||||
max_frames_per_batch: int,
|
||||
path: str = "default",
|
||||
) -> EncoderCudaGraphReplayBuffers:
|
||||
n_out = _count_output_tokens(self._get_grid_thw(mm_kwargs), _SPATIAL_MERGE)
|
||||
p = next(self.parameters())
|
||||
@@ -692,12 +698,14 @@ class SimpleMockViTVideoModel(SimpleMockViTModel):
|
||||
def encoder_cudagraph_forward(
|
||||
self,
|
||||
values: dict[str, torch.Tensor],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
return self._forward(values["pixel_values"])
|
||||
|
||||
def encoder_eager_forward(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
return self._forward(self._get_pixel_values(mm_kwargs))
|
||||
|
||||
@@ -763,8 +771,8 @@ class TestEncoderCudaGraphVideoReplay:
|
||||
# --- capture ---
|
||||
|
||||
def test_capture_creates_one_graph_per_budget(self):
|
||||
assert len(self.mgr.budget_graphs) == len(_BUDGETS)
|
||||
assert set(self.mgr.budget_graphs.keys()) == set(_BUDGETS)
|
||||
assert len(self.mgr.budget_graphs["default"]) == len(_BUDGETS)
|
||||
assert set(self.mgr.budget_graphs["default"].keys()) == set(_BUDGETS)
|
||||
|
||||
# --- output shape ---
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
import math
|
||||
from collections.abc import Iterable, Mapping, Sequence
|
||||
from typing import Annotated, Literal
|
||||
from typing import Annotated, Any, Literal
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
@@ -15,6 +15,7 @@ from vllm.config.multimodal import BaseDummyOptions
|
||||
from vllm.inputs import MultiModalDataDict
|
||||
from vllm.model_executor.models.interfaces import (
|
||||
MultiModalEmbeddings,
|
||||
SupportsEncoderCudaGraph,
|
||||
SupportsLoRA,
|
||||
SupportsMultiModal,
|
||||
SupportsPP,
|
||||
@@ -52,6 +53,7 @@ from vllm.transformers_utils.configs.deepseek_vl2 import DeepseekVLV2Config
|
||||
from vllm.transformers_utils.processors.deepseek_ocr import (
|
||||
BASE_SIZE,
|
||||
CROP_MODE,
|
||||
IMAGE_SIZE,
|
||||
DeepseekOCRProcessor,
|
||||
count_tiles,
|
||||
)
|
||||
@@ -60,12 +62,17 @@ from vllm.v1.sample.logits_processor import (
|
||||
AdapterLogitsProcessor,
|
||||
RequestLogitsProcessor,
|
||||
)
|
||||
from vllm.v1.worker.encoder_cudagraph_defs import (
|
||||
EncoderCudaGraphCaptureInputs,
|
||||
EncoderCudaGraphConfig,
|
||||
EncoderCudaGraphReplayBuffers,
|
||||
EncoderItemSpec,
|
||||
)
|
||||
|
||||
from .deepencoder import DeepCLIPVisionTransformer, build_sam_vit_b
|
||||
from .deepseek_vl2 import MlpProjector
|
||||
|
||||
# The image token id may be various
|
||||
IMAGE_SIZE = 640
|
||||
_IMAGE_TOKEN = "<image>"
|
||||
|
||||
|
||||
@@ -355,7 +362,9 @@ class DeepseekOCRMultiModalProcessor(
|
||||
info=DeepseekOCRProcessingInfo,
|
||||
dummy_inputs=DeepseekOCRDummyInputsBuilder,
|
||||
)
|
||||
class DeepseekOCRForCausalLM(nn.Module, SupportsMultiModal, SupportsPP, SupportsLoRA):
|
||||
class DeepseekOCRForCausalLM(
|
||||
nn.Module, SupportsMultiModal, SupportsPP, SupportsLoRA, SupportsEncoderCudaGraph
|
||||
):
|
||||
hf_to_vllm_mapper = WeightsMapper(
|
||||
orig_to_new_prefix={
|
||||
# map prefix for language backbone
|
||||
@@ -383,6 +392,7 @@ class DeepseekOCRForCausalLM(nn.Module, SupportsMultiModal, SupportsPP, Supports
|
||||
multimodal_config = vllm_config.model_config.multimodal_config
|
||||
|
||||
self.config = config
|
||||
self.model_config = vllm_config.model_config
|
||||
self.multimodal_config = multimodal_config
|
||||
|
||||
self.vision_config = config.vision_config
|
||||
@@ -504,9 +514,14 @@ class DeepseekOCRForCausalLM(nn.Module, SupportsMultiModal, SupportsPP, Supports
|
||||
)
|
||||
features = self.projector(features)
|
||||
|
||||
return self._assemble_patch_grid(features, crop_shape)
|
||||
|
||||
def _assemble_patch_grid(
|
||||
self, features: torch.Tensor, crop_shape: torch.Tensor
|
||||
) -> torch.Tensor:
|
||||
"""Assemble projected patches into a 2-D tile grid with newline columns."""
|
||||
_, hw, dim = features.shape
|
||||
patch_side = int(hw**0.5)
|
||||
|
||||
width_tiles = int(crop_shape[0].item())
|
||||
height_tiles = int(crop_shape[1].item())
|
||||
|
||||
@@ -519,7 +534,6 @@ class DeepseekOCRForCausalLM(nn.Module, SupportsMultiModal, SupportsPP, Supports
|
||||
height_tiles * patch_side, 1, dim
|
||||
)
|
||||
features = torch.cat([features, newline], dim=1)
|
||||
|
||||
return features.view(-1, dim)
|
||||
|
||||
def _pixel_values_to_embedding(
|
||||
@@ -614,3 +628,359 @@ class DeepseekOCRForCausalLM(nn.Module, SupportsMultiModal, SupportsPP, Supports
|
||||
connector="projector",
|
||||
tower_model=["sam_model", "vision_model"],
|
||||
)
|
||||
|
||||
# -- Fixed spatial constants (computed from BASE_SIZE / IMAGE_SIZE) --
|
||||
|
||||
@property
|
||||
def image_side(self) -> int:
|
||||
"""Number of output grid cells per spatial dim for a global image."""
|
||||
return math.ceil((BASE_SIZE // 16) / 4) # 16
|
||||
|
||||
@property
|
||||
def global_image_output_token(self) -> int:
|
||||
"""Tokens per global image (grid + one newline per row)."""
|
||||
return self.image_side * (self.image_side + 1) # 272
|
||||
|
||||
@property
|
||||
def patch_side(self) -> int:
|
||||
"""Number of output grid cells per spatial dim for a local patch."""
|
||||
return math.ceil((IMAGE_SIZE // 16) / 4) # 10
|
||||
|
||||
@property
|
||||
def single_patch_output_token(self) -> int:
|
||||
"""Tokens per local patch (square grid, no newlines)."""
|
||||
return self.patch_side * self.patch_side # 100
|
||||
|
||||
# -- SupportsEncoderCudaGraph protocol methods --
|
||||
|
||||
def _get_num_input_output_tokens(
|
||||
self,
|
||||
image_spatial_crop: torch.Tensor | None = None,
|
||||
) -> tuple[int, int, int, int]:
|
||||
"""
|
||||
Return (num_input_tokens, num_output_tokens, global_output_token,
|
||||
local_output_token) for a single image described by
|
||||
``image_spatial_crop``.
|
||||
"""
|
||||
is_tiled = False
|
||||
if image_spatial_crop is not None:
|
||||
is_tiled = image_spatial_crop[0] > 1 or image_spatial_crop[1] > 1
|
||||
|
||||
# Compute input size:
|
||||
global_input_side = BASE_SIZE // 16 # 64
|
||||
local_input_side = IMAGE_SIZE // 16 # 40
|
||||
num_input_tokens = global_input_side**2
|
||||
|
||||
if is_tiled:
|
||||
num_patches = image_spatial_crop.prod(dim=-1)
|
||||
num_input_tokens += num_patches * (local_input_side**2)
|
||||
|
||||
global_output_token = self.global_image_output_token
|
||||
num_output_tokens = global_output_token
|
||||
|
||||
local_output_token = 0
|
||||
if is_tiled:
|
||||
local_output_token = num_patches * self.single_patch_output_token
|
||||
num_output_tokens += local_output_token
|
||||
|
||||
return (
|
||||
num_input_tokens,
|
||||
num_output_tokens,
|
||||
global_output_token,
|
||||
local_output_token,
|
||||
)
|
||||
|
||||
def get_encoder_cudagraph_config(self):
|
||||
return EncoderCudaGraphConfig(
|
||||
modalities=["image"],
|
||||
buffer_keys=["pixel_values"],
|
||||
out_hidden_size=self.projector_config.n_embed,
|
||||
enable_dual_path_graph=True,
|
||||
global_token_per_image=self.global_image_output_token,
|
||||
local_token_per_patch=self.single_patch_output_token,
|
||||
)
|
||||
|
||||
def get_encoder_cudagraph_budget_range(
|
||||
self,
|
||||
vllm_config,
|
||||
) -> tuple[int, int]:
|
||||
# Min budget: at least one global image with newline tokens (without patches).
|
||||
min_budget = self.global_image_output_token
|
||||
max_budget = min(
|
||||
vllm_config.scheduler_config.max_num_batched_tokens,
|
||||
self.model_config.max_model_len,
|
||||
)
|
||||
return (min_budget, max_budget)
|
||||
|
||||
def get_encoder_cudagraph_item_specs(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
) -> list[EncoderItemSpec]:
|
||||
item_specs = []
|
||||
for image_spatial_crop in mm_kwargs["images_spatial_crop"]:
|
||||
(
|
||||
num_input_tokens,
|
||||
num_output_tokens,
|
||||
global_output_token,
|
||||
local_output_token,
|
||||
) = self._get_num_input_output_tokens(image_spatial_crop)
|
||||
item_specs.append(
|
||||
EncoderItemSpec(
|
||||
input_size=num_input_tokens,
|
||||
output_tokens=num_output_tokens,
|
||||
global_output_tokens=global_output_token,
|
||||
local_output_tokens=local_output_token,
|
||||
)
|
||||
)
|
||||
return item_specs
|
||||
|
||||
def select_encoder_cudagraph_items(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
indices: list[int],
|
||||
) -> dict[str, Any]:
|
||||
pixel_values = mm_kwargs["pixel_values"]
|
||||
images_crop = mm_kwargs["images_crop"]
|
||||
images_spatial_crop = mm_kwargs["images_spatial_crop"]
|
||||
|
||||
if len(indices) == 0:
|
||||
return {
|
||||
"pixel_values": pixel_values[:0],
|
||||
"images_crop": images_crop[:0],
|
||||
"images_spatial_crop": images_spatial_crop[:0],
|
||||
}
|
||||
|
||||
is_tiled = (images_spatial_crop[:, 0] > 1) | (images_spatial_crop[:, 1] > 1)
|
||||
patches_per_image = torch.where(is_tiled, images_spatial_crop.prod(dim=-1), 0)
|
||||
cum_patches = [0]
|
||||
for num_patches in patches_per_image:
|
||||
cum_patches.append(cum_patches[-1] + int(num_patches))
|
||||
|
||||
selected_pv = pixel_values[indices]
|
||||
selected_ic = torch.cat(
|
||||
[images_crop[cum_patches[i] : cum_patches[i + 1]] for i in indices]
|
||||
)
|
||||
selected_sp = images_spatial_crop[indices]
|
||||
|
||||
return {
|
||||
"pixel_values": selected_pv,
|
||||
"images_crop": selected_ic,
|
||||
"images_spatial_crop": selected_sp,
|
||||
}
|
||||
|
||||
def prepare_encoder_cudagraph_capture_inputs(
|
||||
self,
|
||||
token_budget: int,
|
||||
max_batch_size: int,
|
||||
max_frames_per_batch: int,
|
||||
device: torch.device,
|
||||
dtype: torch.dtype,
|
||||
path: str = "default",
|
||||
):
|
||||
assert path in ("global", "local")
|
||||
|
||||
if path == "global":
|
||||
max_num_images = token_budget // self.global_image_output_token
|
||||
max_batch_size = min(max_batch_size, max_num_images)
|
||||
dummy_pixel_values = torch.randn(
|
||||
max_batch_size,
|
||||
3,
|
||||
BASE_SIZE,
|
||||
BASE_SIZE,
|
||||
device=device,
|
||||
dtype=dtype,
|
||||
)
|
||||
values = {"pixel_values": dummy_pixel_values}
|
||||
else:
|
||||
max_num_patches = token_budget // self.single_patch_output_token
|
||||
dummy_images_crop = torch.randn(
|
||||
max_num_patches,
|
||||
3,
|
||||
IMAGE_SIZE,
|
||||
IMAGE_SIZE,
|
||||
device=device,
|
||||
dtype=dtype,
|
||||
)
|
||||
values = {"images_crop": dummy_images_crop}
|
||||
|
||||
return EncoderCudaGraphCaptureInputs(values=values)
|
||||
|
||||
def prepare_encoder_cudagraph_replay_buffers(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
max_batch_size: int,
|
||||
max_frames_per_batch: int,
|
||||
path: str = "default",
|
||||
):
|
||||
assert path in ("global", "local")
|
||||
|
||||
if path == "global":
|
||||
values = {"pixel_values": mm_kwargs["pixel_values"]}
|
||||
else:
|
||||
values = {"images_crop": mm_kwargs["images_crop"]}
|
||||
|
||||
return EncoderCudaGraphReplayBuffers(values=values)
|
||||
|
||||
def _batched_encoder_forward_global_path(
|
||||
self,
|
||||
pixel_values: torch.Tensor,
|
||||
) -> torch.Tensor:
|
||||
"""
|
||||
Encode batched global images with newline tokens inserted.
|
||||
Output shape: ``[B * 272, n_embed]``.
|
||||
"""
|
||||
bsz = pixel_values.shape[0]
|
||||
global_features_1 = self.sam_model(pixel_values)
|
||||
global_features_2 = self.vision_model(pixel_values, global_features_1)
|
||||
features = torch.cat(
|
||||
(
|
||||
global_features_2[:, 1:],
|
||||
global_features_1.flatten(2).permute(0, 2, 1),
|
||||
),
|
||||
dim=-1,
|
||||
)
|
||||
features = self.projector(features)
|
||||
side = self.image_side
|
||||
dim = features.shape[-1]
|
||||
features = features.view(bsz, side, side, dim)
|
||||
newline = self.image_newline.view(1, 1, 1, dim).expand(bsz, side, 1, dim)
|
||||
features = torch.cat([features, newline], dim=2)
|
||||
return features.view(-1, dim)
|
||||
|
||||
def _batched_encoder_forward_local_path(
|
||||
self,
|
||||
images_crop: torch.Tensor,
|
||||
) -> torch.Tensor:
|
||||
"""
|
||||
Encode local patches without newline insertion (newlines are added later
|
||||
in ``postprocess_encoder_output`` via ``_assemble_patch_grid``).
|
||||
Output shape: ``[P * 100, n_embed]``.
|
||||
"""
|
||||
features_1 = self.sam_model(images_crop)
|
||||
features_2 = self.vision_model(images_crop, features_1)
|
||||
features = torch.cat(
|
||||
(
|
||||
features_2[:, 1:],
|
||||
features_1.flatten(2).permute(0, 2, 1),
|
||||
),
|
||||
dim=-1,
|
||||
)
|
||||
features = self.projector(features)
|
||||
return features.view(-1, features.shape[-1])
|
||||
|
||||
def encoder_cudagraph_forward(
|
||||
self,
|
||||
values: dict[str, torch.Tensor],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
assert path in ("global", "local")
|
||||
|
||||
if path == "global":
|
||||
pixel_values = values["pixel_values"]
|
||||
return self._batched_encoder_forward_global_path(pixel_values)
|
||||
else:
|
||||
images_crop = values["images_crop"]
|
||||
return self._batched_encoder_forward_local_path(images_crop)
|
||||
|
||||
def encoder_eager_forward(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
"""Eager encoder forward with optional per-path execution.
|
||||
|
||||
``path="default"``: full forward (global + local + assembly).
|
||||
``path="global"``: global-only batched forward with newlines.
|
||||
``path="local"``: local-only batched forward without newlines.
|
||||
"""
|
||||
if path == "default":
|
||||
# Original eager implementation: process each image one by one
|
||||
# (with both global and local paths) and concatenate results.
|
||||
image_input = DeepseekOCRImagePixelInputs(
|
||||
type="pixel_values",
|
||||
data=mm_kwargs["pixel_values"],
|
||||
images_crop=mm_kwargs["images_crop"],
|
||||
images_spatial_crop=mm_kwargs["images_spatial_crop"],
|
||||
)
|
||||
vision_embeddings = self._process_image_input(image_input)
|
||||
return torch.cat(vision_embeddings, dim=0)
|
||||
|
||||
assert path in ("global", "local")
|
||||
if path == "global":
|
||||
pixel_values = mm_kwargs["pixel_values"]
|
||||
return self._batched_encoder_forward_global_path(pixel_values)
|
||||
else:
|
||||
images_crop = mm_kwargs["images_crop"]
|
||||
return self._batched_encoder_forward_local_path(images_crop)
|
||||
|
||||
def postprocess_encoder_output(
|
||||
self,
|
||||
output: torch.Tensor,
|
||||
indices: list[int],
|
||||
per_item_out_tokens: list[int],
|
||||
dest: dict[int, torch.Tensor] | list[torch.Tensor | None],
|
||||
clone: bool = False,
|
||||
batch_mm_kwargs: dict[str, Any] | None = None,
|
||||
local_output: torch.Tensor | None = None,
|
||||
) -> None:
|
||||
"""
|
||||
Assemble per-image embeddings from global and local encoder outputs.
|
||||
|
||||
``output`` contains global-image features with newlines already
|
||||
inserted (from CUDA graph replay or eager fallback):
|
||||
``[B * 272, n_embed]``.
|
||||
|
||||
``local_output`` contains local-patch features without
|
||||
newlines (from CUDA graph replay or eager fallback):
|
||||
``[P * 100, n_embed]``. May be ``None`` if no patches in batch.
|
||||
|
||||
This method:
|
||||
1. Splits ``output`` into per-image global portions.
|
||||
2. Splits ``local_output`` into per-image patch groups.
|
||||
3. For each image: assembles patch grid with newlines via
|
||||
``_assemble_patch_grid``, then concatenates
|
||||
``[local_tiled, global, view_seperator]``.
|
||||
"""
|
||||
bsz = len(indices)
|
||||
n_embed = output.shape[-1]
|
||||
|
||||
images_spatial_crop = batch_mm_kwargs["images_spatial_crop"]
|
||||
is_tiled = (images_spatial_crop[:, 0] > 1) | (images_spatial_crop[:, 1] > 1)
|
||||
num_patches = [
|
||||
int(np) for np in torch.where(is_tiled, images_spatial_crop.prod(dim=-1), 0)
|
||||
]
|
||||
total_patches = sum(num_patches)
|
||||
|
||||
global_part = output[: bsz * self.global_image_output_token].reshape(
|
||||
bsz, self.global_image_output_token, n_embed
|
||||
)
|
||||
|
||||
# Split local output into per-patch groups.
|
||||
local_flat = None
|
||||
if total_patches > 0 and local_output is not None:
|
||||
local_flat = local_output[: total_patches * self.single_patch_output_token]
|
||||
local_flat = local_flat.reshape(
|
||||
total_patches, self.single_patch_output_token, n_embed
|
||||
)
|
||||
|
||||
cur_patch = 0
|
||||
for i, idx in enumerate(indices):
|
||||
num_patch = num_patches[i]
|
||||
single_image_output: list[torch.Tensor] = []
|
||||
|
||||
# 1. Process local patches: assemble tile grid, add 1 newline per row.
|
||||
if num_patch > 0 and local_flat is not None:
|
||||
patches = local_flat[cur_patch : cur_patch + num_patch]
|
||||
cur_patch += num_patch
|
||||
single_image_output.append(
|
||||
self._assemble_patch_grid(patches, images_spatial_crop[i])
|
||||
)
|
||||
|
||||
# 2. Global image: newlines already inserted.
|
||||
single_image_output.append(global_part[i])
|
||||
|
||||
# 3. Add view separator for each image.
|
||||
single_image_output.append(self.view_seperator[None, :])
|
||||
|
||||
# 4. Save final outputs for each image.
|
||||
dest[idx] = torch.cat(single_image_output, dim=0)
|
||||
|
||||
@@ -2068,6 +2068,7 @@ class Glm4vForConditionalGeneration(
|
||||
max_frames_per_batch: int,
|
||||
device: torch.device,
|
||||
dtype: torch.dtype,
|
||||
path: str = "default",
|
||||
):
|
||||
from vllm.v1.worker.encoder_cudagraph_defs import (
|
||||
EncoderCudaGraphCaptureInputs,
|
||||
@@ -2145,6 +2146,7 @@ class Glm4vForConditionalGeneration(
|
||||
mm_kwargs: dict[str, Any],
|
||||
max_batch_size: int,
|
||||
max_frames_per_batch: int,
|
||||
path: str = "default",
|
||||
):
|
||||
modality = self.get_input_modality(mm_kwargs)
|
||||
grid_thw_list = self._get_grid_thw_by_modality(mm_kwargs)
|
||||
@@ -2170,6 +2172,7 @@ class Glm4vForConditionalGeneration(
|
||||
def encoder_cudagraph_forward(
|
||||
self,
|
||||
values: dict[str, torch.Tensor],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
pixel_values = values.pop("pixel_values")
|
||||
metadata = values
|
||||
@@ -2178,6 +2181,7 @@ class Glm4vForConditionalGeneration(
|
||||
def encoder_eager_forward(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
pixel_values = self._get_pixel_values_by_modality(mm_kwargs)
|
||||
grid_thw = self._get_grid_thw_by_modality(mm_kwargs)
|
||||
|
||||
@@ -1623,6 +1623,7 @@ class SupportsEncoderCudaGraph(Protocol):
|
||||
dest: dict[int, torch.Tensor] | list[torch.Tensor | None],
|
||||
clone: bool = False,
|
||||
batch_mm_kwargs: dict[str, Any] | None = None,
|
||||
local_output: torch.Tensor | None = None,
|
||||
) -> None:
|
||||
"""
|
||||
Post-process encoder output, directly call scatter_output_slices by default.
|
||||
@@ -1643,6 +1644,7 @@ class SupportsEncoderCudaGraph(Protocol):
|
||||
max_frames_per_batch: int,
|
||||
device: torch.device,
|
||||
dtype: torch.dtype,
|
||||
path: str = "default",
|
||||
) -> "EncoderCudaGraphCaptureInputs":
|
||||
"""Create dummy inputs and buffers for CUDA graph capture."""
|
||||
...
|
||||
@@ -1652,6 +1654,7 @@ class SupportsEncoderCudaGraph(Protocol):
|
||||
mm_kwargs: dict[str, Any],
|
||||
max_batch_size: int,
|
||||
max_frames_per_batch: int,
|
||||
path: str = "default",
|
||||
) -> "EncoderCudaGraphReplayBuffers":
|
||||
"""Compute buffer values from actual batch inputs for replay."""
|
||||
...
|
||||
@@ -1659,6 +1662,7 @@ class SupportsEncoderCudaGraph(Protocol):
|
||||
def encoder_cudagraph_forward(
|
||||
self,
|
||||
inputs: dict[str, torch.Tensor],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
"""Run the encoder forward pass with precomputed buffers.
|
||||
|
||||
@@ -1669,6 +1673,7 @@ class SupportsEncoderCudaGraph(Protocol):
|
||||
def encoder_eager_forward(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
"""Run the encoder forward pass without precomputed buffers.
|
||||
|
||||
|
||||
@@ -1004,6 +1004,7 @@ class InternVLChatModel(
|
||||
max_frames_per_batch: int,
|
||||
device: torch.device,
|
||||
dtype: torch.dtype,
|
||||
path: str = "default",
|
||||
):
|
||||
from vllm.v1.worker.encoder_cudagraph_defs import (
|
||||
EncoderCudaGraphCaptureInputs,
|
||||
@@ -1026,6 +1027,7 @@ class InternVLChatModel(
|
||||
mm_kwargs: dict[str, Any],
|
||||
max_batch_size: int,
|
||||
max_frames_per_batch: int,
|
||||
path: str = "default",
|
||||
):
|
||||
from vllm.v1.worker.encoder_cudagraph_defs import (
|
||||
EncoderCudaGraphReplayBuffers,
|
||||
@@ -1042,6 +1044,7 @@ class InternVLChatModel(
|
||||
def encoder_cudagraph_forward(
|
||||
self,
|
||||
values: dict[str, torch.Tensor],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
# The graph is always captured with pixel_values_flat as the input
|
||||
# buffer. During video replay the manager copies video tiles into
|
||||
@@ -1054,6 +1057,7 @@ class InternVLChatModel(
|
||||
def encoder_eager_forward(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
if self.get_input_modality(mm_kwargs) == "image":
|
||||
pixel_values = mm_kwargs["pixel_values_flat"]
|
||||
|
||||
@@ -1122,6 +1122,7 @@ class Lfm2VLForConditionalGeneration(
|
||||
max_frames_per_batch: int,
|
||||
device: torch.device,
|
||||
dtype: torch.dtype,
|
||||
path: str = "default",
|
||||
):
|
||||
from vllm.v1.worker.encoder_cudagraph_defs import (
|
||||
EncoderCudaGraphCaptureInputs,
|
||||
@@ -1167,6 +1168,7 @@ class Lfm2VLForConditionalGeneration(
|
||||
mm_kwargs: dict[str, Any],
|
||||
max_batch_size: int,
|
||||
max_frames_per_batch: int,
|
||||
path: str = "default",
|
||||
):
|
||||
from vllm.v1.worker.encoder_cudagraph_defs import (
|
||||
EncoderCudaGraphReplayBuffers,
|
||||
@@ -1181,6 +1183,7 @@ class Lfm2VLForConditionalGeneration(
|
||||
def encoder_cudagraph_forward(
|
||||
self,
|
||||
values: dict[str, torch.Tensor],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
embeddings = self.vision_tower.vision_model.embeddings
|
||||
pixel_values = values["pixel_values_packed"].to(
|
||||
@@ -1208,6 +1211,7 @@ class Lfm2VLForConditionalGeneration(
|
||||
def encoder_eager_forward(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
image_input = LFM2VLImageInputs(
|
||||
type="pixel_values",
|
||||
|
||||
@@ -926,6 +926,7 @@ class Llama4ForConditionalGeneration(
|
||||
max_frames_per_batch: int,
|
||||
device: torch.device,
|
||||
dtype: torch.dtype,
|
||||
path: str = "default",
|
||||
):
|
||||
from vllm.v1.worker.encoder_cudagraph_defs import (
|
||||
EncoderCudaGraphCaptureInputs,
|
||||
@@ -954,6 +955,7 @@ class Llama4ForConditionalGeneration(
|
||||
mm_kwargs: dict[str, Any],
|
||||
max_batch_size: int,
|
||||
max_frames_per_batch: int,
|
||||
path: str = "default",
|
||||
):
|
||||
from vllm.v1.worker.encoder_cudagraph_defs import (
|
||||
EncoderCudaGraphReplayBuffers,
|
||||
@@ -966,6 +968,7 @@ class Llama4ForConditionalGeneration(
|
||||
def encoder_cudagraph_forward(
|
||||
self,
|
||||
inputs: dict[str, torch.Tensor],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
return self.encode_image_chunks(
|
||||
inputs["pixel_values"],
|
||||
@@ -975,6 +978,7 @@ class Llama4ForConditionalGeneration(
|
||||
def encoder_eager_forward(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
return self.encode_image_chunks(
|
||||
mm_kwargs["pixel_values"],
|
||||
|
||||
@@ -1856,6 +1856,7 @@ class Qwen2_5_VLForConditionalGeneration(
|
||||
max_frames_per_batch: int,
|
||||
device: torch.device,
|
||||
dtype: torch.dtype,
|
||||
path: str = "default",
|
||||
):
|
||||
from vllm.v1.worker.encoder_cudagraph_defs import (
|
||||
EncoderCudaGraphCaptureInputs,
|
||||
@@ -1952,6 +1953,7 @@ class Qwen2_5_VLForConditionalGeneration(
|
||||
mm_kwargs: dict[str, Any],
|
||||
max_batch_size: int,
|
||||
max_frames_per_batch: int,
|
||||
path: str = "default",
|
||||
):
|
||||
modality = self.get_input_modality(mm_kwargs)
|
||||
grid_thw_list = self._get_grid_thw_by_modality(mm_kwargs)
|
||||
@@ -1983,6 +1985,7 @@ class Qwen2_5_VLForConditionalGeneration(
|
||||
def encoder_cudagraph_forward(
|
||||
self,
|
||||
values: dict[str, torch.Tensor],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
pixel_values = values.pop("pixel_values")
|
||||
metadata = values
|
||||
@@ -1991,6 +1994,7 @@ class Qwen2_5_VLForConditionalGeneration(
|
||||
def encoder_eager_forward(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
pixel_values = self._get_pixel_values_by_modality(mm_kwargs)
|
||||
grid_thw = self._get_grid_thw_by_modality(mm_kwargs)
|
||||
|
||||
@@ -1579,6 +1579,7 @@ class Qwen2VLForConditionalGeneration(
|
||||
max_frames_per_batch: int,
|
||||
device: torch.device,
|
||||
dtype: torch.dtype,
|
||||
path: str = "default",
|
||||
):
|
||||
from vllm.v1.worker.encoder_cudagraph_defs import (
|
||||
EncoderCudaGraphCaptureInputs,
|
||||
@@ -1644,6 +1645,7 @@ class Qwen2VLForConditionalGeneration(
|
||||
mm_kwargs: dict[str, Any],
|
||||
max_batch_size: int,
|
||||
max_frames_per_batch: int,
|
||||
path: str = "default",
|
||||
) -> EncoderCudaGraphReplayBuffers:
|
||||
modality = self.get_input_modality(mm_kwargs)
|
||||
grid_thw_list = self._get_grid_thw_by_modality(mm_kwargs)
|
||||
@@ -1667,6 +1669,7 @@ class Qwen2VLForConditionalGeneration(
|
||||
def encoder_cudagraph_forward(
|
||||
self,
|
||||
values: dict[str, torch.Tensor],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
pixel_values = values.pop("pixel_values")
|
||||
metadata = values
|
||||
@@ -1675,6 +1678,7 @@ class Qwen2VLForConditionalGeneration(
|
||||
def encoder_eager_forward(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
pixel_values = self._get_pixel_values_by_modality(mm_kwargs)
|
||||
grid_thw = self._get_grid_thw_by_modality(mm_kwargs)
|
||||
|
||||
@@ -1918,6 +1918,7 @@ class Qwen3VLForConditionalGeneration(
|
||||
max_frames_per_batch: int,
|
||||
device: torch.device,
|
||||
dtype: torch.dtype,
|
||||
path: str = "default",
|
||||
):
|
||||
from vllm.v1.worker.encoder_cudagraph_defs import (
|
||||
EncoderCudaGraphCaptureInputs,
|
||||
@@ -1998,6 +1999,7 @@ class Qwen3VLForConditionalGeneration(
|
||||
mm_kwargs: dict[str, Any],
|
||||
max_batch_size: int,
|
||||
max_frames_per_batch: int,
|
||||
path: str = "default",
|
||||
):
|
||||
modality = self.get_input_modality(mm_kwargs)
|
||||
grid_thw_list = self._get_grid_thw_by_modality(mm_kwargs)
|
||||
@@ -2023,6 +2025,7 @@ class Qwen3VLForConditionalGeneration(
|
||||
def encoder_cudagraph_forward(
|
||||
self,
|
||||
values: dict[str, torch.Tensor],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
pixel_values = values.pop("pixel_values")
|
||||
metadata = values
|
||||
@@ -2031,6 +2034,7 @@ class Qwen3VLForConditionalGeneration(
|
||||
def encoder_eager_forward(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
pixel_values = self._get_pixel_values_by_modality(mm_kwargs)
|
||||
grid_thw = self._get_grid_thw_by_modality(mm_kwargs)
|
||||
|
||||
@@ -804,6 +804,7 @@ class Step3VLForConditionalGeneration(
|
||||
max_frames_per_batch: int,
|
||||
device: torch.device,
|
||||
dtype: torch.dtype,
|
||||
path: str = "default",
|
||||
):
|
||||
from vllm.v1.worker.encoder_cudagraph_defs import (
|
||||
EncoderCudaGraphCaptureInputs,
|
||||
@@ -857,6 +858,7 @@ class Step3VLForConditionalGeneration(
|
||||
def encoder_cudagraph_forward(
|
||||
self,
|
||||
values: dict[str, torch.Tensor],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
# Graph captures only the compute (vision model + conv projector).
|
||||
# Per-item merge happens CPU-side in finalize_encoder_cudagraph_output
|
||||
@@ -884,6 +886,7 @@ class Step3VLForConditionalGeneration(
|
||||
def encoder_eager_forward(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
path: str = "default",
|
||||
) -> torch.Tensor:
|
||||
image_input = Step3VLImagePixelInputs(
|
||||
type="pixel_values",
|
||||
@@ -902,6 +905,7 @@ class Step3VLForConditionalGeneration(
|
||||
dest: dict[int, torch.Tensor] | list[torch.Tensor | None],
|
||||
clone: bool = False,
|
||||
batch_mm_kwargs: dict[str, Any] | None = None,
|
||||
local_output: torch.Tensor | None = None,
|
||||
):
|
||||
"""CPU-side per-item merge after graph replay.
|
||||
|
||||
@@ -959,6 +963,7 @@ class Step3VLForConditionalGeneration(
|
||||
mm_kwargs: dict[str, Any],
|
||||
max_batch_size: int,
|
||||
max_frames_per_batch: int,
|
||||
path: str = "default",
|
||||
):
|
||||
from vllm.v1.worker.encoder_cudagraph_defs import (
|
||||
EncoderCudaGraphReplayBuffers,
|
||||
|
||||
@@ -152,20 +152,40 @@ class EncoderCudaGraphManager:
|
||||
and vllm_config.parallel_config.tensor_parallel_size > 1
|
||||
)
|
||||
|
||||
self.budget_graphs: dict[int, BudgetGraphMetadata] = {}
|
||||
self.budget_graphs: dict[str, dict[int, BudgetGraphMetadata]] = {}
|
||||
self.graph_pool: Any | None = None
|
||||
self.graph_hits = 0
|
||||
self.graph_misses = 0
|
||||
self.log_stats_interval = 100
|
||||
|
||||
logger.info(
|
||||
"EncoderCudaGraphManager initialized with "
|
||||
"budgets=%s, max_batch_size=%d, max_frames_per_batch=%s, use_dp=%s",
|
||||
self.token_budgets,
|
||||
self.max_batch_size,
|
||||
self.max_frames_per_batch,
|
||||
self.use_dp,
|
||||
)
|
||||
if self.config.enable_dual_path_graph:
|
||||
max_budget = self.token_budgets[-1]
|
||||
self.global_token_budgets = self._generate_budgets(
|
||||
self.config.global_token_per_image,
|
||||
max_budget,
|
||||
)
|
||||
self.local_token_budgets = self._generate_budgets(
|
||||
self.config.local_token_per_patch,
|
||||
max_budget,
|
||||
)
|
||||
# When `image_width <= 640 and image_height <= 640`, the mm inputs
|
||||
# will only contain global image, without generating local patches.
|
||||
self.local_token_budgets.insert(0, 0)
|
||||
logger.info(
|
||||
"EncoderCudaGraphManager dual-path mode: "
|
||||
"global_budgets=%s, local_budgets=%s",
|
||||
self.global_token_budgets,
|
||||
self.local_token_budgets,
|
||||
)
|
||||
else:
|
||||
logger.info(
|
||||
"EncoderCudaGraphManager initialized with "
|
||||
"budgets=%s, max_batch_size=%d, max_frames_per_batch=%s, use_dp=%s",
|
||||
self.token_budgets,
|
||||
self.max_batch_size,
|
||||
self.max_frames_per_batch,
|
||||
self.use_dp,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _generate_budgets(min_budget: int, max_budget: int) -> list[int]:
|
||||
@@ -186,25 +206,50 @@ class EncoderCudaGraphManager:
|
||||
|
||||
def clear(self) -> None:
|
||||
"""Release captured encoder CUDA graphs and the manager-local pool."""
|
||||
self.budget_graphs.clear()
|
||||
for graph_set in self.budget_graphs.values():
|
||||
graph_set.clear()
|
||||
self.graph_pool = None
|
||||
|
||||
def capture(self, graph_pool: Any):
|
||||
"""Capture CUDA graphs for all token budgets."""
|
||||
self.graph_pool = graph_pool
|
||||
|
||||
if self.config.enable_dual_path_graph:
|
||||
for token_budget in sorted(self.global_token_budgets, reverse=True):
|
||||
self._capture_budget_graph(token_budget, path="global")
|
||||
for token_budget in sorted(self.local_token_budgets, reverse=True):
|
||||
if token_budget == 0:
|
||||
continue
|
||||
self._capture_budget_graph(token_budget, path="local")
|
||||
logger.info(
|
||||
"Encoder CUDA graph capture complete. "
|
||||
"Captured %d global + %d local budget graphs.",
|
||||
len(self.budget_graphs["global"]),
|
||||
len(self.budget_graphs["local"]),
|
||||
)
|
||||
return
|
||||
|
||||
for token_budget in sorted(self.token_budgets, reverse=True):
|
||||
self._capture_budget_graph(token_budget)
|
||||
|
||||
logger.info(
|
||||
"Encoder CUDA graph capture complete. Captured %d budget graphs.",
|
||||
len(self.budget_graphs),
|
||||
len(self.budget_graphs["default"]),
|
||||
)
|
||||
|
||||
def get_num_graphs_to_capture(self) -> int:
|
||||
if self.config.enable_dual_path_graph:
|
||||
return len(self.global_token_budgets) + len(self.local_token_budgets)
|
||||
return len(self.token_budgets)
|
||||
|
||||
def _capture_budget_graph(self, token_budget: int):
|
||||
def _get_graph_set(self, path: str = "default") -> dict[int, BudgetGraphMetadata]:
|
||||
# Lazy init global/local graph sets for dual-path models, or default graph
|
||||
# set for single-path models.
|
||||
if path not in self.budget_graphs:
|
||||
self.budget_graphs[path] = {}
|
||||
return self.budget_graphs[path]
|
||||
|
||||
def _capture_budget_graph(self, token_budget: int, path: str = "default"):
|
||||
"""Capture CUDA graph for a single token budget."""
|
||||
logger.debug(
|
||||
"Capturing encoder cudagraph for budget=%d, max_batch_size=%d, "
|
||||
@@ -214,26 +259,29 @@ class EncoderCudaGraphManager:
|
||||
self.max_frames_per_batch,
|
||||
)
|
||||
|
||||
graph_set = self._get_graph_set(path)
|
||||
|
||||
capture_inputs = self.model.prepare_encoder_cudagraph_capture_inputs(
|
||||
token_budget,
|
||||
self.max_batch_size,
|
||||
self.max_frames_per_batch,
|
||||
self.device,
|
||||
self.dtype,
|
||||
path,
|
||||
)
|
||||
|
||||
values = capture_inputs.values
|
||||
|
||||
with torch.inference_mode():
|
||||
output = self.model.encoder_cudagraph_forward({**values})
|
||||
output = self.model.encoder_cudagraph_forward({**values}, path=path)
|
||||
output_buffer = torch.empty_like(output)
|
||||
|
||||
graph = torch.cuda.CUDAGraph()
|
||||
with torch.inference_mode(), torch.cuda.graph(graph, pool=self.graph_pool):
|
||||
output = self.model.encoder_cudagraph_forward({**values})
|
||||
output = self.model.encoder_cudagraph_forward({**values}, path=path)
|
||||
output_buffer.copy_(output)
|
||||
|
||||
self.budget_graphs[token_budget] = BudgetGraphMetadata(
|
||||
graph_set[token_budget] = BudgetGraphMetadata(
|
||||
token_budget=token_budget,
|
||||
max_batch_size=self.max_batch_size,
|
||||
max_frames_per_batch=self.max_frames_per_batch,
|
||||
@@ -243,14 +291,15 @@ class EncoderCudaGraphManager:
|
||||
)
|
||||
|
||||
def _find_smallest_fitting_budget_given_tokens(
|
||||
self, total_tokens: int
|
||||
self, total_tokens: int, budgets: list[int] | None = None
|
||||
) -> int | None:
|
||||
"""Find smallest budget >= total_tokens.
|
||||
|
||||
Returns:
|
||||
Token budget if found, None if no fitting budget.
|
||||
"""
|
||||
for budget in self.token_budgets:
|
||||
budgets = budgets if budgets is not None else self.token_budgets
|
||||
for budget in budgets:
|
||||
if budget >= total_tokens:
|
||||
return budget
|
||||
return None
|
||||
@@ -275,35 +324,40 @@ class EncoderCudaGraphManager:
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
token_budget: int,
|
||||
path: str = "default",
|
||||
) -> torch.Tensor | None:
|
||||
"""Execute budget graph.
|
||||
|
||||
Args:
|
||||
mm_kwargs: Multimodal inputs for the batch.
|
||||
token_budget: Token budget to use.
|
||||
|
||||
path: Path for the graph. Should be one of ["default", "global", "local"].
|
||||
Returns:
|
||||
Encoder outputs, or None if graph not captured.
|
||||
"""
|
||||
graph_set = self._get_graph_set(path)
|
||||
num_items = len(self._get_item_specs(mm_kwargs))
|
||||
if token_budget not in self.budget_graphs:
|
||||
|
||||
if token_budget not in graph_set:
|
||||
self.graph_misses += num_items
|
||||
return None
|
||||
|
||||
graph_meta = self.budget_graphs[token_budget]
|
||||
graph_meta = graph_set[token_budget]
|
||||
|
||||
replay = self.model.prepare_encoder_cudagraph_replay_buffers(
|
||||
mm_kwargs,
|
||||
self.max_batch_size,
|
||||
self.max_frames_per_batch,
|
||||
path,
|
||||
)
|
||||
|
||||
# Copy metadata buffers using keys from config.buffer_keys.
|
||||
for key in self.config.buffer_keys:
|
||||
# Copy replay buffers into graph input buffers. Iterate over the
|
||||
# graph's own buffer keys (which may differ per path for dual-path
|
||||
# models) rather than the global config.buffer_keys.
|
||||
for key, buf in graph_meta.input_buffers.items():
|
||||
src = replay.values.get(key)
|
||||
if src is None:
|
||||
continue
|
||||
buf = graph_meta.input_buffers[key]
|
||||
if src.ndim == 0:
|
||||
buf.copy_(src)
|
||||
else:
|
||||
@@ -329,6 +383,11 @@ class EncoderCudaGraphManager:
|
||||
image would overflow either constraint), find the smallest fitting
|
||||
budget once for that batch.
|
||||
|
||||
For dual-path models (``enable_dual_path_graph=True``), two independent
|
||||
graph sets are used: one for global images, one for local patches.
|
||||
Budgets are found independently per path; if only one path fits, the
|
||||
other falls back to eager via partial fallback.
|
||||
|
||||
By exchange argument, greedy smallest-first packing minimises eager
|
||||
fallbacks -- any other ordering yields a higher token sum in some batch,
|
||||
making that batch more likely to exceed the budget.
|
||||
@@ -340,6 +399,15 @@ class EncoderCudaGraphManager:
|
||||
always satisfy total_tokens <= max_budget and therefore
|
||||
always find a valid budget (no miss).
|
||||
"""
|
||||
if self.config.enable_dual_path_graph:
|
||||
return self._execute_local_dual_path(mm_kwargs)
|
||||
return self._execute_local_single_path(mm_kwargs)
|
||||
|
||||
def _execute_local_single_path(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
) -> list[torch.Tensor]:
|
||||
"""Single-path greedy-packing execution (original behaviour)."""
|
||||
item_specs = self._get_item_specs(mm_kwargs)
|
||||
num_items = len(item_specs)
|
||||
max_budget = self.token_budgets[-1]
|
||||
@@ -441,6 +509,166 @@ class EncoderCudaGraphManager:
|
||||
# Return in original batch order (caller maps outputs to token positions)
|
||||
return [outputs_by_orig_idx[i] for i in range(num_items)]
|
||||
|
||||
def _execute_local_dual_path(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
) -> list[torch.Tensor]:
|
||||
"""Dual-path greedy-packing execution.
|
||||
|
||||
Each image contributes both global tokens (constant per image)
|
||||
and local tokens (patches * patch_tokens). Greedy packing
|
||||
respects both budgets independently, then selects the smallest
|
||||
fitting budget per path with partial eager fallback.
|
||||
"""
|
||||
item_specs = self._get_item_specs(mm_kwargs)
|
||||
num_items = len(item_specs)
|
||||
|
||||
max_global_budget = self.global_token_budgets[-1]
|
||||
max_local_budget = self.local_token_budgets[-1]
|
||||
|
||||
per_item_global_tokens = [spec.global_output_tokens for spec in item_specs]
|
||||
per_item_local_tokens = [spec.local_output_tokens for spec in item_specs]
|
||||
per_item_total_tokens = [spec.output_tokens for spec in item_specs]
|
||||
|
||||
# Sort ascending by total output tokens
|
||||
sorted_indices = sorted(
|
||||
range(num_items), key=lambda i: per_item_total_tokens[i]
|
||||
)
|
||||
|
||||
# Each batch is a tuple of (indices, global_budget, local_budget).
|
||||
batches: list[tuple[list[int], int | None, int | None]] = []
|
||||
current_batch: list[int] = []
|
||||
current_global_tokens = 0
|
||||
current_local_tokens = 0
|
||||
|
||||
for orig_idx in sorted_indices:
|
||||
global_token = per_item_global_tokens[orig_idx]
|
||||
local_token = per_item_local_tokens[orig_idx]
|
||||
if (
|
||||
current_global_tokens + global_token <= max_global_budget
|
||||
and current_local_tokens + local_token <= max_local_budget
|
||||
and len(current_batch) < self.max_batch_size
|
||||
):
|
||||
current_batch.append(orig_idx)
|
||||
current_global_tokens += global_token
|
||||
current_local_tokens += local_token
|
||||
else:
|
||||
if current_batch:
|
||||
batches.append(
|
||||
(
|
||||
current_batch,
|
||||
self._find_smallest_fitting_budget_given_tokens(
|
||||
current_global_tokens, self.global_token_budgets
|
||||
),
|
||||
self._find_smallest_fitting_budget_given_tokens(
|
||||
current_local_tokens, self.local_token_budgets
|
||||
),
|
||||
)
|
||||
)
|
||||
current_batch = [orig_idx]
|
||||
current_global_tokens = global_token
|
||||
current_local_tokens = local_token
|
||||
|
||||
if current_batch:
|
||||
batches.append(
|
||||
(
|
||||
current_batch,
|
||||
self._find_smallest_fitting_budget_given_tokens(
|
||||
current_global_tokens, self.global_token_budgets
|
||||
),
|
||||
self._find_smallest_fitting_budget_given_tokens(
|
||||
current_local_tokens, self.local_token_budgets
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
outputs_by_orig_idx: dict[int, torch.Tensor] = {}
|
||||
|
||||
for batch_orig_indices, global_budget, local_budget in batches:
|
||||
batch_mm_kwargs = self.model.select_encoder_cudagraph_items(
|
||||
mm_kwargs, batch_orig_indices
|
||||
)
|
||||
batch_global_tokens = sum(
|
||||
per_item_global_tokens[i] for i in batch_orig_indices
|
||||
)
|
||||
batch_local_tokens = sum(
|
||||
per_item_local_tokens[i] for i in batch_orig_indices
|
||||
)
|
||||
|
||||
both_eager = global_budget is None and local_budget is None
|
||||
|
||||
if both_eager:
|
||||
logger.debug(
|
||||
"Encoder CUDA graph dual-path full eager fallback: "
|
||||
"%d global + %d local tokens from %d images",
|
||||
batch_global_tokens,
|
||||
batch_local_tokens,
|
||||
len(batch_orig_indices),
|
||||
)
|
||||
self.graph_misses += len(batch_orig_indices)
|
||||
with torch.inference_mode():
|
||||
raw = self.model.encoder_eager_forward(batch_mm_kwargs)
|
||||
per_item_total = [
|
||||
per_item_global_tokens[i] + per_item_local_tokens[i] + 1
|
||||
for i in batch_orig_indices
|
||||
]
|
||||
scatter_output_slices(
|
||||
raw, batch_orig_indices, per_item_total, outputs_by_orig_idx
|
||||
)
|
||||
continue
|
||||
|
||||
logger.debug(
|
||||
"Encoder CUDA graph dual-path: batch_size=%d, "
|
||||
"global=%d (budget=%s), local=%d (budget=%s)",
|
||||
len(batch_orig_indices),
|
||||
batch_global_tokens,
|
||||
global_budget,
|
||||
batch_local_tokens,
|
||||
local_budget,
|
||||
)
|
||||
|
||||
# Execute global path: graph or eager fallback
|
||||
if global_budget is not None:
|
||||
global_output = self._run_budget_graph(
|
||||
batch_mm_kwargs,
|
||||
global_budget,
|
||||
path="global",
|
||||
)
|
||||
assert global_output is not None
|
||||
else:
|
||||
with torch.inference_mode():
|
||||
global_output = self.model.encoder_eager_forward(
|
||||
batch_mm_kwargs, path="global"
|
||||
)
|
||||
|
||||
# Execute local path: graph or eager fallback
|
||||
if local_budget is not None and batch_local_tokens > 0:
|
||||
local_output = self._run_budget_graph(
|
||||
batch_mm_kwargs,
|
||||
local_budget,
|
||||
path="local",
|
||||
)
|
||||
assert local_output is not None
|
||||
elif batch_local_tokens > 0:
|
||||
with torch.inference_mode():
|
||||
local_output = self.model.encoder_eager_forward(
|
||||
batch_mm_kwargs, path="local"
|
||||
)
|
||||
else:
|
||||
local_output = None
|
||||
|
||||
self.model.postprocess_encoder_output(
|
||||
global_output,
|
||||
batch_orig_indices,
|
||||
per_item_global_tokens,
|
||||
outputs_by_orig_idx,
|
||||
clone=True,
|
||||
batch_mm_kwargs=batch_mm_kwargs,
|
||||
local_output=local_output,
|
||||
)
|
||||
|
||||
return [outputs_by_orig_idx[i] for i in range(num_items)]
|
||||
|
||||
def _dp_shard(
|
||||
self,
|
||||
mm_kwargs: dict[str, Any],
|
||||
@@ -626,10 +854,12 @@ class EncoderCudaGraphManager:
|
||||
total_requests = self.graph_hits + self.graph_misses
|
||||
hit_rate = self.graph_hits / total_requests if total_requests > 0 else 0.0
|
||||
|
||||
num_budgets = sum(len(g) for g in self.budget_graphs.values())
|
||||
|
||||
return {
|
||||
"graph_hits": self.graph_hits,
|
||||
"graph_misses": self.graph_misses,
|
||||
"hit_rate": hit_rate,
|
||||
"num_budgets": len(self.budget_graphs),
|
||||
"num_budgets": num_budgets,
|
||||
"token_budgets": self.token_budgets,
|
||||
}
|
||||
|
||||
@@ -26,6 +26,14 @@ class EncoderItemSpec:
|
||||
"""Number of output tokens after encoder processing (e.g. after
|
||||
spatial merge)."""
|
||||
|
||||
global_output_tokens: int = 0
|
||||
"""Number of output tokens from the global image path.
|
||||
Only used when ``EncoderCudaGraphConfig.enable_dual_path_graph`` is True."""
|
||||
|
||||
local_output_tokens: int = 0
|
||||
"""Number of output tokens from the local patch path.
|
||||
Only used when ``EncoderCudaGraphConfig.enable_dual_path_graph`` is True."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncoderCudaGraphConfig:
|
||||
@@ -60,6 +68,18 @@ class EncoderCudaGraphConfig:
|
||||
Only relevant when "video" is in ``modalities``.
|
||||
Image-only models can use the default of 1."""
|
||||
|
||||
enable_dual_path_graph: bool = False
|
||||
"""If True, the manager captures two independent graph sets
|
||||
(global + local) and runs dual-path graph selection during inference."""
|
||||
|
||||
global_token_per_image: int = 0
|
||||
"""Tokens per global image (e.g. 272 for DeepSeek-OCR).
|
||||
Only used when ``enable_dual_path_graph`` is True."""
|
||||
|
||||
local_token_per_patch: int = 0
|
||||
"""Tokens per local patch (e.g. 100 for DeepSeek-OCR).
|
||||
Only used when ``enable_dual_path_graph`` is True."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncoderCudaGraphCaptureInputs:
|
||||
|
||||
Reference in New Issue
Block a user