[Perf][DSv4.1] Run decoder replay layers in CUDA graphs and trim in PIECEWISE graphs (#59532)

Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
Juntian Liu
2026-10-05 22:11:45 -07:00
committed by GitHub
co-authored by Claude Opus 5.5 mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent 21d93d0d8c
commit eb2901b735
10 changed files with 445 additions and 71 deletions
+23
View File
@@ -570,6 +570,24 @@ class CompilationConfig:
If we limit the video count per prompt to `0`, it will also be set to `0`
(i.e., fall back to image-only mode)."""
cudagraph_decoder_replay: bool = True
"""Run the decoder replay layers of YOCO models (e.g. DeepSeek-V4.1) in CUDA
graphs of their own, and trim them in PIECEWISE graph steps. Requires
breakable PIECEWISE graphs; off under LoRA, prompt embeddings and non-first
PP ranks."""
decoder_replay_cudagraph_capture_sizes: list[int] = field(default_factory=list)
"""Decoder replay CUDA graph sizes for YOCO models (e.g. DeepSeek-V4.1), at
most max_num_batched_tokens. If empty: multiples of sliding_window up to
min(max_cudagraph_capture_size, max_num_seqs * sliding_window), coarser past
16 * sliding_window. Larger replay batches run eagerly."""
decoder_replay_trim_threshold: int = Field(default=768, ge=0)
"""For YOCO models (e.g. DeepSeek-V4.1), PIECEWISE graphs with at least this
many padded tokens trim the decoder replay batch; must exceed the replay
window (128), below which nothing trims. Independent of the replay graph
capture sizes; eager steps still trim."""
# Inductor capture
compile_sizes: list[int | str] | None = None
"""Sizes to compile for inductor. In addition
@@ -1033,6 +1051,11 @@ class CompilationConfig:
f"Invalid backend for piecewise compilation: {self.backend}"
)
if any(s <= 0 for s in self.decoder_replay_cudagraph_capture_sizes):
raise ValueError(
"All decoder_replay_cudagraph_capture_sizes must be positive"
)
# Validate encoder CUDA graph configuration
if (
self.cudagraph_mm_encoder