[Perf][DSv4.1] Run decoder replay layers in CUDA graphs and trim in PIECEWISE graphs (#59532)
Signed-off-by: Juntian Liu <Juntianl777@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent
21d93d0d8c
commit
eb2901b735
@@ -570,6 +570,24 @@ class CompilationConfig:
|
||||
If we limit the video count per prompt to `0`, it will also be set to `0`
|
||||
(i.e., fall back to image-only mode)."""
|
||||
|
||||
cudagraph_decoder_replay: bool = True
|
||||
"""Run the decoder replay layers of YOCO models (e.g. DeepSeek-V4.1) in CUDA
|
||||
graphs of their own, and trim them in PIECEWISE graph steps. Requires
|
||||
breakable PIECEWISE graphs; off under LoRA, prompt embeddings and non-first
|
||||
PP ranks."""
|
||||
|
||||
decoder_replay_cudagraph_capture_sizes: list[int] = field(default_factory=list)
|
||||
"""Decoder replay CUDA graph sizes for YOCO models (e.g. DeepSeek-V4.1), at
|
||||
most max_num_batched_tokens. If empty: multiples of sliding_window up to
|
||||
min(max_cudagraph_capture_size, max_num_seqs * sliding_window), coarser past
|
||||
16 * sliding_window. Larger replay batches run eagerly."""
|
||||
|
||||
decoder_replay_trim_threshold: int = Field(default=768, ge=0)
|
||||
"""For YOCO models (e.g. DeepSeek-V4.1), PIECEWISE graphs with at least this
|
||||
many padded tokens trim the decoder replay batch; must exceed the replay
|
||||
window (128), below which nothing trims. Independent of the replay graph
|
||||
capture sizes; eager steps still trim."""
|
||||
|
||||
# Inductor capture
|
||||
compile_sizes: list[int | str] | None = None
|
||||
"""Sizes to compile for inductor. In addition
|
||||
@@ -1033,6 +1051,11 @@ class CompilationConfig:
|
||||
f"Invalid backend for piecewise compilation: {self.backend}"
|
||||
)
|
||||
|
||||
if any(s <= 0 for s in self.decoder_replay_cudagraph_capture_sizes):
|
||||
raise ValueError(
|
||||
"All decoder_replay_cudagraph_capture_sizes must be positive"
|
||||
)
|
||||
|
||||
# Validate encoder CUDA graph configuration
|
||||
if (
|
||||
self.cudagraph_mm_encoder
|
||||
|
||||
Reference in New Issue
Block a user