[Perf][DSv4.1] Run decoder replay layers in CUDA graphs and trim in PIECEWISE graphs (#59532)
Signed-off-by: Juntian Liu <Juntianl777@gmail.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent
21d93d0d8c
commit
eb2901b735
@@ -1077,6 +1077,12 @@ class GPUModelRunner(LoRAModelRunnerMixin):
|
||||
self.lora_config, self
|
||||
),
|
||||
)
|
||||
self.model_state.capture_inner_cudagraphs(
|
||||
input_buffers,
|
||||
self.block_tables,
|
||||
self.attn_groups,
|
||||
self.kv_cache_config,
|
||||
)
|
||||
if self.speculator is not None:
|
||||
with use_workspace_lane(self._draft_workspace_lane):
|
||||
self.speculator.capture()
|
||||
|
||||
Reference in New Issue
Block a user