[1/2][Model Runner V2] DBO support, eager mode (#50945)
Signed-off-by: specture724 <specture724@gmail.com> Signed-off-by: Anjie Hou <149605198+specture724@users.noreply.github.com> Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com> Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
This commit is contained in:
co-authored by
jiangkuaixue123
Lucas Wilkinson
parent
784cac7c42
commit
3f41d102c5
@@ -988,7 +988,9 @@ class DiffusionGemmaModelState(ModelState):
|
||||
attn_groups,
|
||||
kv_cache_config,
|
||||
for_capture=False,
|
||||
ubatch_idx: int = 0,
|
||||
) -> dict[str, Any]:
|
||||
assert ubatch_idx == 0, "DBO is not supported"
|
||||
if cudagraph_mode == CUDAGraphMode.FULL:
|
||||
num_reqs = input_batch.num_reqs_after_padding
|
||||
num_tokens = input_batch.num_tokens_after_padding
|
||||
|
||||
Reference in New Issue
Block a user