[Feature][PCP] Support decode-only FULL CUDA graphs (#53867)
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com> Signed-off-by: QiuChunshuo <chunshuoq@gmail.com> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
Codex
Lucas Wilkinson
mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent
ca67438c08
commit
676650397f
@@ -777,11 +777,12 @@ def prepare_inputs_to_capture(
|
||||
input_batch = InputBatch.make_dummy(
|
||||
num_reqs, num_tokens, input_buffers, max_query_len=max_query_len
|
||||
)
|
||||
input_block_tables = block_tables.get_dummy_block_tables(num_reqs)
|
||||
slot_mapping_provider: BlockTables | PCPManager = block_tables
|
||||
if pcp_manager is not None:
|
||||
slot_mapping_provider = pcp_manager
|
||||
slot_mappings = slot_mapping_provider.get_dummy_slot_mappings(num_tokens)
|
||||
input_batch = pcp_manager.prepare_inputs_to_capture(input_batch)
|
||||
|
||||
block_table_provider = pcp_manager or block_tables
|
||||
input_block_tables = block_table_provider.get_dummy_block_tables(num_reqs)
|
||||
slot_mappings = block_table_provider.get_dummy_slot_mappings(num_tokens)
|
||||
slot_mappings_by_layer = build_slot_mappings_by_layer(
|
||||
slot_mappings, kv_cache_config
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user