Support tensor parallel (#2)

This commit is contained in:
Zhuohan Li
2023-03-21 13:45:42 -07:00
committed by GitHub
parent cfae35b861
commit 2f49f15585
24 changed files with 2480 additions and 174 deletions
+1 -1
View File
@@ -112,7 +112,7 @@ class OPTCacheFlowAttention(nn.Module):
output[:num_prompt_tokens],
query[:num_prompt_tokens],
key[:num_prompt_tokens],
value[:num_prompt_tokens],
value[:num_prompt_tokens],
input_metadata.prompt_lens,
)