[Perf] Support internal prefill checkpoints for Mamba prefix caching, 9%~25% TTFT improvement (#52789)
Signed-off-by: yewentao256 <zhyanwentao@126.com> Signed-off-by: Ziming Huang <zelda.huanghuang@gmail.com> Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com> Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com>
This commit is contained in:
co-authored by
Ziming Huang
OpenAI Codex
parent
020829f66b
commit
9eb9d9d395
@@ -10,7 +10,8 @@ STABLE_TORCH_LIBRARY(_flashkda_C, m) {
|
||||
"Tensor(a!) out, Tensor(c!) workspace, Tensor A_log, Tensor dt_bias, "
|
||||
"float lower_bound, "
|
||||
"Tensor? initial_state=None, Tensor(b!)? final_state=None, "
|
||||
"Tensor? cu_seqlens=None) -> ()");
|
||||
"Tensor? cu_seqlens=None, Tensor(d!)? checkpoint_state=None, "
|
||||
"Tensor? checkpoint_offsets=None) -> ()");
|
||||
}
|
||||
|
||||
STABLE_TORCH_LIBRARY_IMPL(_flashkda_C, CompositeExplicitAutograd, m) {
|
||||
|
||||
Reference in New Issue
Block a user