[Attention] MLA with chunked prefill (#12639)

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Patrick Horn <patrick.horn@gmail.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
This commit is contained in:
Lucas Wilkinson
2025-02-21 15:30:12 -08:00
committed by GitHub
co-authored by Patrick Horn simon-mo Tyler Michael Smith
parent 900edbfa48
commit 288cc6c234
18 changed files with 1910 additions and 1275 deletions
+4
View File
@@ -565,6 +565,10 @@ def round_up(x: int, y: int) -> int:
return ((x + y - 1) // y) * y
def round_down(x: int, y: int) -> int:
return (x // y) * y
def _generate_random_fp8(
tensor: torch.Tensor,
low: float,