MOVZXandHinata 18c06ed406 [Offloader] Stream multimodal towers from host RAM with bulk-copy ...

Add --offload-towers: place multimodal tower weights in pinned host
memory and bulk-copy each unit to the GPU only during its forward,
keeping towers out of VRAM between encodes. Unlike UVA zero-copy,
whole-tensor H2D transfers avoid re-reading operand tiles over PCIe
inside the GEMM inner loop. Tower streaming is independent of layer
group scheduling, so group_size may be 0 for tower-only offload.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>

2026-10-16 14:32:14 +07:00

..

test_prefetch.py

[Offloader] Stream multimodal towers from host RAM with bulk-copy

2026-10-16 14:32:14 +07:00