[Pooling] Improve BGE-M3 sync pooling throughput by up to 3.13% (#53464)

Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
This commit is contained in:
Taneem Ibrahim
2026-08-24 08:33:32 -04:00
committed by GitHub
parent 41ae917c12
commit 79bb395eea
10 changed files with 106 additions and 66 deletions
@@ -69,7 +69,7 @@ class AllPool(TokenPoolingMethod):
return hidden_states_lst
pooling_states = pooling_metadata.pooling_states
finished_mask = pooling_cursor.is_finished().tolist()
finished_mask = pooling_cursor.get_finished_mask()
# If chunked_prefill is enabled
# 1. first store the chunked hidden_states in pooling_states.hidden_states_cache