[Perf][Model Runner V2] Compact sampling masks on GPU instead of unpacking the full-vocab bitmask on CPU (#54901)

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: vx120 <893600387@qq.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: vx120 <893600387@qq.com>
This commit is contained in:
aoshen02
2026-09-03 19:03:36 -07:00
committed by GitHub
co-authored by Claude Fable 5.1 vx120
parent d6bce42983
commit 560ef78bfe
10 changed files with 188 additions and 108 deletions
+3 -2
View File
@@ -419,8 +419,9 @@ class RequestState:
sampling_mask = None
if finished and self.sampling_mask_chunks:
merged = SamplingMaskLists.merge(self.sampling_mask_chunks)
sampling_mask = SamplingMask(merged.to_nested_list())
sampling_mask = SamplingMask(
[chunk.token_ids.tolist() for chunk in self.sampling_mask_chunks]
)
# Concatenate routed experts on finish
routed_experts = None