[Perf][Model Runner V2] Compact sampling masks on GPU instead of unpacking the full-vocab bitmask on CPU (#54901)

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: vx120 <893600387@qq.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: vx120 <893600387@qq.com>
This commit is contained in:
aoshen02
2026-09-03 19:03:36 -07:00
committed by GitHub
co-authored by Claude Fable 5.1 vx120
parent d6bce42983
commit 560ef78bfe
10 changed files with 188 additions and 108 deletions
+1 -1
View File
@@ -179,7 +179,7 @@ class AsyncOutput(AsyncModelRunnerOutput):
if self.sampling_mask_tensors is not None:
self.model_runner_output.sampling_masks = (
self.sampling_mask_tensors.tolists(self.num_sampled_tokens_np)
self.sampling_mask_tensors.tolists()
)
if self.num_nans is not None: