[Perf][Model Runner V2] Compact sampling masks on GPU instead of unpacking the full-vocab bitmask on CPU (#54901)

Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: vx120 <893600387@qq.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: vx120 <893600387@qq.com>
This commit is contained in:
aoshen02
2026-09-03 19:03:36 -07:00
committed by GitHub
co-authored by Claude Fable 5.1 vx120
parent d6bce42983
commit 560ef78bfe
10 changed files with 188 additions and 108 deletions
+3 -1
View File
@@ -186,8 +186,10 @@ class Sampler:
sampling_mask_tensors = None
if self.return_sampling_mask:
# Size by the validated top_k batch max; wider supports use the bitmask.
max_num_kept = int(np.max(self.sampling_states.top_k.np[idx_mapping_np]))
sampling_mask_tensors = SamplingMaskTensors.from_logits(
processed_logits, num_sampled
processed_logits, num_sampled, max_num_kept
)
# These are GPU tensors.