[Perf][Model Runner V2] Compact sampling masks on GPU instead of unpacking the full-vocab bitmask on CPU (#54901)
Signed-off-by: aoshen02 <aoshen@inferact.ai> Signed-off-by: vx120 <893600387@qq.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: vx120 <893600387@qq.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
vx120
parent
d6bce42983
commit
560ef78bfe
@@ -186,8 +186,10 @@ class Sampler:
|
||||
|
||||
sampling_mask_tensors = None
|
||||
if self.return_sampling_mask:
|
||||
# Size by the validated top_k batch max; wider supports use the bitmask.
|
||||
max_num_kept = int(np.max(self.sampling_states.top_k.np[idx_mapping_np]))
|
||||
sampling_mask_tensors = SamplingMaskTensors.from_logits(
|
||||
processed_logits, num_sampled
|
||||
processed_logits, num_sampled, max_num_kept
|
||||
)
|
||||
|
||||
# These are GPU tensors.
|
||||
|
||||
Reference in New Issue
Block a user