[Perf][Model Runner V2] Compact sampling masks on GPU instead of unpacking the full-vocab bitmask on CPU (#54901)
Signed-off-by: aoshen02 <aoshen@inferact.ai> Signed-off-by: vx120 <893600387@qq.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: vx120 <893600387@qq.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
vx120
parent
d6bce42983
commit
560ef78bfe
@@ -179,7 +179,7 @@ class AsyncOutput(AsyncModelRunnerOutput):
|
||||
|
||||
if self.sampling_mask_tensors is not None:
|
||||
self.model_runner_output.sampling_masks = (
|
||||
self.sampling_mask_tensors.tolists(self.num_sampled_tokens_np)
|
||||
self.sampling_mask_tensors.tolists()
|
||||
)
|
||||
|
||||
if self.num_nans is not None:
|
||||
|
||||
Reference in New Issue
Block a user