[Perf][Model Runner V2] Compact sampling masks on GPU instead of unpacking the full-vocab bitmask on CPU (#54901)
Signed-off-by: aoshen02 <aoshen@inferact.ai> Signed-off-by: vx120 <893600387@qq.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: vx120 <893600387@qq.com>
This commit is contained in:
co-authored by
Claude Fable 5.1
vx120
parent
d6bce42983
commit
560ef78bfe
@@ -419,8 +419,9 @@ class RequestState:
|
||||
|
||||
sampling_mask = None
|
||||
if finished and self.sampling_mask_chunks:
|
||||
merged = SamplingMaskLists.merge(self.sampling_mask_chunks)
|
||||
sampling_mask = SamplingMask(merged.to_nested_list())
|
||||
sampling_mask = SamplingMask(
|
||||
[chunk.token_ids.tolist() for chunk in self.sampling_mask_chunks]
|
||||
)
|
||||
|
||||
# Concatenate routed experts on finish
|
||||
routed_experts = None
|
||||
|
||||
Reference in New Issue
Block a user