[Feature][Spec Decode] Support sampling mask replay for MRV2 MTP (#59359)
Signed-off-by: Cuiping Cheng <chengcuipingswu@163.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: Cuiping Cheng <chengcuipingswu@163.com> Co-authored-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Cuiping Cheng
aoshen02
Claude Opus 5.5
parent
37cecc8141
commit
70be6aae19
@@ -203,7 +203,10 @@ class Sampler:
|
||||
# Size by the validated top_k batch max; wider supports use the bitmask.
|
||||
max_num_kept = int(np.max(self.sampling_states.top_k.np[idx_mapping_np]))
|
||||
sampling_mask_tensors = SamplingMaskTensors.from_logits(
|
||||
processed_logits, num_sampled, max_num_kept
|
||||
processed_logits,
|
||||
input_batch.cu_num_logits,
|
||||
num_sampled,
|
||||
max_num_kept,
|
||||
)
|
||||
|
||||
# These are GPU tensors.
|
||||
|
||||
Reference in New Issue
Block a user