[V1][Sampler] Faster top-k only implementation (#15478)

Signed-off-by: Nick Hill <nhill@redhat.com>
This commit is contained in:
Nick Hill
2025-03-26 10:56:47 -07:00
committed by GitHub
parent 733e7c9e95
commit 35fad35a48
3 changed files with 91 additions and 5 deletions
+6
View File
@@ -87,6 +87,12 @@ class Sampler(nn.Module):
logits: torch.Tensor,
sampling_metadata: SamplingMetadata,
) -> torch.Tensor:
"""Sample logits based on sampling metadata.
The various logits processing functions called in this method
may update the logits tensor in-place.
"""
assert not (sampling_metadata.all_greedy
and sampling_metadata.all_random)
if sampling_metadata.all_random: