[V1] Support bad_words in sampler (#13376)

Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
This commit is contained in:
22quinn
2025-03-08 14:50:26 -08:00
committed by GitHub
co-authored by Nick Hill
parent 9513290032
commit eb8b5eb183
13 changed files with 266 additions and 28 deletions
+1
View File
@@ -1268,6 +1268,7 @@ class GPUModelRunner(LoRAModelRunnerMixin):
min_tokens={},
logit_bias=[None for _ in range(num_reqs)],
allowed_token_ids_mask=None,
bad_words_token_ids={},
)
sampler_output = self.model.sample(logits=logits,
sampling_metadata=dummy_metadata)