[Model Runner V2] Skip kernel launch for penalties & logit_bias (#32634)

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
This commit is contained in:
Woosuk Kwon
2026-01-19 22:20:19 -08:00
committed by GitHub
parent b75e85dede
commit e9c83cdc51
3 changed files with 35 additions and 5 deletions
+2 -2
View File
@@ -104,10 +104,10 @@ class Sampler:
logits = torch.empty_like(logits, dtype=torch.float32).copy_(logits)
# Apply logit bias (e.g., allowed_token_ids, min_tokens) in place.
self.logit_bias_state.apply_logit_bias(logits, idx_mapping, pos)
self.logit_bias_state.apply_logit_bias(logits, idx_mapping, idx_mapping_np, pos)
# Apply penalties in place.
self.penalties_state.apply_penalties(logits, idx_mapping)
self.penalties_state.apply_penalties(logits, idx_mapping, idx_mapping_np)
# Apply temperature in place.
apply_temperature(logits, idx_mapping, self.sampling_states.temperature.gpu)