[Model Runner V2] Use a different stream for grammar bitmask h2d copy (#33059)

Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Nick Hill <nhill@redhat.com>
This commit is contained in:
Woosuk Kwon
2026-01-27 16:37:43 -08:00
committed by GitHub
co-authored by Nick Hill
parent e82fa448c4
commit 46ec6d71c7
2 changed files with 32 additions and 11 deletions
+1
View File
@@ -168,6 +168,7 @@ class GPUModelRunner(LoRAModelRunnerMixin):
self.structured_outputs_worker = StructuredOutputsWorker(
max_num_logits=self.max_num_reqs * (self.num_speculative_steps + 1),
vocab_size=self.vocab_size,
device=self.device,
)
# LoRA-related workers.
self.lora_state = LoraState(max_num_reqs=self.max_num_reqs)