[BugFix][V1] Fix int32 token index overflow when preparing input ids (#16806)

This commit is contained in:
Yong Hoon Shin
2025-04-23 12:12:35 -07:00
committed by GitHub
parent 3cde34a4a4
commit 32d4b669d0
2 changed files with 4 additions and 2 deletions
+2 -1
View File
@@ -241,10 +241,11 @@ class GPUModelRunner(LoRAModelRunnerMixin):
device=self.device)
# OPTIMIZATION: Cache the tensors rather than creating them every step.
# Keep in int64 to avoid overflow with long context
self.arange_np = np.arange(max(self.max_num_reqs + 1,
self.max_model_len,
self.max_num_tokens),
dtype=np.int32)
dtype=np.int64)
# NOTE(woosuk): These tensors are "stateless", i.e., they are literally
# a faster version of creating a new tensor every time. Thus, we should
# not make any assumptions about the values in these tensors.