[V1] Do not detokenize if sampling param detokenize is False (#14224)

Signed-off-by: Himanshu Jaju <hj@mistral.ai>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
This commit is contained in:
Himanshu Jaju
2025-03-06 10:40:24 -08:00
committed by GitHub
co-authored by Nick Hill
parent 9f1710f1ac
commit cd579352bf
4 changed files with 69 additions and 27 deletions
+2
View File
@@ -68,6 +68,8 @@ class RequestState:
queue: Optional[asyncio.Queue[RequestOutput]],
log_stats: bool,
) -> "RequestState":
if not request.sampling_params.detokenize:
tokenizer = None
return cls(
request_id=request.request_id,
parent_req=parent_req,