[Core] [Frontend] Priority scheduling for embeddings and in the OpenAI-API (#8965)

This commit is contained in:
Sebastian Schoennenbeck
2024-10-01 09:58:06 +00:00
committed by GitHub
parent 1fe0a4264a
commit 35bd215168
8 changed files with 53 additions and 5 deletions
@@ -148,6 +148,7 @@ class OpenAIServingEmbedding(OpenAIServing):
pooling_params,
request_id_item,
lora_request=lora_request,
priority=request.priority,
)
generators.append(generator)