Allow AsyncLLMEngine.generate to target a specific DP rank (#19102)

Signed-off-by: Jon Swenson <jmswen@gmail.com>
This commit is contained in:
jmswen
2025-06-04 08:26:47 -07:00
committed by GitHub
parent 8f4ffbd373
commit c8dcc15921
10 changed files with 97 additions and 5 deletions
+1
View File
@@ -55,6 +55,7 @@ class EngineCoreRequest(
arrival_time: float
lora_request: Optional[LoRARequest]
cache_salt: Optional[str]
data_parallel_rank: Optional[int]
# Index of the client, used to ensure outputs are sent back to the same
# client for this request when scaling out the front-end.