[Bugfix] Sync block_size from EngineCore to frontend for hybrid Mamba… (#42967)
Signed-off-by: Amit Gruner <agruner@crusoe.ai> Co-authored-by: Amit Gruner <agruner@crusoe.ai> Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
This commit is contained in:
co-authored by
Amit Gruner
Jiangyun Zhu
parent
b623f7ea95
commit
654bd2bca4
@@ -713,6 +713,10 @@ class MPClient(EngineCoreClient):
|
||||
num_gpu_blocks += response.num_gpu_blocks
|
||||
vllm_config.cache_config.num_gpu_blocks = num_gpu_blocks
|
||||
|
||||
# Sync block_size: may be enlarged by _align_hybrid_block_size in the
|
||||
# worker for hybrid Mamba models.
|
||||
vllm_config.cache_config.block_size = response.block_size
|
||||
|
||||
# In external DP LB mode, the coordinator address that the
|
||||
# front-end procs connect to is obtained by each engine via it's
|
||||
# initial handshake with the rank 0 front-end.
|
||||
|
||||
Reference in New Issue
Block a user