[Bugfix]MooncakeConnect add query timeout and retry (#55923)

This commit is contained in:
Chao Lei
2026-10-09 09:19:33 +00:00
committed by GitHub
parent 18c19eaf23
commit bc713cf8cc
3 changed files with 29 additions and 6 deletions
+3 -2
View File
@@ -49,12 +49,13 @@ cache), see [ECMooncakeConnector Usage Guide](mooncake_ec_connector_usage.md).
- For headless instances, must be the same as the master instance
- Each instance needs a unique port on its host; using the same port number across different hosts is fine
- `VLLM_MOONCAKE_CONNECTOR_TIMEOUT`: Per-request timeout (in seconds) for MooncakeConnector HTTP calls to the bootstrap server, including prefiller worker registration at startup. (Optional)
- `VLLM_MOONCAKE_CONNECTOR_TIMEOUT`: Per-request timeout (in seconds) for MooncakeConnector HTTP calls to the bootstrap server, including prefiller worker registration at startup and metadata queries. (Optional)
- Default: 30.0
- Global rank 0 hosts the bootstrap server in the same process that mounts the Mooncake transfer engine's host segment, so a large `MOONCAKE_GLOBAL_SEGMENT_SIZE` can delay responses for several seconds during startup
- Raise this if bootstrap server timeout warnings appear on hosts with very large host-memory segments
- Bootstrap registration uses up to 3 attempts internally, with exponential backoff capped at 10 seconds; connection errors and timeouts are both retried
- Once the attempts are exhausted the caller raises instead of blocking, so failures surface rather than hanging
- Once registration attempts are exhausted the caller raises instead of blocking, so failures surface rather than hanging
- Metadata queries reuse the same timeout and 3-attempt limit, with a fixed 100ms delay between attempts. If all attempts fail, the error is logged and waiting queries are notified
- `WITH_NVIDIA_PEERMEM`: Selects how mooncake registers GPU memory for RDMA. Read by mooncake, not vLLM.
- Default: 1, which uses `ibv_reg_mr()` and requires the `nvidia-peermem` kernel module to be loaded