Document NixlConnector backend selection via kv_connector_extra_config (#33552)
Signed-off-by: KrxGu <krishom70@gmail.com>
This commit is contained in:
@@ -36,6 +36,35 @@ export UCX_NET_DEVICES=all # or specify network devices like "mlx5_0:1,mlx5_1:1
|
|||||||
!!! tip
|
!!! tip
|
||||||
When using UCX as the transport backend, NCCL environment variables (like `NCCL_IB_HCA`, `NCCL_SOCKET_IFNAME`) are not applicable to NixlConnector, so configure UCX-specific environment variables instead of NCCL variables.
|
When using UCX as the transport backend, NCCL environment variables (like `NCCL_IB_HCA`, `NCCL_SOCKET_IFNAME`) are not applicable to NixlConnector, so configure UCX-specific environment variables instead of NCCL variables.
|
||||||
|
|
||||||
|
#### Selecting a NIXL transport backend (plugin)
|
||||||
|
|
||||||
|
NixlConnector can use different NIXL transport backends (plugins). By default, NixlConnector uses UCX as the transport backend.
|
||||||
|
|
||||||
|
To select a different backend, set `kv_connector_extra_config.backends` in `--kv-transfer-config`.
|
||||||
|
|
||||||
|
### Example: using LIBFABRIC backend
|
||||||
|
|
||||||
|
```bash
|
||||||
|
vllm serve <MODEL> \
|
||||||
|
--kv-transfer-config '{
|
||||||
|
"kv_connector":"NixlConnector",
|
||||||
|
"kv_role":"kv_both",
|
||||||
|
"kv_connector_extra_config":{"backends":["LIBFABRIC"]}
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
You can also pass JSON keys individually using dotted arguments, and you can append list elements using `+`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
vllm serve <MODEL> \
|
||||||
|
--kv-transfer-config.kv_connector NixlConnector \
|
||||||
|
--kv-transfer-config.kv_role kv_both \
|
||||||
|
--kv-transfer-config.kv_connector_extra_config.backends+ LIBFABRIC
|
||||||
|
```
|
||||||
|
|
||||||
|
!!! note
|
||||||
|
Backend availability depends on how NIXL was built and what plugins are present in your environment. Refer to the [NIXL repository](https://github.com/ai-dynamo/nixl) for available backends and build instructions.
|
||||||
|
|
||||||
## Basic Usage (on the same host)
|
## Basic Usage (on the same host)
|
||||||
|
|
||||||
### Producer (Prefiller) Configuration
|
### Producer (Prefiller) Configuration
|
||||||
|
|||||||
Reference in New Issue
Block a user