[KV Connector]: Support KV push from Prefill to Decode node using Nixl KV Connector (#35264)

Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
This commit is contained in:
snadampal
2026-06-12 10:38:41 +00:00
committed by GitHub
co-authored by Nicolò Lucchesi
parent a014dddbaa
commit 88ed636218
26 changed files with 6335 additions and 3370 deletions
+256
View File
@@ -0,0 +1,256 @@
# NIXL push-mode KV transfer
The default NIXL connector is **pull-based**: the decode (D) instance
reads KV blocks from the prefill (P) instance via `NIXL READ` after
prefill completes. `NixlPushConnector` adds a **push-based** alternative
in which P writes the KV blocks directly into D's pre-allocated memory
via `NIXL WRITE`.
This document describes the threading, queues, and scheduling
interactions specific to the push design. The pull-mode design is
unchanged; the push connector reuses the same handshake, NIXL agent
setup, and metadata path wherever possible.
## High-level flow
```mermaid
sequenceDiagram
autonumber
participant Client
participant Proxy
participant DSched as D Scheduler
participant DWorker as D Worker (main)
participant DWriter as D Writer
participant PWriter as P Writer
participant PWorker as P Worker (main)
participant PSched as P Scheduler
Client->>Proxy: POST /v1/completions
Proxy->>PSched: prefill leg (do_remote_decode=True, max_tokens=1)
Proxy->>DSched: decode leg (do_remote_prefill=True, P coordinates)
note over DSched,DWriter: D side - register blocks with P
DSched->>DSched: update_state_after_alloc, stash registration, arm watchdog
DSched->>DWorker: build_connector_meta -> meta.push_registrations
DWorker->>DWriter: enqueue (req_id, reg_data) on _reg_send_inbox
DWriter->>PWriter: NIXL send_notif PUSH_REG msgpack
note over PSched,PWriter: P side - prefill, stage finished blocks
PSched->>PSched: request_finished, stash blocks
PSched->>PWorker: build_connector_meta -> meta.push_finished_blocks
PWorker->>PWriter: enqueue (req_id, blocks) on _finished_blocks_inbox
note over PWriter: P writer matches and WRITEs
PWriter->>PWriter: get_new_notifs returns PUSH_REG, route via _handle_push_reg_notif
alt PUSH_REG and finished blocks both present
PWriter->>PWriter: pop matching pair, fire WRITE
else only one side present
PWriter->>PWriter: stash and wait, self-poll only when blocks unmatched
end
PWriter->>PWriter: ensure D handshake (one-time)
PWriter->>DWriter: NIXL WRITE direct to D GPU + completion notif
note over DWorker,DWriter: D side - completion accounting
DWriter-->>DWorker: forward HB and completion notifs via _pending_completion_notifs
DWorker->>DWorker: _get_new_notifs drains, HB extends lease, completion marks recv done
DWorker->>DSched: update_connector_output(finished_recving)
DSched->>DSched: clear watchdog deadline
note over PWorker,PWriter: P side - reclaim
PWorker->>PWorker: get_finished, drain _sending_transfers, queue eviction
PWriter->>PWriter: drain _evict_finished_inbox, drop stale state
PWorker->>PSched: update_connector_output(finished_sending)
PSched->>PSched: free lease
DWorker-->>Proxy: stream decode tokens
Proxy-->>Client: response
```
## Threads
``NixlPushConnectorWorker`` introduces a single dedicated background
thread per worker (i.e. per TP rank), named ``nixl-push-writer``.
Each owns the new push-specific NIXL operations on its rank:
* ``nixl_wrapper.get_new_notifs()`` — receive notifications.
* ``nixl_wrapper.send_notif(...)`` for the ``PUSH_REG:<msgpack>`` (D
side) and for the per-WRITE completion notif (P side).
* ``nixl_wrapper.make_prepped_xfer(...) / transfer(...)`` — submit the
WRITE itself.
Heartbeats continue to go out from the engine main thread via the
existing base-worker ``_send_heartbeats`` plumbing inside
``start_load_kv``.
### Wake model
The writer thread blocks on ``_push_writer_wake`` (a
``threading.Event``) when it has no work. Three callers set the
event:
1. **``start_load_kv``** (worker main thread, called once per engine
step with the scheduler's metadata) — sets the wake only when the
step actually hands the writer new work, i.e. when
``meta.push_registrations`` or ``meta.push_finished_blocks`` is
non-empty. This is the wake for new transfers.
2. **``get_finished``** (worker main thread, called once per engine
step to report completions) — always sets the wake. The writer is
the sole consumer of ``nixl_wrapper.get_new_notifs()`` for push,
so this gives it a chance to drain inbound notifs (heartbeats from
D, completion notifs after a WRITE, late-arriving ``PUSH_REG``)
even when there is no new metadata to act on.
3. **Handshake-completion callback** (background handshake executor
thread) — when a deferred D→P handshake finishes successfully, the
future's done-callback re-enqueues the registration onto
``_reg_send_inbox`` and sets the wake so the corresponding
``send_notif`` runs on the writer (we never call ``send_notif`` from
the executor thread). On this second pass ``_ensure_handshake``
returns ``None`` (the agent is now connected), so the writer sends
the ``PUSH_REG`` directly. If the handshake *failed*, the callback
fails the request instead of re-enqueuing, so there is no retry
loop.
In addition to event-driven wakes, the writer self-polls at
``_PUSH_WRITER_POLL_INTERVAL_MS = 1.0`` ms while there are P-side
finished blocks waiting for an unmatched ``PUSH_REG``.
When a request completes on P (lease expires or the WRITE finishes),
``get_finished`` enqueues the request id onto ``_evict_finished_inbox``,
which the writer drains to drop stale ``_push_finished_blocks`` /
``_pending_d_registrations`` and stop self-polling.
## Writer-local matching tables
| Table | Owner | Holds |
|--------------------------------|------------------|------------------------------------------------------------------------|
| `_pending_d_registrations` | writer | D registrations received from a remote D, waiting for P's blocks |
| `_push_finished_blocks` | writer | P blocks staged by the scheduler, waiting for a remote D registration |
Either side can arrive first. The writer matches in both directions:
when a ``PUSH_REG`` arrives we look up ``_push_finished_blocks``, and
when finished blocks arrive we look up ``_pending_d_registrations``.
Both lookups try an exact ``request_id`` match first, then fall back
to comparing the ids after stripping the trailing per-engine random
suffix (via ``get_base_request_id``). The fallback exists because the
proxy hands the same ``X-Request-Id`` to both legs, so P and D wrap it
into the same ``cmpl-<uuid>-<index>`` form and differ only by the
8-hex randomization suffix that ``input_processor.assign_request_id``
appends per engine. Stripping just that suffix normalizes both sides
to the same id while preserving the completion index (so multi-prompt
sub-requests stay distinct). It also works whether or not
``VLLM_DISABLE_REQUEST_ID_RANDOMIZATION`` is set, which matters since
that env var is slated for removal upstream.
## Wire format
A push registration is sent as a NIXL notification:
```text
PUSH_REG:<msgpack-encoded dict>
```
Fields in the dict:
| Field | Set by | Meaning |
|----------------------|--------|------------------------------------------------------------------------|
| ``request_id`` | D | D's own vLLM request id; P's match key, echoed in the completion notif |
| ``decode_engine_id`` | D | D's engine id (P uses this for the reverse handshake) |
| ``decode_host`` | D | D's NIXL side-channel host |
| ``decode_port`` | D | D's NIXL side-channel port |
| ``decode_tp_size`` | D | D's tensor-parallel size |
| ``local_block_ids`` | D | per-group lists of D's *logical* block ids (preallocated) |
| ``remote_engine_id`` | D | P's engine id (for the existing P-side handshake) |
| ``remote_host`` | D | P's NIXL side-channel host |
| ``remote_port`` | D | P's NIXL side-channel port |
| ``remote_tp_size`` | D | P's tensor-parallel size |
D ships **logical** block ids; P expands them to physical block ids at
WRITE-submission time using the ratio learned during the NIXL
handshake (`remote_physical_blocks_per_logical`). This matches the
pull-mode contract — schedulers ship logical ids, workers expand to
physical at submission.
The completion notif sent from P to D after a WRITE is the existing
`<request_id>:<tp_size>` format used in pull mode (here ``request_id``
is D's own request id, taken from the registration), so the D-side
accounting code is unchanged.
## Scheduler-side responsibilities
`NixlPushConnectorScheduler` extends the base scheduler with:
* **D side** — `update_state_after_alloc` stashes registration data in
`_push_pending_registrations` and arms a soft watchdog
(`_push_registration_deadlines`). `build_connector_meta` drains the
stash into `meta.push_registrations` and any expired entries are
dropped with a warning.
* **P side** — `request_finished` stashes block IDs in
`_finished_request_blocks` (for the lease and for
`has_pending_push_work`) and `_newly_finished_push_blocks` (for the
next worker step via `meta.push_finished_blocks`).
* **Both sides** — `has_pending_push_work` keeps the engine main loop
stepping while there is in-flight push state, so the writer always
gets at least one wake per step.
`update_connector_output`:
* `finished_sending` (P side) clears the lease entry.
* `finished_recving` (D side) clears the watchdog deadline.
## Timeouts and watchdogs
Two per-request timers are armed on the scheduler:
* **D-side registration watchdog** — ``_push_registration_deadlines``.
If a registered request does not see a push completion within
``push_registration_timeout`` seconds (defaults to
``decoder_kv_blocks_ttl``), ``build_connector_meta`` drops the stale
registration and the pending entry, logs a warning, and stops trying
to resend the registration. The corresponding request remains tracked
in ``_reqs_need_recv``; it is the engine's request-level abort path
(or the user / proxy timing out the HTTP call) that ultimately fails
the request.
* **P-side block lease** — same ``_kv_lease_duration`` used by pull
mode. ``request_finished`` sets the expiration in ``_reqs_need_send``
and ``update_connector_output(finished_sending=...)`` clears it on
successful WRITE. Stale leases are reaped by ``get_finished`` in the
base worker, which then enqueues the eviction onto
``_evict_finished_inbox`` so the writer also stops self-polling.
## Failure handling
* **D-side handshake failure (P→D handshake before sending PUSH_REG)** —
the future's done-callback calls ``_handle_failed_transfer(rid, None)``,
which marks D's pre-allocated blocks invalid and enqueues onto
``_failed_recv_reqs`` so the next ``get_finished`` reports the
request as a failed recv. Same recv-side accounting as pull mode.
* **D-side ``send_notif`` failure when shipping the PUSH_REG to P** —
identical handling: ``_handle_failed_transfer`` marks the recv as
failed.
* **P-side WRITE submission failure** — the WRITE handle (if any) is
released and ``xfer_stats.record_failed_transfer()`` bumps the
failure counter. We deliberately do not call
``_handle_failed_transfer`` here: ``req_id`` on the P side has no
entry in ``_recving_metadata`` (P is not the receiver), so the
helper would put a P-local request id into ``_failed_recv_reqs``
and trip the assertion in the base worker's ``get_finished``. The
outbound WRITE is dropped on the floor; D's lease watchdog handles
the missing completion.
## Summary
The push design is a small, well-contained extension on top of the
existing NIXL connector:
* one new connector class, one new scheduler class, one new worker
class — all subclasses of the existing base classes;
* one dedicated background thread per worker;
* a few cross-thread queues, each with a single consumer (the writer);
most have one producer, except ``_reg_send_inbox``, which is fed both
by the engine main thread (new registrations) and by the
handshake-completion callback (registrations replayed after their
D→P handshake finishes);
* one new notification type (`PUSH_REG:<msgpack>`).
Behavior on the engine main thread is otherwise unchanged. The writer
thread is event-driven and idle when there is no push work.
@@ -0,0 +1,429 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""
Push-mode disaggregated prefilling proxy demo.
Companion to ``disagg_proxy_demo.py`` (pull mode). The client-facing API is
the same; the difference is in how P and D coordinate the KV transfer:
* Pull mode: proxy forwards P's ``kv_transfer_params`` (including
``remote_block_ids``) to D, and D pulls KV from P via NIXL READ.
* Push mode: proxy hands D **only** P's coordinates
(``remote_engine_id``, ``remote_host``, ``remote_port``, ``tp_size``)
and the shared ``remote_request_id``. D registers its locally allocated
blocks with P over a NIXL notification; P then pushes the KV to D via
NIXL WRITE.
Launch multiple vLLM instances configured with ``NixlPushConnector`` and
matching ``engine_id`` / ``side_channel_port``, then start this proxy:
python3 examples/disaggregated/disaggregated_serving/\
disagg_proxy_pushconnector_demo.py \
--model $model_name \
--prefill localhost:8100 \
--decode localhost:8200 \
--prefill-engine-id prefill-engine-001 \
--prefill-kv-host 10.0.0.1 \
--prefill-side-channel-port 5600 \
--prefill-tp-size 1 \
--port 8000
"""
import argparse
import contextlib
import ipaddress
import itertools
import json
import logging
import os
import sys
import uuid
from abc import ABC, abstractmethod
from collections.abc import Callable
import aiohttp
import uvicorn
from fastapi import APIRouter, Depends, FastAPI, HTTPException, Request
from fastapi.responses import JSONResponse, StreamingResponse
AIOHTTP_TIMEOUT = aiohttp.ClientTimeout(total=6 * 60 * 60)
logger = logging.getLogger()
logging.basicConfig(level=logging.INFO)
class SchedulingPolicy(ABC):
@abstractmethod
def schedule(self, cycler: itertools.cycle):
raise NotImplementedError("Scheduling Proxy is not set.")
class RoundRobinSchedulingPolicy(SchedulingPolicy):
def schedule(self, cycler: itertools.cycle) -> str:
return next(cycler)
class PushProxy:
"""Push-mode proxy.
The structure mirrors the pull-mode ``Proxy`` in
``disagg_proxy_demo.py``: an APIRouter with ``/v1/completions``,
``/v1/chat/completions``, ``/status`` and ``/instances/add``, plus
round-robin scheduling across multiple P / D instances.
Push-specific differences are confined to the request-handling
methods (``create_completion`` / ``create_chat_completion``):
* D's ``kv_transfer_params`` is built from CLI-provided P
coordinates instead of being derived from P's response.
* P and D requests are issued concurrently — D registers blocks and
waits while P prefills and pushes.
"""
def __init__(
self,
prefill_instances: list[str],
decode_instances: list[str],
model: str,
scheduling_policy: SchedulingPolicy,
prefill_engine_id: str,
prefill_kv_host: str,
prefill_side_channel_port: int,
prefill_tp_size: int,
custom_create_completion: Callable[[Request], StreamingResponse] | None = None,
custom_create_chat_completion: Callable[[Request], StreamingResponse]
| None = None,
):
self.prefill_instances = prefill_instances
self.decode_instances = decode_instances
self.prefill_cycler = itertools.cycle(prefill_instances)
self.decode_cycler = itertools.cycle(decode_instances)
self.model = model
self.scheduling_policy = scheduling_policy
# Push-mode metadata: D needs P's coordinates up-front. Pull mode
# learns these from P's response; push mode uses CLI args because
# D issues its registration before P responds.
self.push_metadata = {
"do_remote_decode": False,
"do_remote_prefill": True,
"remote_engine_id": prefill_engine_id,
"remote_host": prefill_kv_host,
"remote_port": prefill_side_channel_port,
"tp_size": prefill_tp_size,
}
self.custom_create_completion = custom_create_completion
self.custom_create_chat_completion = custom_create_chat_completion
self.router = APIRouter()
self.setup_routes()
# ── routes ──────────────────────────────────────────────────────── #
def setup_routes(self):
self.router.post(
"/v1/completions", dependencies=[Depends(self.validate_json_request)]
)(
self.custom_create_completion
if self.custom_create_completion
else self.create_completion
)
self.router.post(
"/v1/chat/completions", dependencies=[Depends(self.validate_json_request)]
)(
self.custom_create_chat_completion
if self.custom_create_chat_completion
else self.create_chat_completion
)
self.router.get("/status", response_class=JSONResponse)(self.get_status)
async def validate_json_request(self, raw_request: Request):
content_type = raw_request.headers.get("content-type", "").lower()
if content_type != "application/json":
raise HTTPException(
status_code=415,
detail="Unsupported Media Type: Only 'application/json' is allowed",
)
# ── HTTP forwarding ─────────────────────────────────────────────── #
async def forward_request(self, url, data, headers, use_chunked=True):
async with aiohttp.ClientSession(timeout=AIOHTTP_TIMEOUT) as session:
try:
async with session.post(
url=url, json=data, headers=headers
) as response:
if 200 <= response.status < 300 or 400 <= response.status < 500:
if use_chunked:
async for chunk_bytes in response.content.iter_chunked(
1024
):
yield chunk_bytes
else:
yield await response.read()
else:
error_content = await response.text()
with contextlib.suppress(json.JSONDecodeError):
error_content = json.loads(error_content)
logger.error(
"Request failed with status %s: %s",
response.status,
error_content,
)
raise HTTPException(
status_code=response.status,
detail=f"Request failed with status {response.status}: "
f"{error_content}",
)
except aiohttp.ClientError as e:
logger.error("ClientError occurred: %s", str(e))
raise HTTPException(
status_code=502,
detail="Bad Gateway: Error communicating with upstream server.",
) from e
except Exception as e:
logger.error("Unexpected error: %s", str(e))
raise HTTPException(status_code=500, detail=str(e)) from e
def schedule(self, cycler: itertools.cycle) -> str:
return self.scheduling_policy.schedule(cycler)
async def get_status(self):
return {
"mode": "push",
"prefill_node_count": len(self.prefill_instances),
"decode_node_count": len(self.decode_instances),
"prefill_nodes": self.prefill_instances,
"decode_nodes": self.decode_instances,
"prefill_engine_id": self.push_metadata["remote_engine_id"],
"prefill_kv_host": self.push_metadata["remote_host"],
"prefill_side_channel_port": self.push_metadata["remote_port"],
"prefill_tp_size": self.push_metadata["tp_size"],
}
# ── push-mode request handling ──────────────────────────────────── #
def _build_decode_kv_params(self, request_id: str) -> dict:
"""Push-mode kv_transfer_params for D.
``remote_block_ids`` is intentionally omitted: D allocates its
own blocks and registers them with P; P determines the
prefill-side block IDs and ships them via the WRITE.
"""
params = self.push_metadata.copy()
params["remote_request_id"] = request_id
return params
def _common_headers(self, request_id: str) -> dict:
h = {"X-Request-Id": request_id}
api_key = os.environ.get("OPENAI_API_KEY")
if api_key:
h["Authorization"] = f"Bearer {api_key}"
return h
async def _push_completion(self, raw_request: Request, path: str):
"""Shared body for /v1/completions and /v1/chat/completions.
Push mode fires P and D concurrently:
* P runs a normal prefill (max_tokens=1, do_remote_decode=True).
* D runs the decode (do_remote_prefill=True, no remote_block_ids).
D blocks waiting for P's WRITE; the response streamed back to the
client is the decode output from D.
"""
request = await raw_request.json()
request_id = str(uuid.uuid4())
# Prefill leg (max_tokens=1, signals P to keep KV around for D).
prefill_request = request.copy()
prefill_request["max_tokens"] = 1
if "max_completion_tokens" in prefill_request:
prefill_request["max_completion_tokens"] = 1
prefill_request["kv_transfer_params"] = {
"do_remote_decode": True,
"do_remote_prefill": False,
"remote_engine_id": None,
"remote_block_ids": None,
"remote_host": None,
"remote_port": None,
}
# Decode leg (push mode: no remote_block_ids).
decode_request = request.copy()
decode_request["kv_transfer_params"] = self._build_decode_kv_params(request_id)
prefill_instance = self.schedule(self.prefill_cycler)
decode_instance = self.schedule(self.decode_cycler)
headers = self._common_headers(request_id)
# Fire prefill; we don't read its body but must drain the
# connection so the upstream server can free its slot.
async for _ in self.forward_request(
f"http://{prefill_instance}{path}", prefill_request, headers
):
continue
generator = self.forward_request(
f"http://{decode_instance}{path}", decode_request, headers
)
return StreamingResponse(generator)
async def create_completion(self, raw_request: Request):
try:
return await self._push_completion(raw_request, "/v1/completions")
except HTTPException:
raise
except Exception:
exc_info = sys.exc_info()
print("Error occurred in disagg push proxy server")
print(exc_info)
raise
async def create_chat_completion(self, raw_request: Request):
try:
return await self._push_completion(raw_request, "/v1/chat/completions")
except HTTPException:
raise
except Exception:
exc_info = sys.exc_info()
error_messages = [str(e) for e in exc_info if e]
print("Error occurred in disagg push proxy server")
print(error_messages)
return StreamingResponse(
content=iter(error_messages), media_type="text/event-stream"
)
class PushProxyServer:
def __init__(
self,
args: argparse.Namespace,
scheduling_policy: SchedulingPolicy | None = None,
create_completion: Callable[[Request], StreamingResponse] | None = None,
create_chat_completion: Callable[[Request], StreamingResponse] | None = None,
):
self.validate_parsed_serve_args(args)
self.port = args.port
self.proxy_instance = PushProxy(
prefill_instances=[] if args.prefill is None else args.prefill,
decode_instances=[] if args.decode is None else args.decode,
model=args.model,
scheduling_policy=(
scheduling_policy
if scheduling_policy is not None
else RoundRobinSchedulingPolicy()
),
prefill_engine_id=args.prefill_engine_id,
prefill_kv_host=args.prefill_kv_host,
prefill_side_channel_port=args.prefill_side_channel_port,
prefill_tp_size=args.prefill_tp_size,
custom_create_completion=create_completion,
custom_create_chat_completion=create_chat_completion,
)
def validate_parsed_serve_args(self, args: argparse.Namespace):
if not args.prefill:
raise ValueError("Please specify at least one prefill node.")
if not args.decode:
raise ValueError("Please specify at least one decode node.")
if not args.prefill_engine_id:
raise ValueError(
"--prefill-engine-id is required in push mode (it must match "
"the engine_id passed to the prefill vLLM instance via "
"--kv-transfer-config)."
)
if not args.prefill_kv_host:
raise ValueError(
"--prefill-kv-host is required in push mode (the IP / host "
"that the prefill vLLM advertises on its NIXL side channel)."
)
self.validate_instances(args.prefill)
self.validate_instances(args.decode)
def validate_instances(self, instances: list):
for instance in instances:
if len(instance.split(":")) != 2:
raise ValueError(f"Invalid instance format: {instance}")
host, port = instance.split(":")
try:
if host != "localhost":
ipaddress.ip_address(host)
port = int(port)
if not (0 < port < 65536):
raise ValueError(f"Invalid port number in instance: {instance}")
except Exception as e:
raise ValueError(f"Invalid instance {instance}: {str(e)}") from e
def run_server(self):
app = FastAPI()
app.include_router(self.proxy_instance.router)
config = uvicorn.Config(app, port=self.port, loop="uvloop")
server = uvicorn.Server(config)
server.run()
def parse_args():
parser = argparse.ArgumentParser("vLLM disaggregated push-mode proxy server.")
parser.add_argument("--model", "-m", type=str, required=True, help="Model name")
parser.add_argument(
"--prefill",
"-p",
type=str,
nargs="+",
help="List of prefill node URLs (host:port)",
)
parser.add_argument(
"--decode",
"-d",
type=str,
nargs="+",
help="List of decode node URLs (host:port)",
)
parser.add_argument(
"--port",
type=int,
default=8000,
help="Server port number",
)
# Push-mode specific: P's coordinates that D needs in advance.
parser.add_argument(
"--prefill-engine-id",
type=str,
required=True,
help=(
"engine_id of the prefill vLLM instance (must match "
"--kv-transfer-config engine_id on the prefill server)"
),
)
parser.add_argument(
"--prefill-kv-host",
type=str,
required=True,
help=(
"IP / host the prefill vLLM advertises on its NIXL side "
"channel (VLLM_NIXL_SIDE_CHANNEL_HOST)"
),
)
parser.add_argument(
"--prefill-side-channel-port",
type=int,
default=5600,
help="NIXL side channel port on the prefill node "
"(VLLM_NIXL_SIDE_CHANNEL_PORT, default 5600)",
)
parser.add_argument(
"--prefill-tp-size",
type=int,
default=1,
help="Tensor parallel size of the prefill vLLM instance",
)
return parser.parse_args()
if __name__ == "__main__":
args = parse_args()
proxy_server = PushProxyServer(args=args)
proxy_server.run_server()
@@ -32,7 +32,7 @@ from unittest.mock import patch
import pytest
from vllm.distributed.kv_transfer.kv_connector.v1.base import KVConnectorRole
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.connector import (
from vllm.distributed.kv_transfer.kv_connector.v1.nixl import (
NixlConnector,
NixlConnectorMetadata,
)
@@ -436,7 +436,7 @@ def test_build_connector_meta_multiple_requests():
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_p_node_pull_kv_from_d(dist_init):
@@ -450,7 +450,7 @@ def test_p_node_pull_kv_from_d(dist_init):
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_p_node_pull_then_send_kv(dist_init):
@@ -472,7 +472,7 @@ def test_p_node_pull_then_send_kv(dist_init):
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_p_node_deferred_pull_on_no_handshake(dist_init):
@@ -366,7 +366,10 @@ def test_multi_example_connector_consistency():
def _ignore_event_collection(events: list[str]) -> list[str]:
return [event for event in events if event != "take_events"]
# Filter out per-step polling hooks that the scheduler calls repeatedly
# and which are not meaningful state transitions for these assertions.
ignored = {"take_events", "has_pending_push_work"}
return [event for event in events if event not in ignored]
def get_connector_events() -> dict[str, list[str]]:
@@ -1072,7 +1075,7 @@ def test_multi_connector_mixed_hma_disables_hybrid_kv_cache(monkeypatch):
)
with patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
):
llm = LLM(
@@ -344,7 +344,7 @@ def test_abort_immediately_remote_prefill_enqueues_empty_recv():
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_kv_transfer_handshake(dist_init):
@@ -560,7 +560,7 @@ class FakeNixlConnectorWorker(NixlConnectorWorker):
class TestNixlHandshake:
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_multi_xfer_one_engine(
@@ -643,7 +643,7 @@ class TestNixlHandshake:
connector.clear_connector_metadata()
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
@pytest.mark.parametrize(
@@ -713,7 +713,7 @@ class TestNixlHandshake:
raise TimeoutError("Took too long to complete async handshake.")
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
@pytest.mark.parametrize("local_tp_size", [1, 2])
@@ -725,7 +725,7 @@ class TestNixlHandshake:
remote configurations.
"""
monkeypatch.setattr(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.get_tensor_model_parallel_world_size",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.get_tensor_model_parallel_world_size",
lambda: local_tp_size,
)
@@ -784,7 +784,7 @@ class TestNixlHandshake:
check_handshake(6)
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_prefill_tp_size_greater_than_decode_tp_size_mla(
@@ -887,7 +887,7 @@ class TestNixlHandshake:
assert req_id not in conn_p1.connector_worker._reqs_to_process
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_concurrent_load_kv(
@@ -952,7 +952,7 @@ class TestNixlHandshake:
raise TimeoutError("Took too long to complete async handshake.")
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_handshake_fails_on_kv_cache_layout_mismatch(
@@ -967,7 +967,7 @@ class TestNixlHandshake:
# Mock TP world size to 2 to force heterogeneous TP when
# remote_tp_size=1
with patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.get_tensor_model_parallel_world_size", # noqa: E501
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.get_tensor_model_parallel_world_size", # noqa: E501
return_value=2,
):
# Initialize connector and worker (with fake NIXL wrapper)
@@ -1007,7 +1007,7 @@ class TestNixlHandshake:
worker.add_remote_agent(meta, remote_tp_size=1)
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_handshake_succeed_on_kv_cache_layout_mismatch_with_experimental(
@@ -1022,7 +1022,7 @@ class TestNixlHandshake:
# Mock TP world size to 2 to force heterogeneous TP when
# remote_tp_size=1
with patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.get_tensor_model_parallel_world_size", # noqa: E501
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.get_tensor_model_parallel_world_size", # noqa: E501
return_value=2,
):
# Initialize connector and worker (with fake NIXL wrapper)
@@ -1064,7 +1064,7 @@ class TestNixlHandshake:
worker.add_remote_agent(meta, remote_tp_size=1)
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_handshake_mixed_fa_mla_hetero_tp(self, default_vllm_config, dist_init):
@@ -1074,7 +1074,7 @@ class TestNixlHandshake:
"""
vllm_config = create_vllm_config()
with patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.get_tensor_model_parallel_world_size", # noqa: E501
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.get_tensor_model_parallel_world_size", # noqa: E501
return_value=2,
):
connector = NixlConnector(
@@ -1149,7 +1149,7 @@ class TestNixlHandshake:
# we put here is important. First run ray, it will clean up the resources, then
# the rest of the tests.
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_kv_connector_stats(default_vllm_config, dist_init):
@@ -1363,7 +1363,7 @@ def test_multi_kv_connector_stats_aggregation():
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_scheduler_kv_connector_stats_aggregation():
@@ -1428,7 +1428,7 @@ def test_scheduler_kv_connector_stats_aggregation():
@pytest.mark.parametrize("distributed_executor_backend", ["ray", None])
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_abort_timeout_on_prefiller(monkeypatch, distributed_executor_backend):
@@ -1615,7 +1615,7 @@ def test_register_kv_caches(
backend_cls = TritonAttentionBackend
nixl_worker = "vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker"
nixl_worker = "vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker"
nixl_connector = "vllm.distributed.kv_transfer.kv_connector.v1.nixl.connector"
with (
patch(f"{nixl_worker}.NixlWrapper") as mock_nixl_wrapper,
@@ -1865,15 +1865,17 @@ def test_kv_buffer_to_nixl_memory_types(
_NIXL_SUPPORTED_DEVICE.update(FakePlatform.get_nixl_supported_devices())
with (
patch("vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper"),
patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.threading.Event"
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper"
),
patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.threading.Thread"
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.threading.Event"
),
patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.current_platform",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.threading.Thread"
),
patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.current_platform",
FakePlatform,
),
patch(
@@ -1892,7 +1894,7 @@ def test_kv_buffer_to_nixl_memory_types(
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_shutdown_cleans_up_resources(default_vllm_config, dist_init):
@@ -1991,7 +1993,7 @@ def _setup_worker_with_remote_engine(
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_engine_ttl_eviction(default_vllm_config, dist_init):
@@ -2026,7 +2028,7 @@ def test_engine_ttl_eviction(default_vllm_config, dist_init):
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_engine_ttl_disabled(default_vllm_config, dist_init):
@@ -2074,7 +2076,7 @@ def test_transfer_topology_unregister():
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_aborted_request_removed_from_worker_in_batch(default_vllm_config, dist_init):
@@ -2194,7 +2196,7 @@ class FailingNixlWrapper(FakeNixlWrapper):
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FailingNixlWrapper,
)
@pytest.mark.parametrize(
@@ -2284,10 +2286,13 @@ def test_transfer_failure_logging(
slot_mapping={},
)
# Capture logs from the nixl.worker logger specifically
# Capture logs from the nixl connector loggers
# vLLM loggers have propagate=False, so we need to capture directly
nixl_logger = logging.getLogger(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker"
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker"
)
pull_logger = logging.getLogger(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.pull_worker"
)
captured_logs: list[logging.LogRecord] = []
@@ -2298,6 +2303,7 @@ def test_transfer_failure_logging(
handler = LogCapture()
handler.setLevel(logging.ERROR)
nixl_logger.addHandler(handler)
pull_logger.addHandler(handler)
try:
connector.start_load_kv(dummy_ctx)
@@ -2313,6 +2319,7 @@ def test_transfer_failure_logging(
connector.get_finished(finished_req_ids=set())
finally:
nixl_logger.removeHandler(handler)
pull_logger.removeHandler(handler)
# Print logs for manual comparison between commits
error_logs = [r for r in captured_logs if r.levelno >= logging.ERROR]
@@ -2349,7 +2356,7 @@ def test_transfer_failure_logging(
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FailingNixlWrapper,
)
def test_handshake_failure_returns_finished(default_vllm_config, dist_init):
@@ -2400,7 +2407,7 @@ def test_handshake_failure_returns_finished(default_vllm_config, dist_init):
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FailingNixlWrapper,
)
def test_transfer_setup_failure_returns_finished(default_vllm_config, dist_init):
@@ -2454,7 +2461,7 @@ def test_transfer_setup_failure_returns_finished(default_vllm_config, dist_init)
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FailingNixlWrapper,
)
@pytest.mark.parametrize(
@@ -2597,7 +2604,7 @@ def test_failed_request_skips_kv_postprocessing(
],
)
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_compatibility_hash_validation(
@@ -2706,7 +2713,7 @@ def test_compatibility_hash_validation(
# Patch zmq_ctx to return our mock socket
with (
patch.object(decode_worker, "add_remote_agent", return_value="fake_agent"),
patch.object(nixl.worker, "zmq_ctx") as mock_zmq_ctx,
patch.object(nixl.base_worker, "zmq_ctx") as mock_zmq_ctx,
):
mock_zmq_ctx.return_value.__enter__.return_value = mock_socket
@@ -2740,7 +2747,7 @@ def test_compatibility_hash_validation(
],
)
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_handshake_decode_errors(default_vllm_config, dist_init, error_scenario):
@@ -2806,7 +2813,7 @@ def test_handshake_decode_errors(default_vllm_config, dist_init, error_scenario)
mock_socket.recv.return_value = msg_bytes
with (
patch.object(decode_worker, "add_remote_agent", return_value="fake_agent"),
patch.object(nixl.worker, "zmq_ctx") as mock_zmq_ctx,
patch.object(nixl.base_worker, "zmq_ctx") as mock_zmq_ctx,
):
mock_zmq_ctx.return_value.__enter__.return_value = mock_socket
@@ -2819,7 +2826,7 @@ def test_handshake_decode_errors(default_vllm_config, dist_init, error_scenario)
)
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper",
FakeNixlWrapper,
)
def test_mla_broadcast_notif_uses_remote_request_id(
@@ -34,7 +34,9 @@ from .utils import (
(False, [0]),
],
)
@patch("vllm.distributed.kv_transfer.kv_connector.v1.nixl.scheduler.current_platform")
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_scheduler.current_platform"
)
def test_sw_sizes(mock_platform, swa_enabled, expected_sw_sizes):
"""Test sw_sizes is correctly computed based on SWA enabled/disabled."""
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.scheduler import (
@@ -782,7 +784,9 @@ def test_mamba_n1_p_side_truncation():
],
ids=["fa_swa_mamba", "fa_swa_only", "fa_only"],
)
@patch("vllm.distributed.kv_transfer.kv_connector.v1.nixl.scheduler.current_platform")
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_scheduler.current_platform"
)
def test_has_mamba_init(
mock_platform,
swa_enabled,
@@ -0,0 +1,815 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Unit tests for NixlPushConnector (scheduler + worker).
These tests cover the end-to-end mechanics of the push design without
requiring a real NIXL agent or network:
* Scheduler stages D registrations on ``update_state_after_alloc`` and
P finished blocks on ``request_finished``.
* ``build_connector_meta`` drains them onto
``meta.push_registrations`` / ``meta.push_finished_blocks``.
* ``has_pending_push_work`` reports True/False over the lifecycle.
* ``update_connector_output`` clears state on ``finished_sending`` and
``finished_recving``.
* The worker matches D registrations against P finished blocks (both
scenario directions) and forwards non-PUSH_REG NIXL notifs to the main
thread's ``_get_new_notifs``.
* ``get_finished`` enqueues evictions for the writer.
"""
from __future__ import annotations
import logging
import queue
import threading
import time
from collections import defaultdict
from typing import Any
from unittest.mock import MagicMock, patch
import msgspec
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.metadata import (
PUSH_REG_NOTIF_PREFIX,
NixlConnectorMetadata,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.push_worker import (
NixlPushConnectorWorker,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.utils import (
get_base_request_id,
)
from vllm.v1.outputs import KVConnectorOutput
from .utils import make_nixl_push_scheduler
# ----------------------------------------------------------------- #
# Helpers / fakes #
# ----------------------------------------------------------------- #
def _make_request(
*,
request_id: str,
is_d_side: bool = True,
remote_engine_id: str = "prefill-engine",
remote_request_id: str | None = None,
remote_host: str = "10.0.0.1",
remote_port: int = 5601,
tp_size: int = 1,
finished: bool = True,
) -> MagicMock:
"""Build a minimal Request mock used by request_finished."""
from vllm.v1.request import RequestStatus
req = MagicMock()
req.request_id = request_id
req.num_computed_tokens = 64
if is_d_side:
# D-side request: do_remote_prefill=True -> prefill on a remote P.
params: dict[str, Any] = {
"do_remote_prefill": True,
"do_remote_decode": False,
"remote_engine_id": remote_engine_id,
"remote_request_id": remote_request_id or f"prefill-{request_id}",
"remote_host": remote_host,
"remote_port": remote_port,
"tp_size": tp_size,
}
else:
# P-side request: do_remote_decode=True (we are the prefiller).
params = {
"do_remote_prefill": False,
"do_remote_decode": True,
}
req.kv_transfer_params = params
req.status = (
RequestStatus.FINISHED_LENGTH_CAPPED if finished else RequestStatus.RUNNING
)
return req
class _BlocksMock:
"""Minimal stand-in for ``KVCacheBlocks`` used in update_state_after_alloc."""
def __init__(self, block_ids: tuple[list[int], ...]):
self._block_ids = block_ids
def get_unhashed_block_ids_all_groups(self) -> tuple[list[int], ...]:
return self._block_ids
def _stub_sw_clipping(scheduler) -> None:
"""Make ``get_sw_clipped_blocks`` a passthrough so tests don't need
the full sliding-window machinery."""
scheduler.get_sw_clipped_blocks = lambda block_ids: block_ids
# ----------------------------------------------------------------- #
# Scheduler-side tests #
# ----------------------------------------------------------------- #
class TestPushScheduler:
def test_d_side_update_state_after_alloc_stages_registration(self):
"""D scheduler stashes registration data + arms watchdog deadline."""
sched = make_nixl_push_scheduler()
_stub_sw_clipping(sched)
request = _make_request(request_id="req-d-1")
blocks = _BlocksMock(block_ids=([10, 11, 12],))
sched.update_state_after_alloc(request, blocks, num_external_tokens=48)
assert request.request_id in sched._push_pending_registrations
reg = sched._push_pending_registrations[request.request_id]
# ``request_id`` is D's own vLLM request id; plus our own (D) coords.
assert reg["request_id"] == request.request_id
assert reg["decode_engine_id"] == sched.engine_id
assert reg["decode_host"] == sched.side_channel_host
assert reg["decode_port"] == sched.side_channel_port
assert reg["local_block_ids"] == ([10, 11, 12],)
assert reg["remote_engine_id"] == "prefill-engine"
# Watchdog deadline set in the future.
deadline = sched._push_registration_deadlines[request.request_id]
assert deadline > time.perf_counter()
# do_remote_prefill flipped off so the request isn't reprocessed.
assert request.kv_transfer_params["do_remote_prefill"] is False
# Tracked as awaiting a recv.
assert request.request_id in sched._reqs_need_recv
def test_p_side_request_finished_stages_blocks(self):
"""P scheduler pushes blocks into both _finished_request_blocks (lease)
and _newly_finished_push_blocks (metadata for next step)."""
sched = make_nixl_push_scheduler()
_stub_sw_clipping(sched)
request = _make_request(request_id="req-p-1", is_d_side=False)
block_ids = ([20, 21, 22, 23],)
delay, ret_params = sched.request_finished(request, block_ids)
assert delay is True
assert ret_params is not None
assert ret_params["do_remote_prefill"] is True
assert ret_params["do_remote_decode"] is False
assert request.request_id in sched._finished_request_blocks
assert request.request_id in sched._newly_finished_push_blocks
assert request.request_id in sched._reqs_need_send # lease armed
def test_build_connector_meta_drains_both_sides(self):
"""meta.push_registrations and meta.push_finished_blocks are filled
from the staging dicts and the staging dicts are cleared."""
sched = make_nixl_push_scheduler()
_stub_sw_clipping(sched)
# Stage one D registration and one P finished entry.
d_req = _make_request(request_id="req-d-9")
sched.update_state_after_alloc(
d_req, _BlocksMock(([1, 2, 3],)), num_external_tokens=48
)
p_req = _make_request(request_id="req-p-9", is_d_side=False)
sched.request_finished(p_req, ([4, 5, 6],))
scheduler_output = MagicMock()
scheduler_output.scheduled_new_reqs = []
scheduler_output.scheduled_cached_reqs = MagicMock(
req_ids=[], resumed_req_ids=set()
)
# Patch parent build_connector_meta so we don't have to set up
# all the base scheduler plumbing.
with patch.object(
sched.__class__.__mro__[1],
"build_connector_meta",
return_value=NixlConnectorMetadata(),
):
meta = sched.build_connector_meta(scheduler_output)
assert isinstance(meta, NixlConnectorMetadata)
assert "req-d-9" in meta.push_registrations
assert "req-p-9" in meta.push_finished_blocks
# Staging dicts cleared.
assert sched._push_pending_registrations == {}
assert sched._newly_finished_push_blocks == {}
# Lease bookkeeping kept until the WRITE completes.
assert "req-p-9" in sched._finished_request_blocks
def test_has_pending_push_work_lifecycle(self):
sched = make_nixl_push_scheduler()
_stub_sw_clipping(sched)
assert sched.has_pending_push_work() is False
# P finished blocks waiting for WRITE completion.
p_req = _make_request(request_id="req-p-7", is_d_side=False)
sched.request_finished(p_req, ([0, 1],))
assert sched.has_pending_push_work() is True
# Drain via build_connector_meta - lease still pending until WRITE.
scheduler_output = MagicMock()
scheduler_output.scheduled_new_reqs = []
scheduler_output.scheduled_cached_reqs = MagicMock(
req_ids=[], resumed_req_ids=set()
)
with patch.object(
sched.__class__.__mro__[1],
"build_connector_meta",
return_value=NixlConnectorMetadata(),
):
sched.build_connector_meta(scheduler_output)
# Lease is pending until WRITE completes -> still True.
assert sched.has_pending_push_work() is True
# Simulate WRITE completion via update_connector_output.
sched.update_connector_output(
KVConnectorOutput(
finished_sending={"req-p-7"},
finished_recving=set(),
invalid_block_ids=set(),
)
)
assert sched.has_pending_push_work() is False
def test_update_connector_output_clears_lease_and_watchdog(self):
sched = make_nixl_push_scheduler()
_stub_sw_clipping(sched)
d_req = _make_request(request_id="req-d-x")
sched.update_state_after_alloc(
d_req, _BlocksMock(([1, 2],)), num_external_tokens=32
)
p_req = _make_request(request_id="req-p-x", is_d_side=False)
sched.request_finished(p_req, ([3, 4],))
sched.update_connector_output(
KVConnectorOutput(
finished_sending={"req-p-x"},
finished_recving={"req-d-x"},
invalid_block_ids=set(),
)
)
assert "req-p-x" not in sched._finished_request_blocks
assert "req-d-x" not in sched._push_registration_deadlines
def test_registration_watchdog_expires(self, caplog):
"""Stale D registrations whose deadline has passed are dropped at
``build_connector_meta`` time."""
# Watchdog logs a WARNING when it drops the stale entry; that's
# what this test is verifying, so silence it in the test report.
caplog.set_level(
logging.CRITICAL,
logger=("vllm.distributed.kv_transfer.kv_connector.v1.nixl.push_scheduler"),
)
sched = make_nixl_push_scheduler()
_stub_sw_clipping(sched)
d_req = _make_request(request_id="req-d-stale")
sched.update_state_after_alloc(
d_req, _BlocksMock(([7, 8],)), num_external_tokens=32
)
# Force the deadline into the past.
sched._push_registration_deadlines[d_req.request_id] = time.perf_counter() - 1.0
scheduler_output = MagicMock()
scheduler_output.scheduled_new_reqs = []
scheduler_output.scheduled_cached_reqs = MagicMock(
req_ids=[], resumed_req_ids=set()
)
with patch.object(
sched.__class__.__mro__[1],
"build_connector_meta",
return_value=NixlConnectorMetadata(),
):
meta = sched.build_connector_meta(scheduler_output)
assert d_req.request_id not in sched._push_registration_deadlines
assert d_req.request_id not in sched._push_pending_registrations
assert d_req.request_id not in meta.push_registrations
# ----------------------------------------------------------------- #
# Worker-side tests #
# ----------------------------------------------------------------- #
class _StubWriterWorker(NixlPushConnectorWorker):
"""Construct a worker without invoking ``__init__`` so we can drive
the matching/notif logic without bringing up NIXL or torch."""
@classmethod
def fresh(cls) -> _StubWriterWorker:
w = object.__new__(cls)
# Push-specific state managed by NixlPushConnectorWorker.
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.metadata import (
ReqId,
TransferHandle,
)
w._sending_transfers = defaultdict[ReqId, list[TransferHandle]](list)
w._sending_transfers_lock = threading.Lock()
w._push_finished_blocks = {}
w._pending_d_registrations = {}
w._reg_send_inbox = queue.Queue()
w._finished_blocks_inbox = queue.Queue()
w._pending_completion_notifs = queue.Queue()
w._evict_finished_inbox = queue.Queue()
w._push_writer_wake = threading.Event()
w._push_writer_stop = threading.Event()
w._push_writer_thread = None
# Base worker fields touched by start_load_kv / _get_new_notifs.
w._recving_metadata = {}
w._recving_transfers = defaultdict(list)
w._reqs_to_process = set()
w._reqs_to_send = {}
w.consumer_notification_counts_by_req = defaultdict(int)
w.tp_rank = 0
w.world_size = 1
w.engine_id = "test-decode-engine"
w._remote_agents = {}
# Track _do_start_push_kv invocations.
calls: list[tuple[str, Any, dict[str, Any]]] = []
w.start_push_calls = calls
return w
def _do_start_push_kv(
self,
request_id: str,
local_block_ids,
registration_data: dict[str, Any],
) -> None: # pragma: no cover - exercised through tests
# Track the call instead of issuing real WRITEs.
self.start_push_calls.append((request_id, local_block_ids, registration_data))
def _registration_data(
request_id: str,
*,
decode_engine_id: str = "decode-engine",
decode_host: str = "10.0.0.2",
decode_port: int = 5602,
decode_tp_size: int = 1,
local_block_ids=((100, 101, 102),),
remote_engine_id: str = "prefill-engine",
remote_host: str = "10.0.0.1",
remote_port: int = 5601,
remote_tp_size: int = 1,
) -> dict[str, Any]:
return {
"request_id": request_id,
"decode_engine_id": decode_engine_id,
"decode_host": decode_host,
"decode_port": decode_port,
"decode_tp_size": decode_tp_size,
"local_block_ids": local_block_ids,
"remote_engine_id": remote_engine_id,
"remote_host": remote_host,
"remote_port": remote_port,
"remote_tp_size": remote_tp_size,
}
class TestPushWriterMatching:
def test_handle_push_reg_matches_existing_finished_blocks(self):
"""PUSH_REG arrives second (P finished first): match + fire."""
w = _StubWriterWorker.fresh()
# P had already finished; its blocks were stashed via metadata.
w._push_finished_blocks["req-A"] = ([200, 201, 202],)
notif = PUSH_REG_NOTIF_PREFIX + msgspec.msgpack.encode(
_registration_data("req-A")
)
w._handle_push_reg_notif(notif)
assert len(w.start_push_calls) == 1
rid, blocks, reg = w.start_push_calls[0]
assert rid == "req-A"
assert blocks == ([200, 201, 202],)
assert reg["decode_engine_id"] == "decode-engine"
# Finished blocks consumed.
assert "req-A" not in w._push_finished_blocks
assert w._pending_d_registrations == {}
def test_handle_push_reg_stashes_when_no_finished_blocks_yet(self):
"""PUSH_REG arrives first (D registered first): stash, no fire."""
w = _StubWriterWorker.fresh()
notif = PUSH_REG_NOTIF_PREFIX + msgspec.msgpack.encode(
_registration_data("req-B")
)
w._handle_push_reg_notif(notif)
assert len(w.start_push_calls) == 0
assert "req-B" in w._pending_d_registrations
def test_handle_push_reg_matches_after_stripping_random_suffix(self):
"""P and D assign the same logical request the same
``cmpl-<uuid>-<index>`` but different per-engine random suffixes;
the writer should still match P's finished blocks via the
suffix-stripping fallback in ``_pop_matching_finished_blocks``.
"""
w = _StubWriterWorker.fresh()
# Same base id + completion index; differ only in the trailing
# ``-<8 hex>`` randomization suffix.
p_id = "cmpl-12345678-aaaa-bbbb-cccc-1234567890ab-0-aaaaaaaa"
d_id = "cmpl-12345678-aaaa-bbbb-cccc-1234567890ab-0-bbbbbbbb"
# Sanity: same base id under the helper used by the connector.
assert get_base_request_id(p_id) == get_base_request_id(d_id)
w._push_finished_blocks[p_id] = ([1, 2, 3],)
notif = PUSH_REG_NOTIF_PREFIX + msgspec.msgpack.encode(_registration_data(d_id))
w._handle_push_reg_notif(notif)
# Suffix-stripped fallback matched and fired.
assert len(w.start_push_calls) == 1
assert w.start_push_calls[0][0] == p_id
assert p_id not in w._push_finished_blocks
def test_handle_push_reg_drops_malformed(self, caplog):
# The writer logs WARNING/ERROR when it sees these bad payloads;
# that's the desired behavior, so suppress the noise from test
# output rather than letting it look like a failure.
caplog.set_level(
logging.CRITICAL,
logger=("vllm.distributed.kv_transfer.kv_connector.v1.nixl.push_worker"),
)
w = _StubWriterWorker.fresh()
# Missing request_id -> should drop without raising.
bad = PUSH_REG_NOTIF_PREFIX + msgspec.msgpack.encode({"decode_engine_id": "x"})
w._handle_push_reg_notif(bad)
assert w._pending_d_registrations == {}
assert w.start_push_calls == []
# Undecodable payload also dropped.
w._handle_push_reg_notif(PUSH_REG_NOTIF_PREFIX + b"\xff\xff\xff")
assert w.start_push_calls == []
class TestPushWriterStartLoadKv:
def test_finished_blocks_inbox_matches_stashed_registration(self):
"""Run the writer-loop's finished-blocks drain against a
pre-populated _pending_d_registrations entry."""
w = _StubWriterWorker.fresh()
w._pending_d_registrations["req-C"] = _registration_data("req-C")
# Simulate start_load_kv enqueuing finished blocks.
w._finished_blocks_inbox.put(("req-C", ([10, 11, 12],)))
# Drain like the writer loop does.
while True:
try:
rid, blocks = w._finished_blocks_inbox.get_nowait()
except queue.Empty:
break
matched = w._pop_matching_registration(rid)
if matched is not None:
w._do_start_push_kv(rid, blocks, matched)
else:
w._push_finished_blocks[rid] = blocks
assert len(w.start_push_calls) == 1
assert w.start_push_calls[0][0] == "req-C"
assert "req-C" not in w._pending_d_registrations
def test_start_load_kv_enqueues_to_writer(self):
"""``start_load_kv`` should hand registrations + finished blocks
to the writer queues without doing matching itself."""
w = _StubWriterWorker.fresh()
# Stub heartbeats to a no-op; tests don't exercise the heartbeat
# path here.
w._send_heartbeats = lambda metadata: None
# Stub logical-to-kernel mapping used by reqs_to_recv.
w._logical_to_kernel_block_ids = lambda x: x
meta = NixlConnectorMetadata()
meta.push_registrations = {
"req-D": _registration_data("req-D"),
}
meta.push_finished_blocks = {
"req-E": ([5, 6, 7],),
}
w.start_load_kv(meta)
# Things are queued for the writer; nothing fires yet.
assert w._reg_send_inbox.qsize() == 1
assert w._finished_blocks_inbox.qsize() == 1
assert w._push_writer_wake.is_set()
assert w.start_push_calls == []
class TestPushWriterNotifs:
def test_get_new_notifs_processes_forwarded_completion_notif(self):
"""Non-PUSH_REG notifs forwarded by the writer thread are drained
on the engine main thread inside ``_get_new_notifs``."""
w = _StubWriterWorker.fresh()
# Pretend the writer thread already forwarded a completion notif
# for a request whose KV is being received.
request_id = "req-recv-1"
w._recving_metadata[request_id] = MagicMock()
# Compose the standard completion notif: req_id:tp_size.
notif_msg = f"{request_id}:1".encode()
w._pending_completion_notifs.put(notif_msg)
# transfer_topo is consulted only for the producer-side path; we
# make it a MagicMock because the D-side branch returns early.
w.transfer_topo = MagicMock()
notified = w._get_new_notifs()
# Notif consumed; D-side just touches _recving_transfers.
assert notified == set()
assert request_id in w._recving_transfers
def test_get_finished_evicts_completed_state(self):
"""``get_finished`` should enqueue evictions and wake the writer."""
w = _StubWriterWorker.fresh()
# Stub the base ``get_finished`` to return one done_sending entry.
# Patch via the MRO's parent class.
with patch.object(
NixlPushConnectorWorker.__mro__[1],
"get_finished",
return_value=({"req-done"}, set()),
):
done_sending, done_recving = w.get_finished()
assert "req-done" in done_sending
assert done_recving == set()
# Eviction enqueued for the writer.
evicted = []
while True:
try:
evicted.append(w._evict_finished_inbox.get_nowait())
except queue.Empty:
break
assert evicted == ["req-done"]
assert w._push_writer_wake.is_set()
# ----------------------------------------------------------------- #
# Negative / error-path tests #
# ----------------------------------------------------------------- #
class TestPushSchedulerNegative:
"""Failure / no-op paths on the scheduler side."""
def test_update_state_after_alloc_no_kv_transfer_params_is_noop(self):
"""Requests without kv_transfer_params must not register anything."""
sched = make_nixl_push_scheduler()
_stub_sw_clipping(sched)
request = MagicMock()
request.request_id = "req-no-params"
request.kv_transfer_params = None
sched.update_state_after_alloc(
request, _BlocksMock(([1, 2, 3],)), num_external_tokens=64
)
assert sched._push_pending_registrations == {}
assert sched._push_registration_deadlines == {}
assert sched._reqs_need_recv == {}
def test_update_state_after_alloc_zero_external_tokens_does_not_register(self):
"""num_external_tokens=0 should not stage a D registration."""
sched = make_nixl_push_scheduler()
_stub_sw_clipping(sched)
request = _make_request(request_id="req-zero-ext")
sched.update_state_after_alloc(
request, _BlocksMock(([1, 2, 3],)), num_external_tokens=0
)
assert sched._push_pending_registrations == {}
assert sched._push_registration_deadlines == {}
def test_request_finished_unfinished_status_does_not_stage(self):
"""If a request is still RUNNING, request_finished must not stash
blocks for the worker (no push needed)."""
sched = make_nixl_push_scheduler()
_stub_sw_clipping(sched)
request = _make_request(
request_id="req-running", is_d_side=False, finished=False
)
delay, ret = sched.request_finished(request, ([1, 2, 3],))
assert delay is False
assert ret is None
assert sched._finished_request_blocks == {}
assert sched._newly_finished_push_blocks == {}
def test_request_finished_empty_blocks_does_not_arm_lease(self):
"""Empty block-id groups should still complete cleanly without
arming the lease/finished maps."""
sched = make_nixl_push_scheduler()
_stub_sw_clipping(sched)
request = _make_request(request_id="req-empty", is_d_side=False)
delay, ret = sched.request_finished(request, ((),))
assert delay is False
assert ret is not None
assert "req-empty" not in sched._finished_request_blocks
assert "req-empty" not in sched._newly_finished_push_blocks
assert "req-empty" not in sched._reqs_need_send
def test_update_connector_output_unknown_request_is_noop(self):
"""Idempotent cleanup: clearing a request that was never staged
must not raise or mutate other state."""
sched = make_nixl_push_scheduler()
_stub_sw_clipping(sched)
# Stage one real request to ensure it's NOT touched.
live = _make_request(request_id="req-live", is_d_side=False)
sched.request_finished(live, ([1],))
sched.update_connector_output(
KVConnectorOutput(
finished_sending={"unknown-1"},
finished_recving={"unknown-2"},
invalid_block_ids=set(),
)
)
# Live entry untouched.
assert "req-live" in sched._finished_request_blocks
class TestPushWriterNegative:
"""Failure / drop / idempotence paths in the writer thread."""
def test_pop_matching_registration_returns_none_when_empty(self):
w = _StubWriterWorker.fresh()
assert w._pop_matching_registration("nope") is None
def test_pop_matching_finished_blocks_returns_none_when_empty(self):
w = _StubWriterWorker.fresh()
assert w._pop_matching_finished_blocks("nope") is None
def test_pop_matching_registration_no_match_when_base_ids_differ(self):
"""A registration whose base id (after stripping the random suffix)
does NOT match the lookup request_id must not be popped."""
w = _StubWriterWorker.fresh()
# Two unrelated requests: different base UUIDs, so stripping the
# trailing ``-<8 hex>`` suffix still yields different base ids.
unrelated_d = "cmpl-aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa-0-11111111"
lookup = "cmpl-bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb-0-22222222"
assert get_base_request_id(unrelated_d) != get_base_request_id(lookup)
w._pending_d_registrations[unrelated_d] = _registration_data(unrelated_d)
result = w._pop_matching_registration(lookup)
assert result is None
# Original entry untouched.
assert unrelated_d in w._pending_d_registrations
def test_handle_push_reg_with_non_dict_payload_is_dropped(self, caplog):
"""msgpack-encoded non-dict payload (e.g. a list) should be
dropped without raising."""
caplog.set_level(
logging.CRITICAL,
logger=("vllm.distributed.kv_transfer.kv_connector.v1.nixl.push_worker"),
)
w = _StubWriterWorker.fresh()
bad = PUSH_REG_NOTIF_PREFIX + msgspec.msgpack.encode([1, 2, 3])
w._handle_push_reg_notif(bad)
assert w._pending_d_registrations == {}
assert w.start_push_calls == []
def test_handle_push_reg_with_non_string_request_id_is_dropped(self, caplog):
"""request_id must be a str; integers, None, etc. must drop."""
caplog.set_level(
logging.CRITICAL,
logger=("vllm.distributed.kv_transfer.kv_connector.v1.nixl.push_worker"),
)
w = _StubWriterWorker.fresh()
for bogus_rid in (123, None, 4.5, b"bytes-not-str"):
payload = _registration_data("placeholder")
payload["request_id"] = bogus_rid # type: ignore[assignment]
notif = PUSH_REG_NOTIF_PREFIX + msgspec.msgpack.encode(payload)
w._handle_push_reg_notif(notif)
assert w._pending_d_registrations == {}
assert w.start_push_calls == []
def test_handle_push_reg_idempotent_for_same_request_id(self):
"""Receiving the same PUSH_REG twice (e.g. P retries after a
flake) keeps the entry staged exactly once and never fires."""
w = _StubWriterWorker.fresh()
notif = PUSH_REG_NOTIF_PREFIX + msgspec.msgpack.encode(
_registration_data("req-dup")
)
w._handle_push_reg_notif(notif)
w._handle_push_reg_notif(notif)
assert "req-dup" in w._pending_d_registrations
assert len(w._pending_d_registrations) == 1
assert w.start_push_calls == []
def test_get_finished_enqueues_eviction_for_each_done_request(self):
"""``get_finished`` must enqueue an eviction for every request
in ``done_sending`` so the writer can drop stale matching state.
Unlike the happy-path test, this verifies the *cardinality*: N
completed requests -> N evictions, in order."""
w = _StubWriterWorker.fresh()
with patch.object(
NixlPushConnectorWorker.__mro__[1],
"get_finished",
return_value=({"req-1", "req-2", "req-3"}, set()),
):
done_sending, _ = w.get_finished()
assert done_sending == {"req-1", "req-2", "req-3"}
evicted: list[str] = []
while True:
try:
evicted.append(w._evict_finished_inbox.get_nowait())
except queue.Empty:
break
assert sorted(evicted) == ["req-1", "req-2", "req-3"]
def test_get_finished_with_no_completions_does_not_enqueue_eviction(self):
"""If there's nothing newly done, no eviction should be enqueued.
The wake event IS still set because ``get_finished`` always wakes
the writer to drain notifs."""
w = _StubWriterWorker.fresh()
with patch.object(
NixlPushConnectorWorker.__mro__[1],
"get_finished",
return_value=(set(), set()),
):
done_sending, done_recving = w.get_finished()
assert done_sending == set()
assert done_recving == set()
assert w._evict_finished_inbox.qsize() == 0
# Wake set so the writer drains NIXL notifs even when idle.
assert w._push_writer_wake.is_set()
def test_get_new_notifs_unknown_request_is_logged_and_skipped(self, caplog):
"""A completion notif for a request the worker doesn't know
about should be logged but not crash."""
caplog.set_level(
logging.CRITICAL,
logger=("vllm.distributed.kv_transfer.kv_connector.v1.nixl.push_worker"),
)
w = _StubWriterWorker.fresh()
w.transfer_topo = MagicMock()
# Forward a completion notif for an unknown request_id.
w._pending_completion_notifs.put(b"never-heard-of-you:1")
notified = w._get_new_notifs()
assert notified == set()
# Did not register anywhere.
assert "never-heard-of-you" not in w._recving_transfers
def test_start_load_kv_with_empty_metadata_is_noop(self):
"""Empty metadata must not wake the writer or enqueue anything."""
w = _StubWriterWorker.fresh()
w._send_heartbeats = lambda metadata: None
w._logical_to_kernel_block_ids = lambda x: x
meta = NixlConnectorMetadata()
w.start_load_kv(meta)
assert w._reg_send_inbox.qsize() == 0
assert w._finished_blocks_inbox.qsize() == 0
# Wake should NOT be set if there was nothing to push.
assert not w._push_writer_wake.is_set()
def test_get_new_notifs_extends_lease_on_heartbeat(self):
"""``HB:`` notifs forwarded by the writer thread must extend the
leases of tracked P-side requests on the engine main thread, and
ignore request IDs that aren't being tracked."""
w = _StubWriterWorker.fresh()
w.transfer_topo = MagicMock()
# _handle_heartbeat reads ``self._lease_extension`` (set in the
# real ``__init__``).
w._lease_extension = 10
# Tracked P-side requests with a lease about to expire.
old_expiry = time.perf_counter() - 5.0
w._reqs_to_send["req-a"] = old_expiry
w._reqs_to_send["req-b"] = old_expiry
# Forwarded heartbeat covers a tracked request, an unknown one,
# and another tracked one.
w._pending_completion_notifs.put(b"HB:req-a,req-unknown,req-b")
notified = w._get_new_notifs()
assert notified == set()
# Tracked leases were renewed strictly forward in time.
now = time.perf_counter()
for rid in ("req-a", "req-b"):
assert w._reqs_to_send[rid] > old_expiry
# New expiry must be roughly now + _lease_extension.
assert w._reqs_to_send[rid] >= now
# Unknown request must not be inserted by the heartbeat path.
assert "req-unknown" not in w._reqs_to_send
@@ -44,7 +44,7 @@ from vllm.v1.simple_kv_offload.metadata import (
)
NIXL_WRAPPER_PATCH = (
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker.NixlWrapper"
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker.NixlWrapper"
)
@@ -587,7 +587,9 @@ def test_cannot_recv():
assert_scheduler_empty(scheduler)
@patch("vllm.distributed.kv_transfer.kv_connector.v1.nixl.scheduler.current_platform")
@patch(
"vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_scheduler.current_platform"
)
def test_p_side_chunked_prefill_mamba(mock_platform):
"""P-side integration: Mamba N-1 truncation + chunked prefill completes.
+63
View File
@@ -524,3 +524,66 @@ def make_nixl_scheduler(
sched.blocks_per_sw = []
sched.is_bidirectional_kv_xfer_enabled = False
return sched
def make_nixl_push_scheduler(
*,
decoder_kv_blocks_ttl: float = 30.0,
push_registration_timeout: float | None = None,
is_bidirectional_kv_xfer_enabled: bool = False,
has_mamba: bool = False,
):
"""Create a NixlPushConnectorScheduler via __new__ (skipping __init__).
The push scheduler can't reuse :func:`make_nixl_scheduler` because it
is a different class (``NixlPushConnectorScheduler`` vs
``NixlConnectorScheduler``) and carries push-specific state. Only the
fields touched by the unit tests are populated.
"""
from unittest.mock import MagicMock
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.push_scheduler import (
NixlPushConnectorScheduler,
)
sched = object.__new__(NixlPushConnectorScheduler)
# Base scheduler fields (shared with pull / heartbeat path).
sched._reqs_need_recv = {}
sched._reqs_need_send = {}
sched._reqs_in_batch = set()
sched._reqs_not_processed = set()
sched._reqs_need_save = {}
sched._kv_lease_duration = 30
sched.decoder_kv_blocks_ttl = decoder_kv_blocks_ttl
sched.use_host_buffer = False
sched.engine_id = "decode-engine"
sched.side_channel_host = "127.0.0.1"
sched.side_channel_port = 5600
sched.is_bidirectional_kv_xfer_enabled = is_bidirectional_kv_xfer_enabled
sched._has_mamba = has_mamba
# vllm_config is consulted for parallel_config.tensor_parallel_size.
vllm_config = MagicMock()
vllm_config.parallel_config.tensor_parallel_size = 1
sched.vllm_config = vllm_config
# Push-specific state.
sched._push_pending_registrations = {}
sched._push_registration_deadlines = {}
sched._finished_request_blocks = {}
sched._newly_finished_push_blocks = {}
sched._push_registration_timeout = (
push_registration_timeout
if push_registration_timeout is not None
else decoder_kv_blocks_ttl
)
# Heartbeat fields touched by base request_finished /
# update_connector_output.
sched._heartbeat_by_engine = {}
sched._heartbeat_req_engine = {}
sched._last_heartbeat_time = 0.0
sched.blocks_per_sw = []
return sched
@@ -179,6 +179,18 @@ KVConnectorFactory.register_connector(
"NixlConnector",
)
KVConnectorFactory.register_connector(
"NixlPullConnector",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl",
"NixlPullConnector",
)
KVConnectorFactory.register_connector(
"NixlPushConnector",
"vllm.distributed.kv_transfer.kv_connector.v1.nixl",
"NixlPushConnector",
)
KVConnectorFactory.register_connector(
"MultiConnector",
"vllm.distributed.kv_transfer.kv_connector.v1.multi_connector",
@@ -569,6 +569,18 @@ class KVConnectorBase_V1(ABC):
"""
return ()
def has_pending_push_work(self) -> bool:
"""Return True if the connector has push-mode work that requires
the engine main loop to keep stepping (e.g. a P-side request whose
KV blocks are waiting to be WRITTEN to a D node).
Connectors that don't implement push-based KV transfer should
leave this as False.
"""
# TODO: replace with a more general connector hook for keeping the
# scheduler alive (e.g. extend has_unfinished_requests).
return False
@classmethod
def get_required_kvcache_layout(cls, vllm_config: "VllmConfig") -> str | None:
"""
@@ -538,6 +538,9 @@ class MultiConnector(KVConnectorBase_V1, SupportsHMA):
for c in self._connectors:
yield from c.take_events()
def has_pending_push_work(self) -> bool:
return any(c.has_pending_push_work() for c in self._connectors)
@classmethod
def get_required_kvcache_layout(cls, vllm_config: "VllmConfig") -> str | None:
"""
@@ -2,14 +2,35 @@
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""NIXL KV-cache transfer connector (disaggregated prefill / decode)."""
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_scheduler import (
NixlBaseConnectorScheduler,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker import (
NixlBaseConnectorWorker,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.connector import (
NixlBaseConnector,
NixlConnector,
NixlPullConnector,
NixlPushConnector,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.metadata import (
NixlAgentMetadata,
NixlConnectorMetadata,
NixlHandshakePayload,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.pull_scheduler import (
NixlPullConnectorScheduler,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.pull_worker import (
NixlPullConnectorWorker,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.push_scheduler import (
NixlPushConnectorScheduler,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.push_worker import (
NixlPushConnectorWorker,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.scheduler import (
NixlConnectorScheduler,
)
@@ -22,10 +43,19 @@ from vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker import (
__all__ = [
"NixlAgentMetadata",
"NixlBaseConnector",
"NixlBaseConnectorScheduler",
"NixlBaseConnectorWorker",
"NixlConnector",
"NixlConnectorMetadata",
"NixlConnectorScheduler",
"NixlConnectorWorker",
"NixlHandshakePayload",
"NixlKVConnectorStats",
"NixlPullConnector",
"NixlPullConnectorScheduler",
"NixlPullConnectorWorker",
"NixlPushConnector",
"NixlPushConnectorScheduler",
"NixlPushConnectorWorker",
]
@@ -0,0 +1,455 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Base scheduler-side logic for the NIXL connector."""
import threading
import time
from typing import TYPE_CHECKING, Any
import msgspec
import zmq
from vllm import envs
from vllm.distributed.kv_transfer.kv_connector.utils import (
BlockIds,
EngineId,
yield_req_data,
)
from vllm.distributed.kv_transfer.kv_connector.v1.base import (
KVConnectorHandshakeMetadata,
KVConnectorMetadata,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.metadata import (
GET_META_MSG,
HeartbeatInfo,
NixlConnectorMetadata,
NixlHandshakePayload,
ReqId,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.utils import zmq_ctx
from vllm.logger import init_logger
from vllm.platforms import current_platform
from vllm.utils.math_utils import cdiv
from vllm.utils.network_utils import make_zmq_path
from vllm.v1.core.sched.output import SchedulerOutput
from vllm.v1.kv_cache_interface import (
FullAttentionSpec,
MambaSpec,
SlidingWindowSpec,
)
if TYPE_CHECKING:
from vllm.config import VllmConfig
from vllm.v1.core.kv_cache_manager import KVCacheBlocks
from vllm.v1.kv_cache_interface import KVCacheConfig
from vllm.v1.outputs import KVConnectorOutput
from vllm.v1.request import Request
logger = init_logger(__name__)
class NixlBaseConnectorScheduler:
"""Base implementation of Scheduler side methods shared by pull and push."""
def __init__(
self,
vllm_config: "VllmConfig",
engine_id: str,
kv_cache_config: "KVCacheConfig",
):
self.vllm_config = vllm_config
self.block_size = vllm_config.cache_config.block_size
self.engine_id: EngineId = engine_id
self.kv_cache_config = kv_cache_config
self.side_channel_host = envs.VLLM_NIXL_SIDE_CHANNEL_HOST
self.side_channel_port = (
envs.VLLM_NIXL_SIDE_CHANNEL_PORT
+ vllm_config.parallel_config.data_parallel_index
)
assert vllm_config.kv_transfer_config is not None
self._kv_lease_duration: int = (
vllm_config.kv_transfer_config.get_from_extra_config(
"kv_lease_duration", 30
)
)
# NOTE (NickLucche): For now we use a hardcoded value for a simpler interface.
self._heartbeat_interval = self._kv_lease_duration // 6
if current_platform.device_type == "cpu":
self.use_host_buffer = False
else:
self.use_host_buffer = (
vllm_config.kv_transfer_config.kv_buffer_device == "cpu"
)
self._is_hma_required = (
not vllm_config.scheduler_config.disable_hybrid_kv_cache_manager
# Also handle unlikely SW-only model case instead of checking num_groups>1.
and any(
not isinstance(g.kv_cache_spec, FullAttentionSpec)
for g in kv_cache_config.kv_cache_groups
)
)
self._has_mamba = any(
isinstance(g.kv_cache_spec, MambaSpec)
for g in kv_cache_config.kv_cache_groups
)
logger.info("Initializing NIXL Scheduler %s", engine_id)
if vllm_config.scheduler_config.disable_hybrid_kv_cache_manager:
logger.info("Hybrid Memory Allocator is enabled with NIXL")
# Background thread for handling new handshake requests.
self._nixl_handshake_listener_t: threading.Thread | None = None
self._stop_event = threading.Event()
# Requests that need to start recv/send.
# New requests are added by update_state_after_alloc in
# the scheduler. Used to make metadata passed to Worker.
self._reqs_need_recv: dict[ReqId, tuple[Request, BlockIds]] = {}
self._reqs_need_save: dict[ReqId, Request] = {}
# Reqs to send and their expiration time
self._reqs_need_send: dict[ReqId, float] = {}
self._reqs_in_batch: set[ReqId] = set()
# Reqs to remove from processed set because they're not to send after
# remote prefill or aborted.
self._reqs_not_processed: set[ReqId] = set()
# Heartbeat tracking: requests needing periodic lease-renewal heartbeats to
# remote P-side, stored as ready-to-send HeartbeatInfo grouped by remote engine
self._heartbeat_by_engine: dict[EngineId, HeartbeatInfo] = {}
# Reverse lookup: local req_id -> (engine_id, remote_req_id) for O(1) removal
self._heartbeat_req_engine: dict[ReqId, tuple[EngineId, ReqId]] = {}
self._last_heartbeat_time: float = 0.0
# Gather Sliding Window sizes for each kv cache group (if any) in number of
# blocks per KV cache group. This is used to clip the local attention window.
sw_sizes_tokens: list[tuple[int, int]] = [
(g.kv_cache_spec.sliding_window, g.kv_cache_spec.block_size)
if isinstance(g.kv_cache_spec, SlidingWindowSpec)
else (0, self.block_size)
for g in kv_cache_config.kv_cache_groups
]
# cdiv(n_tokens, block_size) gives blocks/window; add 1 to conservatively
# account for boundary overlap eg window isn't fully aligned with blocks.
self.blocks_per_sw = [
cdiv(n_tokens, block_size) + 1 if n_tokens else 0
for n_tokens, block_size in sw_sizes_tokens
]
# Threshold to decide whether to compute kv cache locally
# or pull from a remote node: minimum number of remote
# tokens to amortize the xfer latencies
self.kv_recompute_threshold: int = int(
vllm_config.kv_transfer_config.get_from_extra_config(
"kv_recompute_threshold", 64
)
)
# Bi-directional KV transfer feature supports KV block
# transfers from D node to P node
self.is_bidirectional_kv_xfer_enabled = (
vllm_config.kv_transfer_config.get_from_extra_config(
"bidirectional_kv_xfer", False
)
)
self.decoder_kv_blocks_ttl = (
vllm_config.kv_transfer_config.get_from_extra_config(
"decoder_kv_blocks_ttl", 480
)
)
if self.is_bidirectional_kv_xfer_enabled and self.kv_recompute_threshold > 0:
logger.info(
"Bidirectional KV transfer is enabled and the kv "
"recompute threshold is set to %d tokens."
"KV blocks on D are released after a TTL of %d seconds.",
self.kv_recompute_threshold,
self.decoder_kv_blocks_ttl,
)
def shutdown(self):
self._stop_event.set()
if self._nixl_handshake_listener_t is not None:
self._nixl_handshake_listener_t.join()
self._nixl_handshake_listener_t = None
def on_new_request(self, request: "Request") -> None:
"""Track a request that may need heartbeats."""
params = request.kv_transfer_params
# NOTE (NickLucche) This excludes request meant for P, ie heartbeats are
# effectively disabled for Bidirectional KV transfer.
if params is None or not params.get("do_remote_prefill"):
return
# Only track if all required remote fields are present.
remote_engine_id = params.get("remote_engine_id")
remote_request_id = params.get("remote_request_id")
host = params.get("remote_host")
port = params.get("remote_port")
tp_size = params.get("tp_size")
if (
remote_engine_id is None
or remote_request_id is None
or host is None
or port is None
or tp_size is None
):
return
if remote_engine_id not in self._heartbeat_by_engine:
self._heartbeat_by_engine[remote_engine_id] = HeartbeatInfo(
req_ids=set(),
host=host,
port=port,
tp_size=tp_size,
)
self._heartbeat_by_engine[remote_engine_id].req_ids.add(remote_request_id)
self._heartbeat_req_engine[request.request_id] = (
remote_engine_id,
remote_request_id,
)
def _stop_heartbeat(self, req_id: ReqId) -> None:
"""Remove *req_id* from heartbeat tracking (if tracked)."""
if key := self._heartbeat_req_engine.pop(req_id, None):
engine_id, remote_id = key
if info := self._heartbeat_by_engine.get(engine_id):
info.req_ids.discard(remote_id)
if not info.req_ids:
# Clean up empty engines so we don't leak a key when remote dies.
del self._heartbeat_by_engine[engine_id]
def get_sw_clipped_blocks(self, block_ids: BlockIds) -> BlockIds:
"""
Clip the number of blocks to the sliding window size for each kv cache group
that employs SWA.
This is necessary because the KV Cache manager initially allocates blocks for
the entire sequence length, and successively cleans up blocks that are outside
the window prior to the `request_finished_all_groups` hook.
"""
if len(block_ids) == 0 or not self._is_hma_required:
# No blocks to clip eg Full prefix cache hit or not a hybrid model.
return block_ids
# NOTE (NickLucche) This logic is currently handled at the connector level
# because offloading connectors might want to receive the whole sequence even
# for SWA groups. We will abstract this logic once the interface is more stable
assert len(block_ids) == len(self.blocks_per_sw), (
"Number of KV cache groups must match"
)
# For non-SWA groups, blocks_per_sw is 0 so we return all block_ids unchanged
return tuple(
[
blocks[-self.blocks_per_sw[i] :]
if self.blocks_per_sw[i] > 0
else blocks
for i, blocks in enumerate(block_ids)
]
)
def set_xfer_handshake_metadata(
self, metadata: dict[int, KVConnectorHandshakeMetadata]
) -> None:
"""
Set the KV connector handshake metadata for this connector.
Args:
metadata (dict): the handshake metadata to set.
"""
encoded_data: dict[int, bytes] = {}
encoder = msgspec.msgpack.Encoder()
for tp_rank, rank_metadata in metadata.items():
if not isinstance(rank_metadata, NixlHandshakePayload):
raise ValueError(
"NixlConnectorScheduler expects NixlHandshakePayload for "
"handshake metadata."
)
encoded_data[tp_rank] = encoder.encode(rank_metadata)
logger.debug(
"Tp rank %d: encoded NixlHandshakePayload size: %s bytes",
tp_rank,
str(len(encoded_data[tp_rank])),
)
# Only start the listener when we have metadata to serve.
if self._nixl_handshake_listener_t is None:
ready_event = threading.Event()
self._nixl_handshake_listener_t = threading.Thread(
target=self._nixl_handshake_listener,
args=(
encoded_data,
ready_event,
self._stop_event,
self.side_channel_host,
self.side_channel_port,
),
daemon=True,
name="nixl_handshake_listener",
)
self._nixl_handshake_listener_t.start()
ready_event.wait() # Wait for listener ZMQ socket to be ready.
@staticmethod
def _nixl_handshake_listener(
encoded_data: dict[int, Any],
ready_event: threading.Event,
stop_event: threading.Event,
host: str,
port: int,
):
"""Background thread for getting new NIXL handshakes."""
# NOTE(rob): this is a simple implementation. We will move
# to a better approach via HTTP endpoint soon.
# Listen for new requests for metadata.
path = make_zmq_path("tcp", host, port)
logger.debug("Starting listening on path: %s", path)
with zmq_ctx(zmq.ROUTER, path) as sock:
sock.setsockopt(zmq.RCVTIMEO, 1000)
ready_event.set()
while True:
try:
identity, _, msg = sock.recv_multipart()
except zmq.Again:
if stop_event.is_set():
break
continue
# Decode the message which contains (GET_META_MSG, rank)
msg, target_tp_rank = msgspec.msgpack.decode(msg)
logger.debug(
"Received message for tp rank %s",
target_tp_rank,
)
if msg != GET_META_MSG:
logger.warning("Connection listener got unexpected message %s", msg)
sock.send_multipart((identity, b"", encoded_data[target_tp_rank]))
def _mamba_prefill_token_count(self, num_prompt_tokens: int) -> int:
"""D-side only. Returns N-1 for Mamba models since the decoder
always recomputes the last token and must start from h(N-1)."""
if self._has_mamba and num_prompt_tokens > 1:
return num_prompt_tokens - 1
return num_prompt_tokens
def _truncate_mamba_request_for_prefill(self, request: "Request") -> None:
"""P-side only: drop the last prompt token so the prefiller computes
h(N-1) instead of h(N). The decoder recomputes the last token to
derive h(N) correctly.
Guarded by ``_p_side_truncated`` to avoid repeated truncation if the
request is preempted and rescheduled."""
params = request.kv_transfer_params
if (
params is not None
# Guard against repeated truncation after preemption/reschedule.
and not params.get("_p_side_truncated")
and request.num_prompt_tokens > 1
):
if request.prompt_token_ids is not None:
request.prompt_token_ids.pop()
elif request.prompt_embeds is not None:
request.prompt_embeds = request.prompt_embeds[:-1]
else:
return
request._all_token_ids.pop()
request.num_prompt_tokens -= 1
request.max_tokens = 1
params["_p_side_truncated"] = True
def _build_save_meta(
self,
meta: NixlConnectorMetadata,
scheduler_output: SchedulerOutput,
) -> None:
# only called when use_host_buffer is True to build the save metadata
# NOTE: For the prefill side, there might be a chance that an early added
# request is a chunked prefill, so we need to check if new blocks are added
for req_id, new_block_id_groups, _ in yield_req_data(scheduler_output):
req_to_save = self._reqs_need_save.get(req_id)
if req_to_save is None or new_block_id_groups is None:
continue
req = req_to_save
assert req.kv_transfer_params is not None
clipped_block_id_groups = self.get_sw_clipped_blocks(new_block_id_groups)
meta.add_new_req_to_save(
request_id=req_id,
local_block_ids=clipped_block_id_groups,
kv_transfer_params=req.kv_transfer_params,
)
assert scheduler_output.num_scheduled_tokens is not None
num_scheduled_tokens = scheduler_output.num_scheduled_tokens[req_id]
is_partial = (
req.num_computed_tokens + num_scheduled_tokens
) < req.num_prompt_tokens
if not is_partial:
# For non-partial prefills, once new req_meta is scheduled, it
# can be removed from _reqs_need_save.
# For partial prefill case, we will retain the request in
# _reqs_need_save until all blocks are scheduled with req_meta.
# Therefore, only pop if `not is_partial`.
self._reqs_need_save.pop(req_id)
def build_connector_meta(
self,
scheduler_output: SchedulerOutput,
) -> KVConnectorMetadata:
meta = NixlConnectorMetadata()
# Loop through scheduled reqs and convert to ReqMeta.
for req_id, (req, block_ids) in self._reqs_need_recv.items():
assert req.kv_transfer_params is not None
meta.add_new_req_to_recv(
request_id=req_id,
local_block_ids=block_ids,
kv_transfer_params=req.kv_transfer_params,
)
if self.use_host_buffer:
self._build_save_meta(meta, scheduler_output)
meta.reqs_to_send = self._reqs_need_send
meta.reqs_in_batch = self._reqs_in_batch
meta.reqs_not_processed = self._reqs_not_processed
# Package heartbeats, throttled by heartbeat_interval.
if self._heartbeat_by_engine:
now = time.perf_counter()
if now - self._last_heartbeat_time >= self._heartbeat_interval:
self._last_heartbeat_time = now
meta.heartbeat_by_engine = self._heartbeat_by_engine
# Clear the list once workers start the transfers
self._reqs_need_recv.clear()
self._reqs_in_batch = set()
self._reqs_not_processed = set()
self._reqs_need_send = {}
return meta
def update_connector_output(self, connector_output: "KVConnectorOutput") -> None:
"""Stop heartbeating for requests whose KV transfer completed."""
for req_id in connector_output.finished_recving or ():
self._stop_heartbeat(req_id)
def has_pending_push_work(self) -> bool:
return False
############################################################
# Abstract methods that subclasses must implement
############################################################
def get_num_new_matched_tokens(
self, request: "Request", num_computed_tokens: int
) -> tuple[int, bool]:
raise NotImplementedError
def update_state_after_alloc(
self, request: "Request", blocks: "KVCacheBlocks", num_external_tokens: int
):
raise NotImplementedError
def request_finished(
self,
request: "Request",
block_ids: BlockIds,
) -> tuple[bool, dict[str, Any] | None]:
raise NotImplementedError
File diff suppressed because it is too large Load Diff
@@ -1,6 +1,16 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""NixlConnector – thin facade that delegates to scheduler / worker."""
"""NIXL connector facades.
This module hosts the thin facade classes that vLLM's KV-connector layer
instantiates. Almost all the real work lives in the per-mode scheduler
and worker classes; the connector classes here only forward calls.
* :class:`NixlBaseConnector` – common logic shared by pull and push.
* :class:`NixlPullConnector` – pull-based (READ) KV transfer.
* :class:`NixlPushConnector` – push-based (WRITE) KV transfer.
* ``NixlConnector`` – backward-compatible alias for :class:`NixlPullConnector`.
"""
from typing import TYPE_CHECKING, Any
@@ -28,16 +38,22 @@ from vllm.distributed.kv_transfer.kv_connector.v1.metrics import (
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.metadata import (
NixlConnectorMetadata,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.scheduler import (
NixlConnectorScheduler,
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.pull_scheduler import (
NixlPullConnectorScheduler,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.pull_worker import (
NixlPullConnectorWorker,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.push_scheduler import (
NixlPushConnectorScheduler,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.push_worker import (
NixlPushConnectorWorker,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.stats import (
NixlKVConnectorStats,
NixlPromMetrics,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.worker import (
NixlConnectorWorker,
)
from vllm.forward_context import ForwardContext
from vllm.logger import init_logger
from vllm.v1.attention.backend import AttentionBackend, AttentionMetadata
@@ -47,6 +63,12 @@ from vllm.v1.kv_cache_interface import MambaSpec
from vllm.v1.outputs import KVConnectorOutput
if TYPE_CHECKING:
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_scheduler import (
NixlBaseConnectorScheduler,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker import (
NixlBaseConnectorWorker,
)
from vllm.v1.core.kv_cache_manager import KVCacheBlocks
from vllm.v1.kv_cache_interface import KVCacheConfig
from vllm.v1.request import Request
@@ -54,7 +76,9 @@ if TYPE_CHECKING:
logger = init_logger(__name__)
class NixlConnector(KVConnectorBase_V1, SupportsHMA):
class NixlBaseConnector(KVConnectorBase_V1, SupportsHMA):
"""Base connector with common logic shared by pull and push modes."""
@property
def prefer_cross_layer_blocks(self) -> bool:
if any(
@@ -106,16 +130,9 @@ class NixlConnector(KVConnectorBase_V1, SupportsHMA):
self.kv_cache_config = kv_cache_config
self.engine_id: EngineId = vllm_config.kv_transfer_config.engine_id
self.kv_transfer_config = vllm_config.kv_transfer_config
if role == KVConnectorRole.SCHEDULER:
self.connector_scheduler: NixlConnectorScheduler | None = (
NixlConnectorScheduler(vllm_config, self.engine_id, kv_cache_config)
)
self.connector_worker: NixlConnectorWorker | None = None
elif role == KVConnectorRole.WORKER:
self.connector_scheduler = None
self.connector_worker = NixlConnectorWorker(
vllm_config, self.engine_id, kv_cache_config
)
# Subclasses must set self.connector_scheduler and self.connector_worker
self.connector_scheduler: NixlBaseConnectorScheduler | None = None
self.connector_worker: NixlBaseConnectorWorker | None = None
############################################################
# Class Methods
@@ -256,11 +273,6 @@ class NixlConnector(KVConnectorBase_V1, SupportsHMA):
vllm_config, metric_types, labelnames, per_engine_labelvalues
)
def start_load_kv(self, forward_context: "ForwardContext", **kwargs) -> None:
assert self.connector_worker is not None
assert isinstance(self._connector_metadata, NixlConnectorMetadata)
self.connector_worker.start_load_kv(self._connector_metadata)
def wait_for_layer_load(self, layer_name: str) -> None:
"""NixlConnector does not do layerwise saving."""
pass
@@ -281,6 +293,11 @@ class NixlConnector(KVConnectorBase_V1, SupportsHMA):
if self.connector_worker.use_host_buffer and self.connector_worker.copy_blocks:
self.connector_worker.save_kv_to_host(self._connector_metadata)
def has_pending_push_work(self) -> bool:
if self.connector_scheduler is not None:
return self.connector_scheduler.has_pending_push_work()
return False
def shutdown(self):
if self.connector_worker is not None:
self.connector_worker.shutdown()
@@ -299,3 +316,79 @@ class NixlConnector(KVConnectorBase_V1, SupportsHMA):
"""
assert self.connector_worker is not None
return self.connector_worker.xfer_handshake_metadata
class NixlPullConnector(NixlBaseConnector):
"""Pull-based (READ) NIXL KV transfer connector."""
def __init__(
self,
vllm_config: VllmConfig,
role: KVConnectorRole,
kv_cache_config: "KVCacheConfig",
):
super().__init__(vllm_config, role, kv_cache_config)
if role == KVConnectorRole.SCHEDULER:
self.connector_scheduler = NixlPullConnectorScheduler(
vllm_config, self.engine_id, kv_cache_config
)
self.connector_worker = None
elif role == KVConnectorRole.WORKER:
self.connector_scheduler = None
self.connector_worker = NixlPullConnectorWorker(
vllm_config, self.engine_id, kv_cache_config
)
def start_load_kv(self, forward_context: "ForwardContext", **kwargs) -> None:
assert self.connector_worker is not None
assert isinstance(self.connector_worker, NixlPullConnectorWorker)
assert isinstance(self._connector_metadata, NixlConnectorMetadata)
self.connector_worker.start_load_kv(self._connector_metadata)
class NixlPushConnector(NixlBaseConnector):
"""Push-based (WRITE) NIXL KV transfer connector."""
def __init__(
self,
vllm_config: VllmConfig,
role: KVConnectorRole,
kv_cache_config: "KVCacheConfig",
):
super().__init__(vllm_config, role, kv_cache_config)
self.connector_scheduler: NixlPushConnectorScheduler | None = None
self.connector_worker: NixlPushConnectorWorker | None = None
if role == KVConnectorRole.SCHEDULER:
self.connector_scheduler = NixlPushConnectorScheduler(
vllm_config, self.engine_id, kv_cache_config
)
elif role == KVConnectorRole.WORKER:
self.connector_worker = NixlPushConnectorWorker(
vllm_config, self.engine_id, kv_cache_config
)
else:
raise ValueError(f"Unsupported KVConnectorRole: {role}")
def start_load_kv(self, forward_context: "ForwardContext", **kwargs) -> None:
"""Drive push processing on the worker.
The worker enqueues registrations / finished blocks for the
background ``nixl-push-writer`` thread; the writer issues the
WRITE transfers and polls NIXL notifs without further
engine-thread involvement.
"""
assert self.connector_worker is not None
assert isinstance(self._connector_metadata, NixlConnectorMetadata)
self.connector_worker.start_load_kv(self._connector_metadata)
# Backward compatibility: NixlConnector is the pull-based connector.
NixlConnector = NixlPullConnector
__all__ = [
"NixlBaseConnector",
"NixlConnector",
"NixlPullConnector",
"NixlPushConnector",
]
@@ -19,6 +19,11 @@ TransferHandle = int
ReqId = str
GET_META_MSG = b"get_meta_msg"
# Push-mode (WRITE-based) registration notification.
# Sent worker-to-worker over NIXL: D worker -> P worker, encoded as
# PUSH_REG_NOTIF_PREFIX + msgpack(registration_data).
PUSH_REG_NOTIF_PREFIX = b"PUSH_REG:"
#
# NIXL Connector Version
#
@@ -160,6 +165,8 @@ class ReqMeta:
local_physical_block_ids: BlockIds
tp_size: int
remote: RemoteMeta | None = None
# Remote block size, discovered during NIXL handshake (push mode).
remote_block_size: int | None = None
class NixlConnectorMetadata(KVConnectorMetadata):
@@ -171,6 +178,12 @@ class NixlConnectorMetadata(KVConnectorMetadata):
self.reqs_not_processed: set[ReqId] = set()
# Heartbeat data grouped by remote engine, sent by D worker to P.
self.heartbeat_by_engine: dict[EngineId, HeartbeatInfo] = {}
# Push mode (D side): registration data the D worker should send to
# P workers via NIXL notification on this step.
self.push_registrations: dict[ReqId, dict[str, Any]] = {}
# Push mode (P side): newly finished request blocks to be matched
# against pending D registrations on the P worker.
self.push_finished_blocks: dict[ReqId, BlockIds] = {}
def _add_new_req(
self,
@@ -182,6 +195,7 @@ class NixlConnectorMetadata(KVConnectorMetadata):
local_physical_block_ids=local_block_ids,
# P workers don't need to receive tp_size from proxy here.
tp_size=kv_transfer_params.get("tp_size", 1),
remote_block_size=kv_transfer_params.get("remote_block_size"),
)
def add_new_req_to_save(
@@ -0,0 +1,275 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Pull-specific scheduler-side logic for the NIXL connector."""
import time
from typing import TYPE_CHECKING, Any
from vllm.distributed.kv_transfer.kv_connector.utils import BlockIds
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_scheduler import (
NixlBaseConnectorScheduler,
)
from vllm.logger import init_logger
if TYPE_CHECKING:
from vllm.config import VllmConfig
from vllm.v1.core.kv_cache_manager import KVCacheBlocks
from vllm.v1.kv_cache_interface import KVCacheConfig
from vllm.v1.request import Request
logger = init_logger(__name__)
class NixlPullConnectorScheduler(NixlBaseConnectorScheduler):
"""Pull-specific scheduler logic (READ-based KV transfer)."""
def __init__(
self,
vllm_config: "VllmConfig",
engine_id: str,
kv_cache_config: "KVCacheConfig",
):
super().__init__(vllm_config, engine_id, kv_cache_config)
def get_num_new_matched_tokens(
self, request: "Request", num_computed_tokens: int
) -> tuple[int, bool]:
"""
For remote prefill, pull all prompt blocks from remote
asynchronously relative to engine execution.
Args:
request (Request): the request object.
num_computed_tokens (int): the number of locally
computed tokens for this request
Returns:
* the number of tokens that can be loaded from the
external KV cache beyond what is already computed.
* true if the external KV cache tokens will be loaded
asynchronously (between scheduler steps).
"""
params = request.kv_transfer_params
logger.debug(
"NIXLConnector get_num_new_matched_tokens: "
"num_computed_tokens=%s, kv_transfer_params=%s",
num_computed_tokens,
params,
)
if params is not None and params.get("do_remote_prefill"):
# Remote prefill: get all prompt blocks from remote.
token_ids = request.prompt_token_ids or []
actual = self._mamba_prefill_token_count(len(token_ids))
count = actual - num_computed_tokens
if count > 0:
return count, True
if params is not None and params.get("do_remote_decode") and self._has_mamba:
self._truncate_mamba_request_for_prefill(request)
if (
params is not None
and params.get("do_remote_decode")
and params.get("remote_block_ids")
and all(
p in params
for p in (
"remote_engine_id",
"remote_request_id",
"remote_host",
"remote_port",
)
)
):
# Decode node has kv blocks for part of prefill request, so, provide them
# as an external token count to scheduler.
# The tokens will be loaded if not already present
# in the prefill node local cache
remote_num_tokens = params.get("remote_num_tokens") or 0
count = (
min(remote_num_tokens, request.num_prompt_tokens) - num_computed_tokens
)
if count > 0:
# Check kv_recompute_threshold: skip pull if
# remote tokens are below the threshold.
if (
self.kv_recompute_threshold > 0
and count < self.kv_recompute_threshold
):
logger.debug(
"Skipping remote pull for %s: %d remote tokens < threshold %d",
request.request_id,
count,
self.kv_recompute_threshold,
)
return 0, False
return count, True
# No remote prefill for this request.
return 0, False
def update_state_after_alloc(
self, request: "Request", blocks: "KVCacheBlocks", num_external_tokens: int
):
params = request.kv_transfer_params
logger.debug(
"NIXLConnector update_state_after_alloc: "
"num_external_tokens=%s, kv_transfer_params=%s",
num_external_tokens,
params,
)
if not params:
return
if params.get("do_remote_decode") or (
params.get("do_remote_prefill") and self.is_bidirectional_kv_xfer_enabled
):
self._reqs_in_batch.add(request.request_id)
if self.use_host_buffer and params.get("do_remote_decode"):
# NOTE: when accelerator is not directly supported by Nixl,
# prefilled blocks need to be saved to host memory before transfer.
self._reqs_need_save[request.request_id] = request
elif params.get("do_remote_prefill") or (
params.get("do_remote_decode")
and self.is_bidirectional_kv_xfer_enabled
and not params.get("_remote_blocks_processed")
):
if params.get("remote_block_ids"):
if all(
p in params
for p in (
"remote_engine_id",
"remote_request_id",
"remote_host",
"remote_port",
)
):
# If remote_blocks and num_external_tokens = 0, we have
# a full prefix cache hit on the local node. We need to call
# send_notif in _read_blocks to free the memory on the remote node.
unhashed_local_block_ids: BlockIds = (
blocks.get_unhashed_block_ids_all_groups()
if num_external_tokens > 0
else ()
)
local_block_ids = self.get_sw_clipped_blocks(
unhashed_local_block_ids
)
# Get unhashed blocks to pull from remote. Mind that a full prefix
# cache hit is indicated with an empty list.
self._reqs_need_recv[request.request_id] = (
request,
local_block_ids,
)
else:
logger.warning(
"Got invalid KVTransferParams: %s. This "
"request will not utilize KVTransfer",
params,
)
else:
assert num_external_tokens == 0
# Only trigger 1 KV transfer per request.
params["do_remote_prefill"] = False
params["_remote_blocks_processed"] = True
def request_finished(
self,
request: "Request",
block_ids: "BlockIds",
) -> tuple[bool, dict[str, Any] | None]:
"""
Once a request is finished, determine whether request blocks
should be freed now or will be sent asynchronously and freed later.
"""
from vllm.v1.request import RequestStatus
params = request.kv_transfer_params
logger.debug(
"NIXLConnector request_finished(%s), request_status=%s, "
"kv_transfer_params=%s",
request.request_id,
request.status,
params,
)
if not params:
return False, None
is_p_node = bool(params.get("do_remote_decode"))
is_d_node = not is_p_node
# Stop heartbeating for aborted requests that never reached finished_recving:
# normal path cleans up in update_connector_output.
self._stop_heartbeat(request.request_id)
if params.get("do_remote_prefill"):
# If do_remote_prefill is still True when the request is finished,
# update_state_after_alloc must not have been called (the request
# must have been aborted before it was scheduled, e.g. via the
# abort_immediately path used to clean up KV-transfer requests
# rejected at the D-side serving layer).
# To avoid stranding the prefill blocks in the prefill instance,
# we must add empty block_ids to _reqs_need_recv so that our
# worker side will notify and free blocks in the prefill instance.
self._reqs_need_recv[request.request_id] = (request, [])
params["do_remote_prefill"] = False
return False, None
if is_d_node and not self.is_bidirectional_kv_xfer_enabled:
return False, None
if request.status not in (
RequestStatus.FINISHED_LENGTH_CAPPED,
RequestStatus.FINISHED_STOPPED,
):
# Also include the case of a P/D Prefill request with immediate
# block free (eg abort). Stop tracking this request.
self._reqs_not_processed.add(request.request_id)
# Clear _reqs_need_save if a request is aborted as partial prefill.
self._reqs_need_save.pop(request.request_id, None)
return False, None
# TODO: check whether block_ids actually ever be 0. If not we could
# remove the conditional below
delay_free_blocks = any(len(group) > 0 for group in block_ids)
remote_num_tokens = 0
if delay_free_blocks:
# Prefill request on remote. It will be read from D upon completion
request_kv_blocks_ttl = self._kv_lease_duration
if is_d_node:
# For blocks pinned on D, use a simpler timeout for now instead of a
# lease mechanism as turn2 request is client-driven.
request_kv_blocks_ttl = self.decoder_kv_blocks_ttl
logger.debug(
"NIXLConnector request_finished(%s) waiting for %d seconds "
"before releasing blocks",
request.request_id,
request_kv_blocks_ttl,
)
self._reqs_need_send[request.request_id] = (
time.perf_counter() + request_kv_blocks_ttl
)
# NOTE HMA will "mark" empty/null blocks in groups with 0s (eg SWA ones),
# trimming down after allocating for the whole sequence length. Empty
# blocks are always at the start of the list.
# Here we "unpad" blocks to send the actual remote blocks to be read.
block_ids = self.get_sw_clipped_blocks(block_ids)
remote_num_tokens = request.num_computed_tokens
return delay_free_blocks, dict(
do_remote_prefill=is_p_node,
do_remote_decode=is_d_node,
remote_block_ids=block_ids,
remote_engine_id=self.engine_id,
remote_request_id=request.request_id,
remote_host=self.side_channel_host,
remote_port=self.side_channel_port,
tp_size=self.vllm_config.parallel_config.tensor_parallel_size,
remote_num_tokens=remote_num_tokens,
)
@@ -0,0 +1,382 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Pull-specific (READ) worker-side logic for the NIXL connector."""
import time
from typing import TYPE_CHECKING
import numpy as np
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker import (
NixlBaseConnectorWorker,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.metadata import (
NixlConnectorMetadata,
ReqMeta,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.tp_mapping import (
ReadSpec,
)
from vllm.logger import init_logger
if TYPE_CHECKING:
from vllm.config import VllmConfig
from vllm.v1.kv_cache_interface import KVCacheConfig
logger = init_logger(__name__)
class NixlPullConnectorWorker(NixlBaseConnectorWorker):
"""Pull-specific (READ) worker logic."""
def __init__(
self,
vllm_config: "VllmConfig",
engine_id: str,
kv_cache_config: "KVCacheConfig",
):
super().__init__(vllm_config, engine_id, kv_cache_config)
def start_load_kv(self, metadata: NixlConnectorMetadata):
"""
Start loading by triggering non-blocking nixl_xfer.
We check for these trnxs to complete in each step().
"""
for req_id, meta in metadata.reqs_to_recv.items():
meta.local_physical_block_ids = self._logical_to_kernel_block_ids(
meta.local_block_ids
)
assert meta.remote is not None
# Remote block IDs are kept logical here; expanded in
# _read_blocks_for_req using the remote engine's phys ratio.
remote_engine_id = meta.remote.engine_id
logger.debug(
"start_load_kv for request %s from remote engine %s. "
"Num local_block_ids: %s. Num remote_block_ids: %s. ",
req_id,
remote_engine_id,
len(meta.local_physical_block_ids),
len(meta.remote.block_ids),
)
# always store metadata for failure recovery
self._recving_metadata[req_id] = meta
if remote_engine_id not in self._remote_agents:
# Initiate handshake with remote engine to exchange metadata.
with self._handshake_lock:
if remote_engine_id not in self._remote_agents:
self._background_nixl_handshake(req_id, remote_engine_id, meta)
continue
# Handshake already completed, start async read xfer.
self._read_blocks_for_req(req_id, meta)
# Start transfers for requests whose handshakes have now finished.
while not self._ready_requests.empty():
self._read_blocks_for_req(*self._ready_requests.get_nowait())
# Keep around the requests that have been part of a batch. This is
# needed because async scheduling pushes the misalignment between the
# moment in which requests expiration is set (P side) and the moment in
# which blocks are read from D. As P can now more easily lag behind D
# while processing the next batch, we make sure to only set an
# expiration for requests that have not been read from D yet.
for req_id in metadata.reqs_in_batch:
self._reqs_to_process.add(req_id)
# Remove all requests that are not to be processed (eg aborted).
for req_id in metadata.reqs_not_processed:
self._reqs_to_process.discard(req_id)
# We should never get an abort after setting an expiry timer
assert req_id not in self._reqs_to_send
# Add to requests that are waiting to be read and track expiration.
for req_id, expiration_time in metadata.reqs_to_send.items():
if req_id in self._reqs_to_process:
self._reqs_to_send[req_id] = expiration_time
# Send heartbeats to P-side engines to keep KV blocks alive while
# requests sit in the D scheduler WAITING queue.
self._send_heartbeats(metadata)
def _read_blocks_for_req(self, req_id: str, meta: ReqMeta):
assert meta.remote is not None and self.transfer_topo is not None
engine_id = meta.remote.engine_id
# Update last activity from this remote. Mind that cleanup is done on main
# thread (this one), so we don't race on this structure.
self._engine_last_active[engine_id] = time.perf_counter()
plan = self.tp_mappings[engine_id]
remote_info = self.transfer_topo.get_engine_info(engine_id)
tp_ratio = self.transfer_topo.tp_ratio(remote_info.remote_tp_size)
meta.remote.block_ids = self._logical_to_remote_kernel_block_ids(
meta.remote.block_ids,
remote_info.remote_physical_blocks_per_logical,
)
remote_block_ids = meta.remote.block_ids
local_block_ids = meta.local_physical_block_ids
num_groups = len(local_block_ids)
read_specs = [
ReadSpec(
remote_rank=rank,
local_block_ids=[
list(local_block_ids[g])
if rank in plan.source_ranks_per_group[g]
else []
for g in range(num_groups)
],
remote_block_ids=[
list(remote_block_ids[g])
if rank in plan.source_ranks_per_group[g]
else []
for g in range(num_groups)
],
)
for rank in plan.all_source_ranks
]
# D may have to perform multiple reads from different remote ranks.
# MLA opt: when P TP > D TP, only a single read is executed for
# the first remote rank (cache is duplicated)..
if self.use_mla and tp_ratio < 0:
assert len(read_specs) == 1
for i, spec in enumerate(read_specs):
remote_block_size = remote_info.remote_block_size
logger.debug(
"Remote agent %s available, calling _read_blocks"
" on remote rank %s with remote block size %s for req %s",
meta.remote.engine_id,
spec.remote_rank,
remote_block_size,
req_id,
)
# Get side handles.
if tp_ratio < 0 and not self.use_mla:
assert remote_block_size == self.block_size
# Remote tp_size > local tp_size: we must perform multiple
# reads. Get the memory chunk onto which we will write to.
local_xfer_side_handle = self.src_xfer_handles_by_tp_ratio[tp_ratio][i]
else:
# Single read from remote, we write to the whole memory region.
# Also handle remote block size different from local block size.
local_xfer_side_handle = self.src_xfer_handles_by_block_size[
remote_block_size
]
# Destination handle: remote_engine_id -> remote_rank -> handle.
remote_xfer_side_handle = self.dst_xfer_side_handles[meta.remote.engine_id][
spec.remote_rank
]
self._read_blocks(
read_spec=spec,
request_id=req_id,
dst_engine_id=meta.remote.engine_id,
remote_request_id=meta.remote.request_id,
local_xfer_side_handle=local_xfer_side_handle,
remote_xfer_side_handle=remote_xfer_side_handle,
)
if self.use_mla and tp_ratio < 0 and read_specs:
# ..but we still need to notify the other remote ranks that we
# have the blocks we need so they can update the request state.
notif_id = f"{meta.remote.request_id}:{self.world_size}".encode()
remote_agents = self._remote_agents[meta.remote.engine_id]
for rank_to_notify, agent in remote_agents.items():
if rank_to_notify != read_specs[0].remote_rank:
self.nixl_wrapper.send_notif(agent, notif_msg=notif_id)
def _read_blocks(
self,
read_spec: ReadSpec,
dst_engine_id: str,
request_id: str,
remote_request_id: str,
local_xfer_side_handle: int,
remote_xfer_side_handle: int,
):
"""
Post a READ point-to-point xfer request from a single local worker to
a single remote worker.
"""
assert self.transfer_topo is not None
remote_rank = read_spec.remote_rank
local_block_ids = read_spec.local_block_ids
remote_block_ids = read_spec.remote_block_ids
remote_info = self.transfer_topo.get_engine_info(dst_engine_id)
block_size_ratio = self.transfer_topo.block_size_ratio(
remote_info.remote_block_size
)
if block_size_ratio > 1:
# TODO (NickLucche) assume HMA is off. Change to handle multiple KV groups.
assert not self._is_hma_required
local_block_ids0 = local_block_ids[0] if local_block_ids else []
remote_block_ids0 = remote_block_ids[0]
local_block_ids_mapped = self.get_mapped_blocks(
np.asarray(local_block_ids0), block_size_ratio
).tolist()
if len(local_block_ids_mapped) > len(remote_block_ids0):
# NOTE:
# get_mapped_blocks will always expand block_ids for n times.
# ex:
# prefill block_ids with block_size as 4:
# [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# Local decode block_ids with block_size as 16: [1, 2, 3]
# expanded decode block_ids with get_mapped_blocks from [1, 2, 3] to
# [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
# Then we clip local to align with prefill
# [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] to
# [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
local_block_ids_mapped = local_block_ids_mapped[
: len(remote_block_ids0)
]
local_block_ids = [local_block_ids_mapped] if local_block_ids_mapped else []
remote_block_ids = [remote_block_ids0]
# NOTE(rob): having the staging blocks be on the READER side is
# not going to work well (since we will have to call rearrange tensors).
# after we detect the txn is complete (which means we cannot make the
# read trxn async easily). If we want to make "READ" happen cleanly,
# then we will need to have the staging blocks on the remote side.
# NOTE(rob): according to nvidia the staging blocks are used to
# saturate IB with heterogeneous TP sizes.
# Number of D TP workers that will read from dst P. Propagate info
# on notification so that dst worker can wait before freeing blocks.
notif_id = f"{remote_request_id}:{self.world_size}".encode()
# Full prefix cache hit: do not need to read remote blocks,
# just notify P worker that we have the blocks we need.
if len(local_block_ids) == 0:
# A full prefix cache hit is indicated with an empty list.
agent_name = self._remote_agents[dst_engine_id][remote_rank]
try:
self.nixl_wrapper.send_notif(agent_name, notif_msg=notif_id)
except Exception as e:
self._log_failure(
failure_type="notification_failed",
msg="P worker blocks will be freed after timeout. "
"This may indicate network issues.",
req_id=request_id,
error=e,
dst_engine_id=dst_engine_id,
remote_rank=remote_rank,
remote_agent_name=agent_name,
)
self.xfer_stats.record_failed_notification()
return
assert (
len(remote_block_ids)
== len(local_block_ids)
== len(self.kv_cache_config.kv_cache_groups)
)
remote_physical_per_logical = remote_info.remote_physical_blocks_per_logical
local_block_ids, remote_block_ids = self._apply_prefix_caching(
local_block_ids, remote_block_ids, remote_physical_per_logical
)
# NOTE (nicolo) With homogeneous TP, each TP worker loads KV from
# corresponding rank. With heterogeneous TP, fixing D>P, the D tp
# workers will issue xfers to parts of the P worker remote kv caches.
# Get descs ids.
remote_block_descs_ids = self._compute_desc_ids(
block_ids=remote_block_ids,
dst_num_blocks=self.dst_num_blocks[dst_engine_id],
block_size_ratio=None,
physical_blocks_per_logical=remote_info.remote_physical_blocks_per_logical,
)
local_block_descs_ids = self._compute_desc_ids(
block_ids=local_block_ids,
dst_num_blocks=self.dst_num_blocks[self.engine_id],
block_size_ratio=block_size_ratio,
physical_blocks_per_logical=self._physical_blocks_per_logical_kv_block,
)
assert len(local_block_descs_ids) == len(remote_block_descs_ids)
# Prepare transfer with Nixl.
handle = None
try:
handle = self.nixl_wrapper.make_prepped_xfer(
"READ",
local_xfer_side_handle,
local_block_descs_ids,
remote_xfer_side_handle,
remote_block_descs_ids,
notif_msg=notif_id,
)
# Begin async xfer.
self.nixl_wrapper.transfer(handle)
# Use handle to check completion in future step().
self._recving_transfers[request_id].append(handle)
except Exception as e:
# mark all (logical) blocks for this request as invalid
self._log_failure(
failure_type="transfer_setup_failed",
req_id=request_id,
msg="Marking blocks as invalid",
error=e,
dst_engine_id=dst_engine_id,
remote_rank=remote_rank,
)
self._handle_failed_transfer(request_id, handle)
def _get_new_notifs(self) -> set[str]:
"""
Get req_ids which got a remote xfer message. When multiple consumers
are reading from the same producer (heterogeneous TP scenario), wait
for all consumers to be done pulling.
Also handles heartbeat notifications ("HB:req1,req2,...") by
extending the lease on the referenced requests.
"""
assert self.transfer_topo is not None
notified_req_ids: set[str] = set()
for notifs in self.nixl_wrapper.get_new_notifs().values():
for notif in notifs:
msg = notif.decode("utf-8")
# Handle heartbeat messages from D-side.
if msg.startswith("HB:"):
self._handle_heartbeat(msg[3:])
continue
req_id, tp_size = msg.rsplit(":", 1)
if (
req_id not in self._reqs_to_send
and req_id not in self._reqs_to_process
):
logger.error(
"Potentially invalid KV blocks for "
"unrecognized request %s were retrieved by "
"a decode worker. They may have expired.",
req_id,
)
continue
# NOTE: `tp_ratio` is the opposite when swapping local<>remote
n_consumers = int(tp_size)
tp_ratio = self.transfer_topo.tp_ratio(n_consumers)
# Number of reads *per producer* to wait for.
# When remote D TP > local P TP we expect `tp_ratio` reads.
consumers_per_producer = (
-tp_ratio if n_consumers > self.world_size else 1
)
self.consumer_notification_counts_by_req[req_id] += 1
# Wait all consumers (D) to be done reading before freeing.
if (
self.consumer_notification_counts_by_req[req_id]
== consumers_per_producer
):
notified_req_ids.add(req_id)
del self.consumer_notification_counts_by_req[req_id]
self._reqs_to_process.remove(req_id)
self._reqs_to_send.pop(req_id, None)
return notified_req_ids
@@ -0,0 +1,348 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Push-specific scheduler-side logic for the NIXL connector.
In push mode, scheduler-side responsibilities are:
* D side (decode): on ``update_state_after_alloc``, stash registration data
(D's identity + locally allocated block IDs) into
``_push_pending_registrations``. The D worker drains it from
``meta.push_registrations`` next step and sends a NIXL notification to the
P worker (no scheduler-level networking).
* P side (prefill): on ``request_finished``, stash the finished block IDs
into ``_finished_request_blocks`` for the lease, and into
``_newly_finished_push_blocks`` so the P worker picks them up via
``meta.push_finished_blocks`` and matches against any D registrations
it already received via NIXL notifications.
* Both sides: ``has_pending_push_work`` keeps the engine main loop stepping
while pushes are in flight. ``update_connector_output`` cleans up
``_finished_request_blocks`` once the WRITE completes.
A soft per-registration watchdog on the D scheduler fails requests that have
been registered but not fulfilled within a configurable timeout.
"""
from __future__ import annotations
import time
from typing import TYPE_CHECKING, Any
from vllm.distributed.kv_transfer.kv_connector.utils import BlockIds
from vllm.distributed.kv_transfer.kv_connector.v1.base import (
KVConnectorMetadata,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_scheduler import (
NixlBaseConnectorScheduler,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.metadata import (
NixlConnectorMetadata,
ReqId,
)
from vllm.logger import init_logger
if TYPE_CHECKING:
from vllm.config import VllmConfig
from vllm.v1.core.kv_cache_manager import KVCacheBlocks
from vllm.v1.core.sched.output import SchedulerOutput
from vllm.v1.kv_cache_interface import KVCacheConfig
from vllm.v1.outputs import KVConnectorOutput
from vllm.v1.request import Request
logger = init_logger(__name__)
class NixlPushConnectorScheduler(NixlBaseConnectorScheduler):
"""Push-specific scheduler logic (WRITE-based KV transfer).
All P2P communication is deferred to the worker level via NIXL
notifications. The scheduler communicates with workers only through
the standard ``build_connector_meta`` / ``update_connector_output``
hooks.
"""
def __init__(
self,
vllm_config: VllmConfig,
engine_id: str,
kv_cache_config: KVCacheConfig,
):
super().__init__(vllm_config, engine_id, kv_cache_config)
# D-side: registration data to pass to D workers via metadata on
# the next ``build_connector_meta`` call.
self._push_pending_registrations: dict[ReqId, dict[str, Any]] = {}
# D-side: track the wall-clock deadline for each registered request
# to detect "registered but never fulfilled" failures (e.g. the P
# node disappeared after registration). Keyed by D request_id.
self._push_registration_deadlines: dict[ReqId, float] = {}
# P-side: block IDs for finished requests, kept for the lease and
# used to drive ``has_pending_push_work``.
self._finished_request_blocks: dict[ReqId, BlockIds] = {}
# P-side: newly finished blocks to ship to P workers on next step.
self._newly_finished_push_blocks: dict[ReqId, BlockIds] = {}
# Soft watchdog timeout (seconds) for D-side registrations that
# never receive a push completion. Defaults to the existing
# decoder KV blocks TTL so behaviour matches the lease.
assert vllm_config.kv_transfer_config is not None
self._push_registration_timeout: float = float(
vllm_config.kv_transfer_config.get_from_extra_config(
"push_registration_timeout",
self.decoder_kv_blocks_ttl,
)
)
def get_num_new_matched_tokens(
self, request: Request, num_computed_tokens: int
) -> tuple[int, bool]:
"""In push mode, D doesn't pull — it registers blocks and waits.
However, we still need to handle the do_remote_prefill case where D
needs to know how many tokens will be pushed.
"""
params = request.kv_transfer_params
logger.debug(
"NixlPushConnector get_num_new_matched_tokens: "
"num_computed_tokens=%s, kv_transfer_params=%s",
num_computed_tokens,
params,
)
if params is not None and params.get("do_remote_prefill"):
token_ids = request.prompt_token_ids or []
actual = self._mamba_prefill_token_count(len(token_ids))
count = actual - num_computed_tokens
if count > 0:
return count, True
if params is not None and params.get("do_remote_decode") and self._has_mamba:
self._truncate_mamba_request_for_prefill(request)
return 0, False
def update_state_after_alloc(
self, request: Request, blocks: KVCacheBlocks, num_external_tokens: int
):
"""In push mode, D stores registration data for the worker to send
to P via NIXL notification (deferred to ``build_connector_meta``).
"""
params = request.kv_transfer_params
logger.debug(
"NixlPushConnector update_state_after_alloc: "
"num_external_tokens=%s, kv_transfer_params=%s",
num_external_tokens,
params,
)
if not params:
return
# P side: track the request as in-batch so the lease accounting
# matches what the worker expects on the next step.
if params.get("do_remote_decode"):
self._reqs_in_batch.add(request.request_id)
# P side with host-buffer offload: defer save to the worker.
if self.use_host_buffer and params.get("do_remote_decode"):
self._reqs_need_save[request.request_id] = request
return
# D side: only act on the first call (``do_remote_prefill`` is
# unset on re-entry by the marker below).
if not params.get("do_remote_prefill"):
return
if num_external_tokens <= 0:
# Nothing to receive: full prefix-cache hit on D, no
# registration to stage.
return
# First-pass D path: stash registration data the worker will
# ship to P on the next ``build_connector_meta`` cycle.
logger.debug(
"KV PUSH mode: D node storing registration for request %s",
request.request_id,
)
local_block_ids: BlockIds = blocks.get_unhashed_block_ids_all_groups()
local_block_ids = self.get_sw_clipped_blocks(local_block_ids)
# ``remote_*`` fields are P's coordinates (from D's perspective).
# ``decode_*`` fields are D's own info that P needs for the
# reverse handshake before WRITE-ing.
self._push_pending_registrations[request.request_id] = {
"request_id": request.request_id,
"decode_engine_id": self.engine_id,
"decode_host": self.side_channel_host,
"decode_port": self.side_channel_port,
"decode_tp_size": (self.vllm_config.parallel_config.tensor_parallel_size),
"local_block_ids": local_block_ids,
"remote_engine_id": params["remote_engine_id"],
"remote_host": params["remote_host"],
"remote_port": params["remote_port"],
"remote_tp_size": params["tp_size"],
}
self._push_registration_deadlines[request.request_id] = (
time.perf_counter() + self._push_registration_timeout
)
# In push mode D doesn't know P's blocks; P determines them
# from the registration. We still track the request as
# needing recv so the engine waits for P's WRITE completion.
# ``remote_block_ids`` is also seeded to an empty tuple so the
# base scheduler's ``add_new_req_to_recv`` can build the
# ReqMeta without a KeyError — the actual remote block IDs are
# learned by P over the NIXL handshake at WRITE time.
params["remote_block_ids"] = ()
self._reqs_need_recv[request.request_id] = (request, local_block_ids)
# Mark as processed so a re-entry (e.g. preemption + reschedule)
# doesn't re-stage the registration.
params["do_remote_prefill"] = False
def request_finished(
self,
request: Request,
block_ids: BlockIds,
) -> tuple[bool, dict[str, Any] | None]:
"""Push-mode request_finished: stores blocks for workers."""
from vllm.v1.request import RequestStatus
params = request.kv_transfer_params
logger.debug(
"NixlPushConnector request_finished(%s), request_status=%s, "
"kv_transfer_params=%s",
request.request_id,
request.status,
params,
)
if not params:
return False, None
is_p_node = bool(params.get("do_remote_decode"))
self._stop_heartbeat(request.request_id)
# Drop any pending registration deadline; the request either
# completed or was cancelled.
self._push_registration_deadlines.pop(request.request_id, None)
if params.get("do_remote_prefill"):
# ``do_remote_prefill`` is still set, which means
# ``update_state_after_alloc`` never ran (it would have
# flipped this flag to False). The request was aborted
# before it could be scheduled — e.g. rejected at the D
# serving layer via abort_immediately. To keep P from
# stranding the prefill blocks, we still register an empty
# recv so the worker emits a notif that lets P free them.
self._reqs_need_recv[request.request_id] = (request, [])
params["do_remote_prefill"] = False
return False, None
# Push connector only acts on the P-side terminal path; D-side
# finishing without a remote prefill is a no-op.
if not is_p_node:
return False, None
if request.status not in (
RequestStatus.FINISHED_LENGTH_CAPPED,
RequestStatus.FINISHED_STOPPED,
):
self._reqs_not_processed.add(request.request_id)
self._reqs_need_save.pop(request.request_id, None)
return False, None
delay_free_blocks = any(len(group) > 0 for group in block_ids)
remote_num_tokens = 0
if delay_free_blocks:
logger.debug(
"NixlPushConnector request_finished(%s) waiting for %d seconds "
"before releasing blocks",
request.request_id,
self._kv_lease_duration,
)
self._reqs_need_send[request.request_id] = (
time.perf_counter() + self._kv_lease_duration
)
block_ids = self.get_sw_clipped_blocks(block_ids)
remote_num_tokens = request.num_computed_tokens
# Store finished blocks for worker-level matching with D
# registrations (via NIXL notifications).
self._finished_request_blocks[request.request_id] = block_ids
self._newly_finished_push_blocks[request.request_id] = block_ids
return delay_free_blocks, dict(
do_remote_prefill=True,
do_remote_decode=False,
remote_block_ids=block_ids,
remote_engine_id=self.engine_id,
remote_request_id=request.request_id,
remote_host=self.side_channel_host,
remote_port=self.side_channel_port,
tp_size=self.vllm_config.parallel_config.tensor_parallel_size,
remote_num_tokens=remote_num_tokens,
)
def build_connector_meta(
self,
scheduler_output: SchedulerOutput,
) -> KVConnectorMetadata:
meta = super().build_connector_meta(scheduler_output)
assert isinstance(meta, NixlConnectorMetadata)
# Watchdog: any D-side registration whose deadline has passed without
# a corresponding push completion is treated as failed and cleaned up.
# The corresponding request is already tracked via _reqs_need_recv;
# the engine layer will eventually time it out via the lease, but we
# at least drop the stale registration so we don't keep retrying.
now = time.perf_counter()
# Deadlines are inserted in non-decreasing order (monotonic clock +
# constant timeout, armed once per request), and dict insertion order
# is preserved across key deletions, so we can stop at the first
# not-yet-expired entry instead of scanning the whole dict.
expired = []
for rid, deadline in self._push_registration_deadlines.items():
if deadline > now:
break
expired.append(rid)
for rid in expired:
self._push_registration_deadlines.pop(rid, None)
# Avoid resending a registration that already timed out.
self._push_pending_registrations.pop(rid, None)
logger.warning(
"NixlPushConnector: registration for request %s timed out "
"after %.1fs without a push completion",
rid,
self._push_registration_timeout,
)
# D side: package pending registrations for D workers to send out.
if self._push_pending_registrations:
meta.push_registrations = dict(self._push_pending_registrations)
self._push_pending_registrations.clear()
# P side: package newly finished blocks for P workers to match against
# any D registrations they have received via NIXL notifications.
if self._newly_finished_push_blocks:
meta.push_finished_blocks = dict(self._newly_finished_push_blocks)
self._newly_finished_push_blocks.clear()
return meta
def has_pending_push_work(self) -> bool:
# Keep the engine main loop alive while we have:
# - finished P blocks awaiting WRITE completion, or
# - pending D registrations the worker has not yet shipped, or
# - newly finished blocks not yet shipped to P workers.
return bool(self._finished_request_blocks or self._push_pending_registrations)
def update_connector_output(self, connector_output: KVConnectorOutput) -> None:
"""Clean up finished request blocks after push completes."""
super().update_connector_output(connector_output)
for req_id in connector_output.finished_sending or ():
self._finished_request_blocks.pop(req_id, None)
# On D side, finished_recving means the push completed; clear the
# watchdog so we don't trip an expiration on a fulfilled request.
for req_id in connector_output.finished_recving or ():
self._push_registration_deadlines.pop(req_id, None)
@@ -0,0 +1,742 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Push-specific (WRITE) worker-side logic for the NIXL connector.
A dedicated ``nixl-push-writer`` thread owns all push-related NIXL ops:
calls ``get_new_notifs`` (routing PUSH_REG internally; HB / completion
notifs are forwarded to the engine main thread), sends PUSH_REG via
``send_notif``, matches D registrations with P finished blocks, and
issues WRITE transfers via ``make_prepped_xfer`` / ``transfer``.
The engine main thread feeds the writer through three queues:
``_reg_send_inbox`` (D-side regs to send), ``_finished_blocks_inbox``
(P-side blocks from metadata) and ``_pending_completion_notifs``
(non-PUSH_REG notifs forwarded back for HB / completion accounting).
Wake model: the writer self-polls every
``_PUSH_WRITER_POLL_INTERVAL_MS`` only while it has unmatched
``_push_finished_blocks`` (i.e. P-side blocks waiting for a D PUSH_REG
notif that has no other wake source). All other progress is
event-driven: the engine main thread sets ``_push_writer_wake`` from
``start_load_kv`` (when handing it new work) and from ``get_finished``
(so each engine step gives the writer a chance to drain NIXL notifs);
the handshake-completion callback sets the same event after a deferred
PUSH_REG send has been queued. When a request's lease expires (the base
worker reports it via ``done_sending``) or the WRITE completes,
``get_finished`` enqueues an eviction onto ``_evict_finished_inbox`` so
the writer drops any leftover ``_push_finished_blocks`` /
``_pending_d_registrations`` and stops self-polling.
"""
import queue
import threading
import time
from collections import defaultdict
from concurrent.futures import Future
from typing import TYPE_CHECKING, Any
import msgspec
import numpy as np
from vllm.distributed.kv_transfer.kv_connector.utils import BlockIds
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.base_worker import (
NixlBaseConnectorWorker,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.metadata import (
PUSH_REG_NOTIF_PREFIX,
NixlConnectorMetadata,
RemoteMeta,
ReqId,
ReqMeta,
TransferHandle,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.tp_mapping import ReadSpec
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.utils import get_base_request_id
from vllm.logger import init_logger
if TYPE_CHECKING:
import torch
from vllm.config import VllmConfig
from vllm.v1.kv_cache_interface import KVCacheConfig
logger = init_logger(__name__)
# Writer-thread poll cadence while there is in-flight push state. When
# fully idle, the writer blocks on a wake event signalled by the engine
# main thread (start_load_kv / get_finished). Smaller -> lower latency
# while active, slightly more CPU.
_PUSH_WRITER_POLL_INTERVAL_MS = 1.0
class NixlPushConnectorWorker(NixlBaseConnectorWorker):
"""Push-specific (WRITE) worker logic. See module docstring."""
def __init__(
self,
vllm_config: "VllmConfig",
engine_id: str,
kv_cache_config: "KVCacheConfig",
):
super().__init__(vllm_config, engine_id, kv_cache_config)
# Push-specific state.
# P-side: outgoing WRITE handles awaiting completion, keyed by
# request_id. Mutated by writer (submit) and main thread
# (``_pop_done_transfers``); guarded by
# ``_sending_transfers_lock``.
self._sending_transfers = defaultdict[ReqId, list[TransferHandle]](list)
self._sending_transfers_lock = threading.Lock()
# Writer-thread owned matching state.
# P-side: finished request blocks received from scheduler metadata
# that have not yet been matched with an incoming D registration.
self._push_finished_blocks: dict[ReqId, BlockIds] = {}
# P-side: D registrations received via NIXL notification that have
# not yet been matched with a finished P request.
self._pending_d_registrations: dict[ReqId, dict[str, Any]] = {}
# Cross-thread channels.
self._reg_send_inbox: queue.Queue[tuple[str, dict[str, Any]]] = queue.Queue()
self._finished_blocks_inbox: queue.Queue[tuple[str, BlockIds]] = queue.Queue()
self._pending_completion_notifs: queue.Queue[bytes] = queue.Queue()
# Main thread → writer: req_ids whose lease has expired or whose
# WRITE has completed. Writer drops them from
# ``_push_finished_blocks`` so an unmatched entry doesn't keep the
# writer busy-polling forever.
self._evict_finished_inbox: queue.Queue[str] = queue.Queue()
# Wake signal from engine main thread (start_load_kv / get_finished).
# Writer self-polls at _PUSH_WRITER_POLL_INTERVAL_MS while it has
# active in-flight state; otherwise it blocks until signalled.
self._push_writer_wake = threading.Event()
self._push_writer_stop = threading.Event()
self._push_writer_thread: threading.Thread | None = None
# --- Lifecycle ----------------------------------------------------- #
def register_kv_caches(self, kv_caches: dict[str, "torch.Tensor"]):
super().register_kv_caches(kv_caches)
if self._push_writer_thread is None:
self._push_writer_thread = threading.Thread(
target=self._push_writer_loop,
daemon=True,
name="nixl-push-writer",
)
self._push_writer_thread.start()
logger.info("nixl-push-writer thread started (rank=%d)", self.tp_rank)
def shutdown(self):
self._push_writer_stop.set()
# Unblock the writer if it's waiting in the no-active-state branch.
self._push_writer_wake.set()
if self._push_writer_thread is not None:
self._push_writer_thread.join(timeout=2)
self._push_writer_thread = None
with self._sending_transfers_lock:
for handles in self._sending_transfers.values():
for handle in handles:
self.nixl_wrapper.release_xfer_handle(handle)
self._sending_transfers.clear()
super().shutdown()
# --- Engine-main-thread entry point -------------------------------- #
def start_load_kv(self, metadata: NixlConnectorMetadata):
"""Pre-process metadata; defer NIXL ops to the writer thread."""
# D-side: track reqs waiting for P to push.
for req_id, meta in metadata.reqs_to_recv.items():
meta.local_physical_block_ids = self._logical_to_kernel_block_ids(
meta.local_block_ids
)
assert meta.remote is not None
remote_engine_id = meta.remote.engine_id
logger.debug(
"start_load_kv (push) for request %s from remote engine %s. "
"Num local_block_ids: %s. Num remote_block_ids: %s. ",
req_id,
remote_engine_id,
len(meta.local_physical_block_ids),
len(meta.remote.block_ids),
)
self._recving_metadata[req_id] = meta
# --- D-side: registrations to send to P via NIXL ---
if metadata.push_registrations:
for req_id, reg_data in metadata.push_registrations.items():
self._reg_send_inbox.put((req_id, reg_data))
self._push_writer_wake.set()
# --- P-side: newly finished blocks awaiting a D registration match ---
if metadata.push_finished_blocks:
for req_id, block_ids in metadata.push_finished_blocks.items():
self._finished_blocks_inbox.put((req_id, block_ids))
self._push_writer_wake.set()
# Batch + lease tracking (same as pull).
for req_id in metadata.reqs_in_batch:
self._reqs_to_process.add(req_id)
for req_id in metadata.reqs_not_processed:
self._reqs_to_process.discard(req_id)
assert req_id not in self._reqs_to_send
for req_id, expiration_time in metadata.reqs_to_send.items():
if req_id in self._reqs_to_process:
self._reqs_to_send[req_id] = expiration_time
# Heartbeats still leave from the main thread (base worker behaviour).
self._send_heartbeats(metadata)
# --- Writer thread ------------------------------------------------- #
def _push_writer_loop(self) -> None:
sleep_s = _PUSH_WRITER_POLL_INTERVAL_MS / 1000.0
while not self._push_writer_stop.is_set():
try:
# 1. D registrations to send.
while True:
try:
rid, rd = self._reg_send_inbox.get_nowait()
except queue.Empty:
break
self._send_registration_to_p(rid, rd)
# 2. P-side finished blocks; match against pending regs.
while True:
try:
rid, blocks = self._finished_blocks_inbox.get_nowait()
except queue.Empty:
break
matched = self._pop_matching_registration(rid)
if matched is not None:
self._do_start_push_kv(rid, blocks, matched)
else:
self._push_finished_blocks[rid] = blocks
# 2b. Evict finished blocks for requests that have either
# completed (WRITE acknowledged) or whose lease expired
# without a D registration. Drop pending registrations
# for the same reason so we don't leak state.
while True:
try:
rid = self._evict_finished_inbox.get_nowait()
except queue.Empty:
break
self._push_finished_blocks.pop(rid, None)
self._pending_d_registrations.pop(rid, None)
# 3. NIXL notifs: route PUSH_REG; forward the rest.
for notifs in self.nixl_wrapper.get_new_notifs().values():
for notif in notifs:
if notif.startswith(PUSH_REG_NOTIF_PREFIX):
self._handle_push_reg_notif(notif)
else:
self._pending_completion_notifs.put(notif)
except Exception:
logger.exception("nixl-push-writer error; continuing")
# Self-poll only while there is no other wake source: P-side
# finished blocks waiting for a D PUSH_REG match. All other
# progress is event-driven (see module docstring).
if self._push_finished_blocks:
self._push_writer_stop.wait(timeout=sleep_s)
else:
self._push_writer_wake.wait()
self._push_writer_wake.clear()
def _handle_push_reg_notif(self, notif: bytes) -> None:
try:
reg_data = msgspec.msgpack.decode(notif[len(PUSH_REG_NOTIF_PREFIX) :])
except Exception:
logger.exception("Failed to decode PUSH_REG notification payload")
return
rid = reg_data.get("request_id") if isinstance(reg_data, dict) else None
if not isinstance(rid, str):
logger.warning("PUSH_REG notif missing request_id; dropping")
return
match = self._pop_matching_finished_blocks(rid)
if match is not None:
fin_id, blocks = match
self._do_start_push_kv(fin_id, blocks, reg_data)
else:
self._pending_d_registrations[rid] = reg_data
# --- D-side registration send (writer thread) ---------------------- #
def _send_registration_to_p(
self,
req_id: str,
reg_data: dict[str, Any],
) -> None:
"""Handshake (if needed) then send PUSH_REG. ``send_notif`` always
executes on the writer; the handshake runs on the background executor
and the request is re-queued onto ``_reg_send_inbox`` once it
completes (at which point ``_ensure_handshake`` returns ``None`` and we
send directly)."""
fut = self._ensure_handshake(
reg_data["remote_engine_id"],
reg_data["remote_host"],
reg_data["remote_port"],
reg_data["remote_tp_size"],
)
if fut is None:
self._do_send_reg_notif(req_id, reg_data)
return
def _on_handshake(
f: Future[dict[int, str]],
rid: str = req_id,
rd: dict[str, Any] = reg_data,
) -> None:
try:
f.result()
except Exception as e:
self._log_failure(
failure_type="push_reg_handshake_failed", req_id=rid, error=e
)
self._handle_failed_transfer(rid, None)
return
# Re-queue for the writer to send now that the handshake is done.
self._reg_send_inbox.put((rid, rd))
# Wake the writer so it sends the PUSH_REG promptly even if
# otherwise parked.
self._push_writer_wake.set()
fut.add_done_callback(_on_handshake)
def _do_send_reg_notif(self, req_id: str, reg_data: dict[str, Any]) -> None:
engine_id = reg_data["remote_engine_id"]
notif_msg = PUSH_REG_NOTIF_PREFIX + msgspec.msgpack.encode(reg_data)
agents = self._remote_agents.get(engine_id)
if not agents:
logger.error(
"No remote agents for engine %s; cannot send registration for %s",
engine_id,
req_id,
)
self._handle_failed_transfer(req_id, None)
return
for rank, agent_name in agents.items():
try:
self.nixl_wrapper.send_notif(agent_name, notif_msg=notif_msg)
except Exception as e:
self._log_failure(
failure_type="push_reg_notif_failed",
req_id=req_id,
error=e,
remote_rank=rank,
)
logger.debug(
"Sent PUSH_REG for %s to engine %s (%dB)", req_id, engine_id, len(notif_msg)
)
# --- Matching helpers --------------------------------------------- #
def _pop_matching_registration(self, request_id: str) -> dict[str, Any] | None:
"""Pop the D-side registration matching *request_id*.
Exact key first, then a match after stripping the random suffix from
both sides. No match leaves the request unmatched (push not started).
"""
data = self._pending_d_registrations.pop(request_id, None)
if data is not None:
return data
base_id = get_base_request_id(request_id)
for reg_id in list(self._pending_d_registrations):
if get_base_request_id(reg_id) == base_id:
return self._pending_d_registrations.pop(reg_id)
return None
def _pop_matching_finished_blocks(
self, request_id: str
) -> tuple[str, BlockIds] | None:
"""Pop the P-side finished blocks matching *request_id*.
Same lookup as ``_pop_matching_registration``: exact key, then a
match after stripping the random suffix from both sides.
"""
blocks = self._push_finished_blocks.pop(request_id, None)
if blocks is not None:
return request_id, blocks
base_id = get_base_request_id(request_id)
for fin_id in list(self._push_finished_blocks):
if get_base_request_id(fin_id) == base_id:
return fin_id, self._push_finished_blocks.pop(fin_id)
return None
# --- WRITE transfer logic (writer thread) ------------------------- #
def _do_start_push_kv(
self,
request_id: str,
local_block_ids: BlockIds,
registration_data: dict[str, Any],
) -> None:
"""Start push-based KV transfer from P worker to D node.
``local_block_ids`` are P's *logical* block IDs (from the P
scheduler's metadata). ``registration_data["local_block_ids"]``
are D's *logical* block IDs (from D's scheduler, sent over the
PUSH_REG notif). All conversion to physical block IDs is
deferred to ``_xfer_blocks_for_req`` so each side uses its own
physical-blocks-per-logical ratio (P uses
``self._physical_blocks_per_logical_kv_block``; D's ratio is
learned during the NIXL handshake)."""
decode_engine_id = registration_data["decode_engine_id"]
remote_block_ids = registration_data["local_block_ids"]
decode_host = registration_data["decode_host"]
decode_port = registration_data["decode_port"]
decode_request_id = registration_data["request_id"]
if not local_block_ids:
logger.warning("No local blocks to push for request %s", request_id)
return
if not self._ensure_d_handshake(
decode_engine_id,
decode_host,
decode_port,
registration_data["decode_tp_size"],
request_id,
):
return
# Both sides are kept in logical form here; ``_xfer_blocks_for_req``
# expands each side using the appropriate ratio.
logical_local = self._as_grouped_block_ids(local_block_ids)
logical_remote = self._as_grouped_block_ids(remote_block_ids)
physical_local = self._logical_to_kernel_block_ids(logical_local)
push_meta = ReqMeta(
local_block_ids=logical_local,
local_physical_block_ids=physical_local,
tp_size=self.world_size,
remote=RemoteMeta(
block_ids=logical_remote,
host="",
port=0,
engine_id=decode_engine_id,
request_id=decode_request_id,
),
)
t0 = time.perf_counter()
self._xfer_blocks_for_req(req_id=request_id, meta=push_meta)
elapsed_ms = (time.perf_counter() - t0) * 1000.0
if elapsed_ms > 200.0:
logger.warning(
"_do_start_push_kv for %s took %.1fms (slow NIXL submission)",
request_id,
elapsed_ms,
)
def _ensure_d_handshake(
self,
decode_engine_id: str,
decode_host: str,
decode_port: int,
decode_tp_size: int,
request_id: str,
) -> bool:
"""First-time P→D handshake. Blocking call on the writer thread.
Returns True iff the handshake succeeded (or had already been
completed). Returns False if the handshake raised; the request is
skipped in that case (the engine layer will reschedule or fail it
via the standard lease/timeout path)."""
if decode_engine_id in self._remote_agents:
return True
try:
remote_agents = self._nixl_handshake(
decode_host,
decode_port,
decode_tp_size,
decode_engine_id,
)
except Exception:
logger.exception(
"Failed handshake to D %s for push %s",
decode_engine_id,
request_id,
)
return False
with self._handshake_lock:
self._remote_agents[decode_engine_id] = remote_agents
logger.info(
"Push handshake to D %s done (%d agents)",
decode_engine_id,
len(remote_agents),
)
return True
@staticmethod
def _as_grouped_block_ids(block_ids: BlockIds) -> BlockIds:
"""Normalise a sequence of block IDs to a tuple-of-groups shape.
``BlockIds`` is canonically a tuple of per-group lists, but some
registration payloads collapse a single-group case to a flat
list. Re-wrap that case so downstream group-aware helpers see a
consistent shape."""
if block_ids and not isinstance(block_ids[0], (list, tuple)):
return (list(block_ids),)
return block_ids
def _xfer_blocks_for_req(self, req_id: str, meta: ReqMeta):
"""Issue WRITE transfers to one or more remote TP ranks."""
assert meta.remote is not None and self.transfer_topo is not None
engine_id = meta.remote.engine_id
plan = self.tp_mappings[engine_id]
remote_info = self.transfer_topo.get_engine_info(engine_id)
tp_ratio = self.transfer_topo.tp_ratio(remote_info.remote_tp_size)
# Expand D's logical IDs using the ratio learned during the
# NIXL handshake. ``meta`` is freshly built by
# ``_do_start_push_kv`` so mutating it here is safe.
meta.remote.block_ids = self._logical_to_remote_kernel_block_ids(
meta.remote.block_ids,
remote_info.remote_physical_blocks_per_logical,
)
remote_block_ids = meta.remote.block_ids
local_block_ids = meta.local_physical_block_ids
num_groups = len(local_block_ids)
read_specs = [
ReadSpec(
remote_rank=rank,
local_block_ids=[
list(local_block_ids[g])
if rank in plan.source_ranks_per_group[g]
else []
for g in range(num_groups)
],
remote_block_ids=[
list(remote_block_ids[g])
if rank in plan.source_ranks_per_group[g]
else []
for g in range(num_groups)
],
)
for rank in plan.all_source_ranks
]
if self.use_mla and tp_ratio < 0:
assert len(read_specs) == 1
for i, spec in enumerate(read_specs):
remote_block_size = remote_info.remote_block_size
logger.debug(
"Remote agent %s available, calling _xfer_blocks"
" on remote rank %s with remote block size %s for req %s",
meta.remote.engine_id,
spec.remote_rank,
remote_block_size,
req_id,
)
if tp_ratio < 0 and not self.use_mla:
assert remote_block_size == self.block_size
local_xfer_side_handle = self.src_xfer_handles_by_tp_ratio[tp_ratio][i]
else:
local_xfer_side_handle = self.src_xfer_handles_by_block_size[
remote_block_size
]
remote_xfer_side_handle = self.dst_xfer_side_handles[meta.remote.engine_id][
spec.remote_rank
]
self._xfer_blocks(
read_spec=spec,
request_id=req_id,
dst_engine_id=meta.remote.engine_id,
remote_request_id=meta.remote.request_id,
local_xfer_side_handle=local_xfer_side_handle,
remote_xfer_side_handle=remote_xfer_side_handle,
)
if self.use_mla and tp_ratio < 0 and read_specs:
notif_id = f"{meta.remote.request_id}:{self.world_size}".encode()
remote_agents = self._remote_agents[meta.remote.engine_id]
for rank_to_notify, agent in remote_agents.items():
if rank_to_notify != read_specs[0].remote_rank:
self.nixl_wrapper.send_notif(agent, notif_msg=notif_id)
def _xfer_blocks(
self,
read_spec: ReadSpec,
dst_engine_id: str,
request_id: str,
remote_request_id: str,
local_xfer_side_handle: int,
remote_xfer_side_handle: int,
):
"""Post a WRITE point-to-point xfer request."""
assert self.transfer_topo is not None
remote_rank = read_spec.remote_rank
local_block_ids = read_spec.local_block_ids
remote_block_ids = read_spec.remote_block_ids
remote_info = self.transfer_topo.get_engine_info(dst_engine_id)
block_size_ratio = self.transfer_topo.block_size_ratio(
remote_info.remote_block_size
)
if block_size_ratio > 1:
assert not self._is_hma_required
local_block_ids0 = local_block_ids[0] if local_block_ids else []
remote_block_ids0 = remote_block_ids[0]
local_block_ids_mapped = self.get_mapped_blocks(
np.asarray(local_block_ids0), block_size_ratio
).tolist()
if len(local_block_ids_mapped) > len(remote_block_ids0):
local_block_ids_mapped = local_block_ids_mapped[
: len(remote_block_ids0)
]
local_block_ids = [local_block_ids_mapped] if local_block_ids_mapped else []
remote_block_ids = [remote_block_ids0]
notif_id = f"{remote_request_id}:{self.world_size}".encode()
if len(local_block_ids) == 0:
logger.warning("No blocks to push for request %s", request_id)
return
# Align per-group block counts for push.
local_block_ids = list(local_block_ids)
remote_block_ids = list(remote_block_ids)
for i in range(min(len(local_block_ids), len(remote_block_ids))):
num_local = len(local_block_ids[i])
num_remote = len(remote_block_ids[i])
if num_local > num_remote:
local_block_ids[i] = local_block_ids[i][:num_remote]
elif num_local < num_remote:
remote_block_ids[i] = remote_block_ids[i][:num_local]
# Get descs ids.
remote_block_descs_ids = self._compute_desc_ids(
block_ids=remote_block_ids,
dst_num_blocks=self.dst_num_blocks[dst_engine_id],
block_size_ratio=None,
physical_blocks_per_logical=remote_info.remote_physical_blocks_per_logical,
)
local_block_descs_ids = self._compute_desc_ids(
block_ids=local_block_ids,
dst_num_blocks=self.dst_num_blocks[self.engine_id],
block_size_ratio=block_size_ratio,
physical_blocks_per_logical=self._physical_blocks_per_logical_kv_block,
)
assert len(local_block_descs_ids) == len(remote_block_descs_ids)
handle = None
try:
handle = self.nixl_wrapper.make_prepped_xfer(
"WRITE",
local_xfer_side_handle,
local_block_descs_ids,
remote_xfer_side_handle,
remote_block_descs_ids,
notif_msg=notif_id,
)
self.nixl_wrapper.transfer(handle)
# Track push WRITE handles so P can free blocks once done.
with self._sending_transfers_lock:
self._sending_transfers[request_id].append(handle)
except Exception as e:
self._log_failure(
failure_type="transfer_setup_failed",
req_id=request_id,
msg="Push WRITE submission failed; releasing handle",
error=e,
dst_engine_id=dst_engine_id,
remote_rank=remote_rank,
)
# On the P side this WRITE failure is purely outbound; we
# don't have a ``_recving_metadata`` entry to invalidate, so
# we just release the handle and let the engine reschedule
# via the lease / watchdog.
if handle is not None:
self.nixl_wrapper.release_xfer_handle(handle)
self.xfer_stats.record_failed_transfer()
# --- Notification handling on engine main thread ------------------ #
def _get_new_notifs(self) -> set[str]:
"""Drain HB / completion notifs forwarded by the writer thread.
The writer owns ``nixl_wrapper.get_new_notifs`` for push; PUSH_REG
notifs are handled there. Everything else is forwarded here for
existing accounting.
"""
assert self.transfer_topo is not None
notified_req_ids: set[str] = set()
while True:
try:
notif = self._pending_completion_notifs.get_nowait()
except queue.Empty:
break
msg = notif.decode("utf-8")
if msg.startswith("HB:"):
self._handle_heartbeat(msg[3:])
continue
req_id, tp_size = msg.rsplit(":", 1)
# Not tracked as a P-side send/process for this notif.
if req_id not in self._reqs_to_send and req_id not in self._reqs_to_process:
if req_id in self._recving_metadata:
# D-side: P signalled push completion. The transfer was
# driven entirely by P (we don't own a NIXL handle here),
# so materialise an empty entry in ``_recving_transfers``
# and let ``_pop_done_transfers`` report it done on the
# next ``get_finished``.
self._recving_transfers.setdefault(req_id, [])
else:
# Not tracked on either side (lease may have expired
# before the notif arrived). Log and skip.
logger.error(
"Unrecognized request %s notif (may have expired).",
req_id,
)
continue
n_consumers = int(tp_size)
tp_ratio = self.transfer_topo.tp_ratio(n_consumers)
consumers_per_producer = -tp_ratio if n_consumers > self.world_size else 1
self.consumer_notification_counts_by_req[req_id] += 1
if (
self.consumer_notification_counts_by_req[req_id]
== consumers_per_producer
):
notified_req_ids.add(req_id)
del self.consumer_notification_counts_by_req[req_id]
self._reqs_to_process.remove(req_id)
self._reqs_to_send.pop(req_id, None)
return notified_req_ids
def get_finished(self) -> tuple[set[str], set[str]]:
# Engine main thread asking for completions: also wake the writer
# so it gets a chance to drain NIXL notifs (heartbeats, completion
# notifs, late PUSH_REGs) even if it had been parked.
self._push_writer_wake.set()
done_sending, done_recving = super().get_finished()
# ``_pop_done_transfers`` mutates ``_sending_transfers``; the
# writer thread also appends to it, so guard the pop.
with self._sending_transfers_lock:
done_pushing = self._pop_done_transfers(self._sending_transfers)
for req_id in done_pushing:
self._reqs_to_send.pop(req_id, None)
self._reqs_to_process.discard(req_id)
self.consumer_notification_counts_by_req.pop(req_id, None)
done_sending.add(req_id)
# Tell the writer to drop any state it still holds for any
# request that just finished (push completed) or expired
# (lease ran out without a D registration ever arriving).
for req_id in done_sending:
self._evict_finished_inbox.put(req_id)
if done_sending:
self._push_writer_wake.set()
return done_sending, done_recving
@@ -1,674 +1,12 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Scheduler-side logic for the NIXL connector."""
"""Backward-compatible re-export of NixlPullConnectorScheduler."""
import threading
import time
from typing import TYPE_CHECKING, Any
import msgspec
import zmq
from vllm import envs
from vllm.distributed.kv_transfer.kv_connector.utils import (
BlockIds,
EngineId,
yield_req_data,
)
from vllm.distributed.kv_transfer.kv_connector.v1.base import (
KVConnectorHandshakeMetadata,
KVConnectorMetadata,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.metadata import (
GET_META_MSG,
HeartbeatInfo,
NixlConnectorMetadata,
NixlHandshakePayload,
ReqId,
)
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.utils import zmq_ctx
from vllm.logger import init_logger
from vllm.platforms import current_platform
from vllm.utils.math_utils import cdiv
from vllm.utils.network_utils import make_zmq_path
from vllm.v1.core.sched.output import SchedulerOutput
from vllm.v1.kv_cache_interface import (
FullAttentionSpec,
MambaSpec,
SlidingWindowSpec,
from vllm.distributed.kv_transfer.kv_connector.v1.nixl.pull_scheduler import (
NixlPullConnectorScheduler,
)
if TYPE_CHECKING:
from vllm.config import VllmConfig
from vllm.v1.core.kv_cache_manager import KVCacheBlocks
from vllm.v1.kv_cache_interface import KVCacheConfig
from vllm.v1.outputs import KVConnectorOutput
from vllm.v1.request import Request
# Backward compatibility: NixlConnectorScheduler is the pull-based scheduler.
NixlConnectorScheduler = NixlPullConnectorScheduler
logger = init_logger(__name__)
class NixlConnectorScheduler:
"""Implementation of Scheduler side methods"""
def __init__(
self,
vllm_config: "VllmConfig",
engine_id: str,
kv_cache_config: "KVCacheConfig",
):
self.vllm_config = vllm_config
self.block_size = vllm_config.cache_config.block_size
self.engine_id: EngineId = engine_id
self.kv_cache_config = kv_cache_config
self.side_channel_host = envs.VLLM_NIXL_SIDE_CHANNEL_HOST
self.side_channel_port = (
envs.VLLM_NIXL_SIDE_CHANNEL_PORT
+ vllm_config.parallel_config.data_parallel_index
)
assert vllm_config.kv_transfer_config is not None
self._kv_lease_duration: int = (
vllm_config.kv_transfer_config.get_from_extra_config(
"kv_lease_duration", 30
)
)
# NOTE (NickLucche): For now we use a hardcoded value for a simpler interface.
self._heartbeat_interval = self._kv_lease_duration // 6
if current_platform.device_type == "cpu":
self.use_host_buffer = False
else:
self.use_host_buffer = (
vllm_config.kv_transfer_config.kv_buffer_device == "cpu"
)
self._is_hma_required = (
not vllm_config.scheduler_config.disable_hybrid_kv_cache_manager
# Also handle unlikely SW-only model case instead of checking num_groups>1.
and any(
not isinstance(g.kv_cache_spec, FullAttentionSpec)
for g in kv_cache_config.kv_cache_groups
)
)
self._has_mamba = any(
isinstance(g.kv_cache_spec, MambaSpec)
for g in kv_cache_config.kv_cache_groups
)
logger.info("Initializing NIXL Scheduler %s", engine_id)
if vllm_config.scheduler_config.disable_hybrid_kv_cache_manager:
logger.info("Hybrid Memory Allocator is enabled with NIXL")
# Background thread for handling new handshake requests.
self._nixl_handshake_listener_t: threading.Thread | None = None
self._stop_event = threading.Event()
# Requests that need to start recv/send.
# New requests are added by update_state_after_alloc in
# the scheduler. Used to make metadata passed to Worker.
self._reqs_need_recv: dict[ReqId, tuple[Request, BlockIds]] = {}
self._reqs_need_save: dict[ReqId, Request] = {}
# Reqs to send and their expiration time
self._reqs_need_send: dict[ReqId, float] = {}
self._reqs_in_batch: set[ReqId] = set()
# Reqs to remove from processed set because they're not to send after
# remote prefill or aborted.
self._reqs_not_processed: set[ReqId] = set()
# Heartbeat tracking: requests needing periodic lease-renewal heartbeats to
# remote P-side, stored as ready-to-send HeartbeatInfo grouped by remote engine
self._heartbeat_by_engine: dict[EngineId, HeartbeatInfo] = {}
# Reverse lookup: local req_id -> (engine_id, remote_req_id) for O(1) removal
self._heartbeat_req_engine: dict[ReqId, tuple[EngineId, ReqId]] = {}
self._last_heartbeat_time: float = 0.0
# Gather Sliding Window sizes for each kv cache group (if any) in number of
# blocks per KV cache group. This is used to clip the local attention window.
sw_sizes_tokens: list[tuple[int, int]] = [
(g.kv_cache_spec.sliding_window, g.kv_cache_spec.block_size)
if isinstance(g.kv_cache_spec, SlidingWindowSpec)
else (0, self.block_size)
for g in kv_cache_config.kv_cache_groups
]
# cdiv(n_tokens, block_size) gives blocks/window; add 1 to conservatively
# account for boundary overlap eg window isn't fully aligned with blocks.
self.blocks_per_sw = [
cdiv(n_tokens, block_size) + 1 if n_tokens else 0
for n_tokens, block_size in sw_sizes_tokens
]
# Threshold to decide whether to compute kv cache locally
# or pull from a remote node: minimum number of remote
# tokens to amortize the xfer latencies
self.kv_recompute_threshold: int = int(
vllm_config.kv_transfer_config.get_from_extra_config(
"kv_recompute_threshold", 64
)
)
# Bi-directional KV transfer feature supports KV block
# transfers from D node to P node
self.is_bidirectional_kv_xfer_enabled = (
vllm_config.kv_transfer_config.get_from_extra_config(
"bidirectional_kv_xfer", False
)
)
self.decoder_kv_blocks_ttl = (
vllm_config.kv_transfer_config.get_from_extra_config(
"decoder_kv_blocks_ttl", 480
)
)
if self.is_bidirectional_kv_xfer_enabled and self.kv_recompute_threshold > 0:
logger.info(
"Bidirectional KV transfer is enabled and the kv "
"recompute threshold is set to %d tokens."
"KV blocks on D are released after a TTL of %d seconds.",
self.kv_recompute_threshold,
self.decoder_kv_blocks_ttl,
)
def shutdown(self):
self._stop_event.set()
if self._nixl_handshake_listener_t is not None:
self._nixl_handshake_listener_t.join()
self._nixl_handshake_listener_t = None
def on_new_request(self, request: "Request") -> None:
"""Track a request that may need heartbeats."""
params = request.kv_transfer_params
# NOTE (NickLucche) This excludes request meant for P, ie heartbeats are
# effectively disabled for Bidirectional KV transfer.
if params is None or not params.get("do_remote_prefill"):
return
# Only track if all required remote fields are present.
remote_engine_id = params.get("remote_engine_id")
remote_request_id = params.get("remote_request_id")
host = params.get("remote_host")
port = params.get("remote_port")
tp_size = params.get("tp_size")
if (
remote_engine_id is None
or remote_request_id is None
or host is None
or port is None
or tp_size is None
):
return
if remote_engine_id not in self._heartbeat_by_engine:
self._heartbeat_by_engine[remote_engine_id] = HeartbeatInfo(
req_ids=set(),
host=host,
port=port,
tp_size=tp_size,
)
self._heartbeat_by_engine[remote_engine_id].req_ids.add(remote_request_id)
self._heartbeat_req_engine[request.request_id] = (
remote_engine_id,
remote_request_id,
)
def _stop_heartbeat(self, req_id: ReqId) -> None:
"""Remove *req_id* from heartbeat tracking (if tracked)."""
if key := self._heartbeat_req_engine.pop(req_id, None):
engine_id, remote_id = key
if info := self._heartbeat_by_engine.get(engine_id):
info.req_ids.discard(remote_id)
if not info.req_ids:
# Clean up empty engines so we don't leak a key when remote dies.
del self._heartbeat_by_engine[engine_id]
def get_sw_clipped_blocks(self, block_ids: BlockIds) -> BlockIds:
"""
Clip the number of blocks to the sliding window size for each kv cache group
that employs SWA.
This is necessary because the KV Cache manager initially allocates blocks for
the entire sequence length, and successively cleans up blocks that are outside
the window prior to the `request_finished_all_groups` hook.
"""
if len(block_ids) == 0 or not self._is_hma_required:
# No blocks to clip eg Full prefix cache hit or not a hybrid model.
return block_ids
# NOTE (NickLucche) This logic is currently handled at the connector level
# because offloading connectors might want to receive the whole sequence even
# for SWA groups. We will abstract this logic once the interface is more stable
assert len(block_ids) == len(self.blocks_per_sw), (
"Number of KV cache groups must match"
)
# For non-SWA groups, blocks_per_sw is 0 so we return all block_ids unchanged
return tuple(
[
blocks[-self.blocks_per_sw[i] :]
if self.blocks_per_sw[i] > 0
else blocks
for i, blocks in enumerate(block_ids)
]
)
def set_xfer_handshake_metadata(
self, metadata: dict[int, KVConnectorHandshakeMetadata]
) -> None:
"""
Set the KV connector handshake metadata for this connector.
Args:
metadata (dict): the handshake metadata to set.
"""
encoded_data: dict[int, bytes] = {}
encoder = msgspec.msgpack.Encoder()
for tp_rank, rank_metadata in metadata.items():
if not isinstance(rank_metadata, NixlHandshakePayload):
raise ValueError(
"NixlConnectorScheduler expects NixlHandshakePayload for "
"handshake metadata."
)
encoded_data[tp_rank] = encoder.encode(rank_metadata)
logger.debug(
"Tp rank %d: encoded NixlHandshakePayload size: %s bytes",
tp_rank,
str(len(encoded_data[tp_rank])),
)
# Only start the listener when we have metadata to serve.
if self._nixl_handshake_listener_t is None:
ready_event = threading.Event()
self._nixl_handshake_listener_t = threading.Thread(
target=self._nixl_handshake_listener,
args=(
encoded_data,
ready_event,
self._stop_event,
self.side_channel_host,
self.side_channel_port,
),
daemon=True,
name="nixl_handshake_listener",
)
self._nixl_handshake_listener_t.start()
ready_event.wait() # Wait for listener ZMQ socket to be ready.
@staticmethod
def _nixl_handshake_listener(
encoded_data: dict[int, Any],
ready_event: threading.Event,
stop_event: threading.Event,
host: str,
port: int,
):
"""Background thread for getting new NIXL handshakes."""
# NOTE(rob): this is a simple implementation. We will move
# to a better approach via HTTP endpoint soon.
# Listen for new requests for metadata.
path = make_zmq_path("tcp", host, port)
logger.debug("Starting listening on path: %s", path)
with zmq_ctx(zmq.ROUTER, path) as sock:
sock.setsockopt(zmq.RCVTIMEO, 1000)
ready_event.set()
while True:
try:
identity, _, msg = sock.recv_multipart()
except zmq.Again:
if stop_event.is_set():
break
continue
# Decode the message which contains (GET_META_MSG, rank)
msg, target_tp_rank = msgspec.msgpack.decode(msg)
logger.debug(
"Received message for tp rank %s",
target_tp_rank,
)
if msg != GET_META_MSG:
logger.warning("Connection listener got unexpected message %s", msg)
sock.send_multipart((identity, b"", encoded_data[target_tp_rank]))
def _mamba_prefill_token_count(self, num_prompt_tokens: int) -> int:
"""D-side only. Returns N-1 for Mamba models since the decoder
always recomputes the last token and must start from h(N-1)."""
if self._has_mamba and num_prompt_tokens > 1:
return num_prompt_tokens - 1
return num_prompt_tokens
def _truncate_mamba_request_for_prefill(self, request: "Request") -> None:
"""P-side only: drop the last prompt token so the prefiller computes
h(N-1) instead of h(N). The decoder recomputes the last token to
derive h(N) correctly.
Guarded by ``_p_side_truncated`` to avoid repeated truncation if the
request is preempted and rescheduled."""
params = request.kv_transfer_params
if (
params is not None
# Guard against repeated truncation after preemption/reschedule.
and not params.get("_p_side_truncated")
and request.num_prompt_tokens > 1
):
if request.prompt_token_ids is not None:
request.prompt_token_ids.pop()
elif request.prompt_embeds is not None:
request.prompt_embeds = request.prompt_embeds[:-1]
else:
return
request._all_token_ids.pop()
request.num_prompt_tokens -= 1
request.max_tokens = 1
params["_p_side_truncated"] = True
def get_num_new_matched_tokens(
self, request: "Request", num_computed_tokens: int
) -> tuple[int, bool]:
"""
For remote prefill, pull all prompt blocks from remote
asynchronously relative to engine execution.
Args:
request (Request): the request object.
num_computed_tokens (int): the number of locally
computed tokens for this request
Returns:
* the number of tokens that can be loaded from the
external KV cache beyond what is already computed.
* true if the external KV cache tokens will be loaded
asynchronously (between scheduler steps).
"""
params = request.kv_transfer_params
logger.debug(
"NIXLConnector get_num_new_matched_tokens: "
"num_computed_tokens=%s, kv_transfer_params=%s",
num_computed_tokens,
params,
)
if params is not None and params.get("do_remote_prefill"):
# Remote prefill: get all prompt blocks from remote.
token_ids = request.prompt_token_ids or []
actual = self._mamba_prefill_token_count(len(token_ids))
count = actual - num_computed_tokens
if count > 0:
return count, True
if params is not None and params.get("do_remote_decode") and self._has_mamba:
self._truncate_mamba_request_for_prefill(request)
if (
params is not None
and params.get("do_remote_decode")
and params.get("remote_block_ids")
and all(
p in params
for p in (
"remote_engine_id",
"remote_request_id",
"remote_host",
"remote_port",
)
)
):
# Decode node has kv blocks for part of prefill request, so, provide them
# as an external token count to scheduler.
# The tokens will be loaded if not already present
# in the prefill node local cache
remote_num_tokens = params.get("remote_num_tokens") or 0
count = (
min(remote_num_tokens, request.num_prompt_tokens) - num_computed_tokens
)
if count > 0:
# Check kv_recompute_threshold: skip pull if
# remote tokens are below the threshold.
if (
self.kv_recompute_threshold > 0
and count < self.kv_recompute_threshold
):
logger.debug(
"Skipping remote pull for %s: %d remote tokens < threshold %d",
request.request_id,
count,
self.kv_recompute_threshold,
)
return 0, False
return count, True
# No remote prefill for this request.
return 0, False
def update_state_after_alloc(
self, request: "Request", blocks: "KVCacheBlocks", num_external_tokens: int
):
params = request.kv_transfer_params
logger.debug(
"NIXLConnector update_state_after_alloc: "
"num_external_tokens=%s, kv_transfer_params=%s",
num_external_tokens,
params,
)
if not params:
return
if params.get("do_remote_decode") or (
params.get("do_remote_prefill") and self.is_bidirectional_kv_xfer_enabled
):
self._reqs_in_batch.add(request.request_id)
if self.use_host_buffer and params.get("do_remote_decode"):
# NOTE: when accelerator is not directly supported by Nixl,
# prefilled blocks need to be saved to host memory before transfer.
self._reqs_need_save[request.request_id] = request
elif params.get("do_remote_prefill") or (
params.get("do_remote_decode")
and self.is_bidirectional_kv_xfer_enabled
and not params.get("_remote_blocks_processed")
):
if params.get("remote_block_ids"):
if all(
p in params
for p in (
"remote_engine_id",
"remote_request_id",
"remote_host",
"remote_port",
)
):
# If remote_blocks and num_external_tokens = 0, we have
# a full prefix cache hit on the local node. We need to call
# send_notif in _read_blocks to free the memory on the remote node.
unhashed_local_block_ids: BlockIds = (
blocks.get_unhashed_block_ids_all_groups()
if num_external_tokens > 0
else ()
)
local_block_ids = self.get_sw_clipped_blocks(
unhashed_local_block_ids
)
# Get unhashed blocks to pull from remote. Mind that a full prefix
# cache hit is indicated with an empty list.
self._reqs_need_recv[request.request_id] = (
request,
local_block_ids,
)
else:
logger.warning(
"Got invalid KVTransferParams: %s. This "
"request will not utilize KVTransfer",
params,
)
else:
assert num_external_tokens == 0
# Only trigger 1 KV transfer per request.
params["do_remote_prefill"] = False
params["_remote_blocks_processed"] = True
def _build_save_meta(
self,
meta: NixlConnectorMetadata,
scheduler_output: SchedulerOutput,
) -> None:
# only called when use_host_buffer is True to build the save metadata
# NOTE: For the prefill side, there might be a chance that an early added
# request is a chunked prefill, so we need to check if new blocks are added
for req_id, new_block_id_groups, _ in yield_req_data(scheduler_output):
req_to_save = self._reqs_need_save.get(req_id)
if req_to_save is None or new_block_id_groups is None:
continue
req = req_to_save
assert req.kv_transfer_params is not None
clipped_block_id_groups = self.get_sw_clipped_blocks(new_block_id_groups)
meta.add_new_req_to_save(
request_id=req_id,
local_block_ids=clipped_block_id_groups,
kv_transfer_params=req.kv_transfer_params,
)
assert scheduler_output.num_scheduled_tokens is not None
num_scheduled_tokens = scheduler_output.num_scheduled_tokens[req_id]
is_partial = (
req.num_computed_tokens + num_scheduled_tokens
) < req.num_prompt_tokens
if not is_partial:
# For non-partial prefills, once new req_meta is scheduled, it
# can be removed from _reqs_need_save.
# For partial prefill case, we will retain the request in
# _reqs_need_save until all blocks are scheduled with req_meta.
# Therefore, only pop if `not is_partial`.
self._reqs_need_save.pop(req_id)
def build_connector_meta(
self,
scheduler_output: SchedulerOutput,
) -> KVConnectorMetadata:
meta = NixlConnectorMetadata()
# Loop through scheduled reqs and convert to ReqMeta.
for req_id, (req, block_ids) in self._reqs_need_recv.items():
assert req.kv_transfer_params is not None
meta.add_new_req_to_recv(
request_id=req_id,
local_block_ids=block_ids,
kv_transfer_params=req.kv_transfer_params,
)
if self.use_host_buffer:
self._build_save_meta(meta, scheduler_output)
meta.reqs_to_send = self._reqs_need_send
meta.reqs_in_batch = self._reqs_in_batch
meta.reqs_not_processed = self._reqs_not_processed
# Package heartbeats, throttled by heartbeat_interval.
if self._heartbeat_by_engine:
now = time.perf_counter()
if now - self._last_heartbeat_time >= self._heartbeat_interval:
self._last_heartbeat_time = now
meta.heartbeat_by_engine = self._heartbeat_by_engine
# Clear the list once workers start the transfers
self._reqs_need_recv.clear()
self._reqs_in_batch = set()
self._reqs_not_processed = set()
self._reqs_need_send = {}
return meta
def update_connector_output(self, connector_output: "KVConnectorOutput") -> None:
"""Stop heartbeating for requests whose KV transfer completed."""
for req_id in connector_output.finished_recving or ():
self._stop_heartbeat(req_id)
def request_finished(
self,
request: "Request",
block_ids: BlockIds,
) -> tuple[bool, dict[str, Any] | None]:
"""
Once a request is finished, determine whether request blocks
should be freed now or will be sent asynchronously and freed later.
"""
from vllm.v1.request import RequestStatus
params = request.kv_transfer_params
logger.debug(
"NIXLConnector request_finished(%s), request_status=%s, "
"kv_transfer_params=%s",
request.request_id,
request.status,
params,
)
if not params:
return False, None
is_p_node = bool(params.get("do_remote_decode"))
is_d_node = not is_p_node
# Stop heartbeating for aborted requests that never reached finished_recving:
# normal path cleans up in update_connector_output.
self._stop_heartbeat(request.request_id)
if params.get("do_remote_prefill"):
# If do_remote_prefill is still True when the request is finished,
# update_state_after_alloc must not have been called (the request
# must have been aborted before it was scheduled, e.g. via the
# abort_immediately path used to clean up KV-transfer requests
# rejected at the D-side serving layer).
# To avoid stranding the prefill blocks in the prefill instance,
# we must add empty block_ids to _reqs_need_recv so that our
# worker side will notify and free blocks in the prefill instance.
self._reqs_need_recv[request.request_id] = (request, [])
params["do_remote_prefill"] = False
return False, None
if is_d_node and not self.is_bidirectional_kv_xfer_enabled:
return False, None
if request.status not in (
RequestStatus.FINISHED_LENGTH_CAPPED,
RequestStatus.FINISHED_STOPPED,
):
# Also include the case of a P/D Prefill request with immediate
# block free (eg abort). Stop tracking this request.
self._reqs_not_processed.add(request.request_id)
# Clear _reqs_need_save if a request is aborted as partial prefill.
self._reqs_need_save.pop(request.request_id, None)
return False, None
# TODO: check whether block_ids actually ever be 0. If not we could
# remove the conditional below
delay_free_blocks = any(len(group) > 0 for group in block_ids)
remote_num_tokens = 0
if delay_free_blocks:
# Prefill request on remote. It will be read from D upon completion
request_kv_blocks_ttl = self._kv_lease_duration
if is_d_node:
# For blocks pinned on D, use a simpler timeout for now instead of a
# lease mechanism as turn2 request is client-driven.
request_kv_blocks_ttl = self.decoder_kv_blocks_ttl
logger.debug(
"NIXLConnector request_finished(%s) waiting for %d seconds "
"before releasing blocks",
request.request_id,
request_kv_blocks_ttl,
)
self._reqs_need_send[request.request_id] = (
time.perf_counter() + request_kv_blocks_ttl
)
# NOTE HMA will "mark" empty/null blocks in groups with 0s (eg SWA ones),
# trimming down after allocating for the whole sequence length. Empty
# blocks are always at the start of the list.
# Here we "unpad" blocks to send the actual remote blocks to be read.
block_ids = self.get_sw_clipped_blocks(block_ids)
remote_num_tokens = request.num_computed_tokens
return delay_free_blocks, dict(
do_remote_prefill=is_p_node,
do_remote_decode=is_d_node,
remote_block_ids=block_ids,
remote_engine_id=self.engine_id,
remote_request_id=request.request_id,
remote_host=self.side_channel_host,
remote_port=self.side_channel_port,
tp_size=self.vllm_config.parallel_config.tensor_parallel_size,
remote_num_tokens=remote_num_tokens,
)
__all__ = ["NixlConnectorScheduler", "NixlPullConnectorScheduler"]
@@ -6,6 +6,7 @@ import contextlib
from collections.abc import Iterator
from typing import Any
import regex as re
import zmq
from vllm.platforms import current_platform
@@ -55,3 +56,13 @@ def get_representative_spec_type(spec: KVCacheSpec) -> type[KVCacheSpec]:
inner = next(iter(spec.kv_cache_specs.values()))
return type(inner)
return type(spec)
# Trailing 8-hex randomization suffix appended by
# ``input_processor.assign_request_id`` as ``-{random_uuid():.8}``.
_RANDOM_SUFFIX_RE = re.compile(r"-[0-9a-f]{8}$", re.IGNORECASE)
def get_base_request_id(request_id: str) -> str:
"""Strip the per-request ``-<8 hex>`` randomization suffix, if present."""
return _RANDOM_SUFFIX_RE.sub("", request_id)
File diff suppressed because it is too large Load Diff
+13
View File
@@ -2009,6 +2009,19 @@ class Scheduler(SchedulerInterface):
)
return len(self.requests) > num_in_queues
def has_requests(self) -> bool:
# Override the interface default to also keep the engine alive while a
# connector still has pending push work (e.g. push-mode WRITE transfers
# in flight after all "live" requests have finished). Without this hook
# the engine would quiesce before the connector can drain completions.
# TODO: replace with a more general mechanism for connectors to keep
# the scheduler alive.
return (
self.has_unfinished_requests()
or self.has_finished_requests()
or (self.connector is not None and self.connector.has_pending_push_work())
)
def reset_prefix_cache(
self, reset_running_requests: bool = False, reset_connector: bool = False
) -> bool: