[Mini-VLLM] Remove LoRA, plugins, speech-to-text and scale-out surfaces

Production runs one static model behind the OpenAI-compatible API. Every
adapter path and non-OpenAI HTTP surface is unreachable there.

- Delete the LoRA stack: vllm/lora, LoRARequest on the engine wire, the
  scheduler, worker, cudagraph and MoE plumbing, LoRA metrics, the lora
  CLI group, and the LoRA tests.
- Drop the plugin loader in vllm/plugins. Platform resolution is now
  CUDA-only and the general/stat-logger plugin hooks are gone.
- Delete the speech-to-text and realtime entrypoints, the scale-out
  render routes, and vllm/config/speech_to_text.py. GenerationTask is
  generate-only.
- Cut DecodeBenchConnector and the disagg-prefill doc and image. The
  Multi, Offloading and Example connectors stay; production uses the
  CPU KV offload tier.
- Delete the ROCm-only fused KDA chunk kernel (2346 lines) with its CMake
  arch gate, declarations and op registrations. The CUDA build never
  defined the guard and no Python caller existed.
- Delete the orphan Mistral and Pixtral chat fixtures and examples/.
- Fix _check_model: removing the LoRA early return made every request
  fail with "The model ... does not exist. Valid aliases: ...".

Verified on an RTX 3090: import sweep reports 1242 modules with 7 known
optional-dependency failures; a live server answers chat, completions,
responses, tokenize and detokenize with zero log errors and 404 for the
removed surfaces; collection yields 8558 tests with the 3 pre-existing
errors; ruff adds no new finding; the incremental _C build succeeds.

Co-authored-by: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
2026-10-19 21:06:19 +07:00
co-authored by Hinata
parent 5f3c7bd8ab
commit 5aa2ac64f8
269 changed files with 417 additions and 29932 deletions
+4 -12
View File
@@ -59,20 +59,13 @@ class BlockStored(KVCacheEvent):
token_ids: list[int]
block_size: int
"""Deprecated: use `lora_name` for KV block key hash.
Retained for backward compatibility.
"""
lora_id: int | None
medium: str | None
lora_name: str | None
"""Extra keys used in block hash computation, one entry per block in
block_hashes. Each entry contains, in order, the LoRA name (if
`lora_name` is set), `(mm_identifier, offset)` pairs, the cache_salt
string and prompt embedding digests for that specific block. These are
published untagged; the block hash itself uses each value tagged with
its source, e.g. `("lora", name)` or `("mm", mm_identifier, offset)`.
block_hashes. Each entry contains, in order, the `(mm_identifier, offset)`
pairs, the cache_salt string and prompt embedding digests for that
specific block. These are published untagged; the block hash itself uses
each value tagged with its source, e.g. `("mm", mm_identifier, offset)`.
"""
extra_keys: list[tuple[Any, ...] | None] | None = None
@@ -100,7 +93,6 @@ class BlockStored(KVCacheEvent):
self.parent_block_hash,
tuple(self.token_ids),
self.block_size,
self.lora_id,
self.medium,
tuple(self.extra_keys) if self.extra_keys else None,
self.group_idx,
-29
View File
@@ -1,29 +0,0 @@
# Distributed KV cache transfer
This folder implements distributed KV cache transfer across vLLM instances.
Currently the main use case is for disaggregated prefilling.
## Abstractions
The KV cache transfer contains three layer of abstractions:
- KV pipe: a FIFO pipe for torch.tensor transmission. Key APIs: `send_tensor` and `recv_tensor`.
- KV lookup buffer: a lookup buffer for KV caches. Key: the tokens, value: the KV caches (and/or hidden states). Key APIs: `insert` and `drop_select` (similar to SQL semantics).
- KV connector: a connector that connects the KV pipe and KV lookup buffer to vLLM. Key APIs: `send_kv_caches_and_hidden_states` and `recv_kv_caches_and_hidden_states`.
Why we need KV lookup buffer: FIFO pipe itself is not enough as prefill vLLM worker may process requests in a different order compared to decode vLLM worker. Say the QPS is really high, prefill worker may handle requests in order A -> B -> C, but the decode worker may process request C first. This is not the case that can be naturally handled by FIFO pipe, so we provide KV lookup buffer to help translate a FIFO pipe to a lookup buffer.
NOTE: KV pipe layer is bypassable: you can skip this layer if your distributed
communication service already supports key-value-based lookup (like redis or
RDMA database).
NOTE: If you want to not only transfer KV caches, but adjust the model execution flow of vLLM as well (for example, allow vLLM to receive KV caches on some tokens and do prefill on the remaining tokens), you can bypass both KV pipe layer and KV lookup buffer layer, and directly implement on KV connector layer. Bear in mind that as vLLM's model input is constantly changing, this implementation will likely be broken when vLLM has new updates.
## Disaggregated prefilling
Example usage can be found in [this directory](../../../examples/disaggregated).
Here is the diagram of how we run disaggregated prefilling.
![Disaggregated prefill workflow](./disagg_prefill_workflow.jpg)
Binary file not shown.

Before

Width:  |  Height:  |  Size: 139 KiB

@@ -161,9 +161,3 @@ KVConnectorFactory.register_connector(
"vllm.distributed.kv_transfer.kv_connector.v1.offloading_connector",
"OffloadingConnector",
)
KVConnectorFactory.register_connector(
"DecodeBenchConnector",
"vllm.distributed.kv_transfer.kv_connector.v1.decode_bench_connector",
"DecodeBenchConnector",
)
@@ -6,14 +6,10 @@ from vllm.distributed.kv_transfer.kv_connector.v1.base import (
SupportsHMA,
supports_hma,
)
from vllm.distributed.kv_transfer.kv_connector.v1.decode_bench_connector import ( # noqa: E501
DecodeBenchConnector,
)
__all__ = [
"KVConnectorRole",
"KVConnectorBase_V1",
"supports_hma",
"SupportsHMA",
"DecodeBenchConnector",
]
@@ -1,734 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""DecodeBenchConnector: A KV Connector for decode instance performance testing.
This connector emulates a prefill-decode disaggregated setting by filling
the KV cache with dummy values, allowing measurement of decoder performance
under larger input sequence lengths (ISL) in resource-limited environments.
Usage:
To use this connector for benchmarking, configure it in the kv_transfer_config:
Example:
vllm serve <model> --kv-transfer-config '{
"kv_connector": "DecodeBenchConnector",
"kv_role": "kv_both",
"kv_connector_extra_config": {
"fill_mean": 0.015,
"fill_std": 0.0
}
}'
Then run your benchmark with desired input/output lengths:
vllm bench serve --base-url http://127.0.0.1:8000 --model <model> \\
--dataset-name random --random-input-len 40000 \\
--random-output-len 100 --max-concurrency 10
Configuration options (via kv_connector_extra_config):
- fill_mean (float): Mean value for random normal fill (default: 0.015)
- fill_std (float): Standard deviation for random fill (default: 0.0)
Set to 0 for constant values, >0 for random sampling
- startup_fill (bool): Fill the whole KV cache once at startup instead
of filling each new request's blocks in the step that schedules it
(default: False). Circular-buffer caches are still zeroed per
request, and attention caches whose new blocks the engine zeroes
before use (hybrid Mamba or mixed-precision KV caches) are still
filled per request. The fill is not redone after sleep mode
discards the KV cache.
The dummy KV cache is only meant for performance measurement; outputs are
not meaningful for accuracy evaluation.
"""
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any
import torch
from vllm.distributed.kv_transfer.kv_connector.v1 import (
KVConnectorBase_V1,
KVConnectorRole,
)
from vllm.distributed.kv_transfer.kv_connector.v1.base import (
KVConnectorMetadata,
SupportsHMA,
)
from vllm.logger import init_logger
from vllm.platforms import current_platform
from vllm.utils.math_utils import cdiv
from vllm.v1.attention.backend import AttentionMetadata
from vllm.v1.core.kv_cache_utils import (
resolve_dcp_kv_block_size,
)
from vllm.v1.kv_cache_interface import (
AttentionSpec,
CircularBufferSpec,
KVCacheSpec,
KVQuantMode,
UniformTypeKVCacheSpecs,
iter_layer_specs,
)
if TYPE_CHECKING:
from vllm.config import VllmConfig
from vllm.forward_context import ForwardContext
from vllm.v1.core.kv_cache_manager import KVCacheBlocks
from vllm.v1.core.sched.output import SchedulerOutput
from vllm.v1.kv_cache_interface import KVCacheConfig
from vllm.v1.request import Request
logger = init_logger(__name__)
def _get_fp8_dtype(spec: KVCacheSpec, cache_dtype: str) -> torch.dtype | None:
"""The fp8 dtype of a KV cache that stores plain fp8 values as uint8.
Returns None for other layouts, including packed ones that embed scales
or other metadata in the uint8 page.
"""
if (
not isinstance(spec, AttentionSpec)
or spec.dtype != torch.uint8
or spec.kv_quant_mode != KVQuantMode.FP8_PER_TENSOR
or spec.state_content_bytes is not None
):
return None
cache_dtype = getattr(spec, "cache_dtype_str", None) or cache_dtype
if cache_dtype in ("fp8", "fp8_e4m3"):
return current_platform.fp8_dtype()
if cache_dtype == "fp8_e5m2":
return torch.float8_e5m2
return None
def _get_startup_fill_group_ids(kv_cache_config: "KVCacheConfig") -> set[int]:
"""KV cache groups that startup fill covers.
Circular buffers are zero-filled per request. Attention groups whose new
blocks the engine zeroes before use are filled per request, since the
zeroing would wipe the startup fill.
"""
group_ids = set()
for group_idx, group in enumerate(kv_cache_config.kv_cache_groups):
spec = group.kv_cache_spec
if any(isinstance(s, CircularBufferSpec) for s in iter_layer_specs(spec)):
continue
if kv_cache_config.needs_kv_cache_zeroing and isinstance(spec, AttentionSpec):
continue
group_ids.add(group_idx)
return group_ids
@dataclass
class DecodeBenchConnectorMetadata(KVConnectorMetadata):
"""Metadata for DecodeBenchConnector.
Contains information about which requests need their KV cache filled
with dummy values for benchmarking purposes.
"""
# request_id -> (block_ids_per_group, num_tokens_to_fill)
# block_ids_per_group is a tuple of lists, one per KV cache group
# One group: ([1, 2, 3],)
# Multiple groups: ([1, 2], [5, 6])
reqs_to_fill: dict[str, tuple[tuple[list[int], ...], int]]
class DecodeBenchConnector(KVConnectorBase_V1, SupportsHMA):
"""A KV Connector for decode instance performance testing.
This connector fills the KV cache with dummy values to emulate a
prefill-decode disaggregated setting, enabling performance testing of the
decoder with larger input sequence lengths.
"""
def __init__(
self,
vllm_config: "VllmConfig",
role: KVConnectorRole,
kv_cache_config: "KVCacheConfig",
):
super().__init__(vllm_config, role, kv_cache_config)
self.connector_scheduler: DecodeBenchConnectorScheduler | None = None
self.connector_worker: DecodeBenchConnectorWorker | None = None
if role == KVConnectorRole.SCHEDULER:
self.connector_scheduler = DecodeBenchConnectorScheduler(
vllm_config, kv_cache_config
)
elif role == KVConnectorRole.WORKER:
self.connector_worker = DecodeBenchConnectorWorker(
vllm_config, kv_cache_config
)
# ==============================
# Worker-side methods
# ==============================
def register_kv_caches(self, kv_caches: dict[str, torch.Tensor]):
assert self.connector_worker is not None
self.connector_worker.register_kv_caches(kv_caches)
def start_load_kv(self, forward_context: "ForwardContext", **kwargs: Any) -> None:
assert self.connector_worker is not None
assert isinstance(self._connector_metadata, DecodeBenchConnectorMetadata)
self.connector_worker.start_fill_kv(self._connector_metadata)
def wait_for_layer_load(self, layer_name: str) -> None:
# All operations are synchronous, so nothing to wait for
pass
def save_kv_layer(
self,
layer_name: str,
kv_layer: torch.Tensor,
attn_metadata: AttentionMetadata,
**kwargs: Any,
) -> None:
# This connector doesn't save KV cache (benchmarking only)
pass
def wait_for_save(self):
# This connector doesn't save KV cache (benchmarking only)
pass
# ==============================
# Scheduler-side methods
# ==============================
def get_num_new_matched_tokens(
self,
request: "Request",
num_computed_tokens: int,
) -> tuple[int | None, bool]:
assert self.connector_scheduler is not None
return self.connector_scheduler.get_num_new_matched_tokens(
request, num_computed_tokens
)
def update_state_after_alloc(
self, request: "Request", blocks: "KVCacheBlocks", num_external_tokens: int
):
assert self.connector_scheduler is not None
return self.connector_scheduler.update_state_after_alloc(
request, blocks, num_external_tokens
)
def build_connector_meta(
self, scheduler_output: "SchedulerOutput"
) -> KVConnectorMetadata:
assert self.connector_scheduler is not None
return self.connector_scheduler.build_connector_meta(scheduler_output)
def request_finished(
self,
request: "Request",
block_ids: list[int],
) -> tuple[bool, dict[str, Any] | None]:
assert self.connector_scheduler is not None
self.connector_scheduler.request_finished(request)
return False, None
def request_finished_all_groups(
self,
request: "Request",
block_ids: tuple[list[int], ...],
) -> tuple[bool, dict[str, Any] | None]:
# HMA-enabled path: same cleanup as the single-group variant since
# this connector owns no external state per block.
assert self.connector_scheduler is not None
self.connector_scheduler.request_finished(request)
return False, None
class DecodeBenchConnectorScheduler:
"""Scheduler-side implementation for DecodeBenchConnector."""
def __init__(self, vllm_config: "VllmConfig", kv_cache_config: "KVCacheConfig"):
kv_transfer_config = vllm_config.kv_transfer_config
assert kv_transfer_config is not None
self.startup_fill = kv_transfer_config.get_from_extra_config(
"startup_fill", False
)
self._startup_fill_group_ids = (
_get_startup_fill_group_ids(kv_cache_config) if self.startup_fill else set()
)
dcp_world_size = vllm_config.parallel_config.decode_context_parallel_size
self.kv_cache_groups = kv_cache_config.kv_cache_groups
self.group_block_sizes = tuple(
resolve_dcp_kv_block_size(group.kv_cache_spec, dcp_world_size)
for group in self.kv_cache_groups
)
# Track which requests have already been filled
self._filled_requests: set[str] = set()
# Track pending fills for the current scheduler step
# request_id -> (block_ids_per_group, num_tokens_to_fill)
# Note: _pending_fills doesn't need explicit cleanup - it's cleared
# after build_connector_meta() is called in the same scheduler step
self._pending_fills: dict[str, tuple[tuple[list[int], ...], int]] = {}
def get_num_new_matched_tokens(
self,
request: "Request",
num_computed_tokens: int,
) -> tuple[int, bool]:
"""For new requests, return the number of tokens that should be filled
with dummy KV cache values.
Returns:
(num_tokens_to_fill, is_async)
- num_tokens_to_fill: number of uncomputed tokens minus 1
(we fill everything except the last token for decode)
- is_async: False (synchronous filling)
"""
req_id = request.request_id
# Only fill once per request on first scheduling
if req_id in self._filled_requests:
return 0, False
# Calculate how many tokens we need to fill
# Fill all uncomputed tokens except the last one (which will be decoded)
# This simulates having processed a long prefill
num_uncomputed_tokens = request.num_tokens - num_computed_tokens
num_tokens_to_fill = max(0, num_uncomputed_tokens - 1)
if num_tokens_to_fill == 0:
return 0, False
# Return False for synchronous operation - the fill is fast enough
# that async overhead isn't worth it
return num_tokens_to_fill, False
def update_state_after_alloc(
self, request: "Request", blocks: "KVCacheBlocks", num_external_tokens: int
):
"""Called after blocks are allocated. Store the block IDs so we can
fill them with dummy values.
Supports both single- and multi-group KV cache configurations.
"""
req_id = request.request_id
if num_external_tokens == 0:
return
total_computed_tokens = request.num_tokens - 1
num_local_computed_tokens = total_computed_tokens - num_external_tokens
block_ids_per_group_list: list[list[int]] = []
for group_idx, (
group,
group_blocks,
group_block_size,
) in enumerate(
zip(
self.kv_cache_groups,
blocks.blocks,
self.group_block_sizes,
strict=True,
)
):
if group_idx in self._startup_fill_group_ids:
block_ids_per_group_list.append([])
continue
is_circular_buffer = all(
isinstance(spec, CircularBufferSpec)
for spec in iter_layer_specs(group.kv_cache_spec)
)
if not is_circular_buffer:
num_computed_blocks = cdiv(total_computed_tokens, group_block_size)
external_block_start = num_local_computed_tokens // group_block_size
assert (
0
<= external_block_start
<= num_computed_blocks
<= len(group_blocks)
), (
"DecodeBenchConnector block range exceeds allocated blocks: "
f"request={req_id}, group={group_idx}, "
f"range=[{external_block_start}, {num_computed_blocks}), "
f"allocated={len(group_blocks)}"
)
selected_blocks = group_blocks[external_block_start:num_computed_blocks]
else:
selected_blocks = group_blocks
block_ids = [
block.block_id for block in selected_blocks if not block.is_null
]
if not block_ids:
logger.warning(
"DecodeBenchConnector: No blocks selected for KV cache group "
"%d with %d external tokens for request %s",
group_idx,
num_external_tokens,
req_id,
)
block_ids_per_group_list.append(block_ids)
block_ids_per_group = tuple(block_ids_per_group_list)
self._filled_requests.add(req_id)
if self.startup_fill and not any(block_ids_per_group):
return
# Store the blocks to fill for all group. _pending_fills doesn't need cleanup
# as it's cleared after build_connector_meta
self._pending_fills[req_id] = (
block_ids_per_group,
num_external_tokens,
)
block_counts = tuple(len(group) for group in block_ids_per_group)
logger.debug(
"DecodeBenchConnector: Selected %d total blocks across %d KV cache "
"groups for request %s (per-group counts: %s)",
sum(block_counts),
len(blocks.blocks),
req_id,
", ".join(map(str, block_counts)),
)
def build_connector_meta(
self, scheduler_output: "SchedulerOutput"
) -> KVConnectorMetadata:
"""Build metadata containing information about which blocks to fill
with dummy KV values.
"""
meta = DecodeBenchConnectorMetadata(reqs_to_fill=self._pending_fills.copy())
# Clear pending fills after building metadata
self._pending_fills.clear()
return meta
def request_finished(self, request: "Request"):
"""Called when a request has finished. Clean up any state."""
self._filled_requests.discard(request.request_id)
class DecodeBenchConnectorWorker:
"""Worker-side implementation for DecodeBenchConnector."""
def __init__(self, vllm_config: "VllmConfig", kv_cache_config: "KVCacheConfig"):
# Get fill parameters from extra config
kv_transfer_config = vllm_config.kv_transfer_config
assert kv_transfer_config is not None
self.fill_mean = kv_transfer_config.get_from_extra_config("fill_mean", 0.015)
self.fill_std = kv_transfer_config.get_from_extra_config("fill_std", 0.0)
self.startup_fill = kv_transfer_config.get_from_extra_config(
"startup_fill", False
)
self._startup_fill_group_ids = (
_get_startup_fill_group_ids(kv_cache_config) if self.startup_fill else set()
)
# Will be populated via register_kv_caches
self.kv_caches: dict[str, torch.Tensor] | None = None
# Layers whose uint8 KV cache stores plain fp8 values
self._fp8_dtypes: dict[str, torch.dtype] = {}
for group in kv_cache_config.kv_cache_groups:
for layer_name in group.layer_names:
spec = group.kv_cache_spec
if isinstance(spec, UniformTypeKVCacheSpecs):
spec = spec.kv_cache_specs[layer_name]
fp8_dtype = _get_fp8_dtype(spec, vllm_config.cache_config.cache_dtype)
if fp8_dtype is not None:
self._fp8_dtypes[layer_name] = fp8_dtype
# Mapping from KV cache group index to list of layer names in that group
self.group_to_layers = {
group_idx: list(group.layer_names)
for group_idx, group in enumerate(kv_cache_config.kv_cache_groups)
}
self._zero_fill_group_ids = {
group_idx
for group_idx, group in enumerate(kv_cache_config.kv_cache_groups)
if any(
isinstance(spec, CircularBufferSpec)
for spec in iter_layer_specs(group.kv_cache_spec)
)
}
num_zeroed_groups = len(
self.group_to_layers.keys()
- self._startup_fill_group_ids
- self._zero_fill_group_ids
)
if self.startup_fill and num_zeroed_groups:
logger.info(
"DecodeBenchConnector: the engine zeroes new blocks of %d KV "
"cache groups before use, so they are still filled per request.",
num_zeroed_groups,
)
def register_kv_caches(self, kv_caches: dict[str, torch.Tensor]):
"""Store references to the KV cache tensors."""
self.kv_caches = kv_caches
logger.debug(
"DecodeBenchConnector: Registered %d KV cache layers",
len(kv_caches),
)
if self.startup_fill:
self._fill_all_kv_caches()
def _fill_all_kv_caches(self):
"""Fill the KV cache tensors of the startup-filled groups once, in
place, with dummy values, so that no per-request fill is needed.
"""
assert self.kv_caches is not None
num_layers = 0
for group_idx in sorted(self._startup_fill_group_ids):
for layer_name in self.group_to_layers[group_idx]:
kv_cache = self.kv_caches.get(layer_name)
if isinstance(kv_cache, torch.Tensor):
tensors = [kv_cache]
elif isinstance(kv_cache, (list, tuple)) and all(
isinstance(t, torch.Tensor) for t in kv_cache
):
tensors = list(kv_cache)
else:
logger.warning_once(
"DecodeBenchConnector: skipping startup fill for layer "
"%s whose KV cache is %s, not a tensor or a list/tuple "
"of tensors.",
layer_name,
type(kv_cache).__name__,
)
continue
for tensor in tensors:
fill_dtype = self._fp8_dtypes.get(layer_name, tensor.dtype)
self._fill_tensor(tensor, self.fill_mean, self.fill_std, fill_dtype)
num_layers += 1
logger.info(
"DecodeBenchConnector: Filled %d KV cache layers at startup with %s "
"values (mean=%.3f, std=%.3f)",
num_layers,
"random" if self.fill_std > 0 else "constant",
self.fill_mean,
self.fill_std,
)
def start_fill_kv(self, metadata: DecodeBenchConnectorMetadata):
"""Fill the allocated KV cache blocks with dummy values.
This simulates having a populated KV cache from a prefill phase,
allowing decode performance testing with larger context sizes.
Supports both single- and multi-group KV cache configurations.
"""
if not metadata.reqs_to_fill:
return
assert self.kv_caches is not None, "KV caches must be registered before filling"
for req_id, (block_ids_per_group, num_tokens) in metadata.reqs_to_fill.items():
# Fill blocks for each KV cache group
for group_idx, block_ids in enumerate(block_ids_per_group):
self._fill_blocks(group_idx, block_ids, num_tokens)
block_counts = tuple(len(group) for group in block_ids_per_group)
logger.debug(
"DecodeBenchConnector: Filled %d total blocks (%d tokens) across "
"%d groups for request %s (per-group counts: %s)",
sum(block_counts),
num_tokens,
len(block_ids_per_group),
req_id,
", ".join(map(str, block_counts)),
)
def _fill_blocks(self, group_idx: int, block_ids: list[int], num_tokens: int):
"""Fill specified blocks with dummy values for a specific KV cache group.
Args:
group_idx: The KV cache group index to fill
block_ids: List of block IDs to fill in this group
num_tokens: Total number of tokens to fill across these blocks
"""
if not block_ids:
return
assert self.kv_caches is not None
# Circular buffers may pack non-floating metadata alongside their
# floating-point state, so arbitrary fill values are not representation-safe.
fill_mean, fill_std = (
(0.0, 0.0)
if group_idx in self._zero_fill_group_ids
else (self.fill_mean, self.fill_std)
)
# Get the layers that belong to this group
layer_names = self.group_to_layers.get(group_idx, [])
# Fill only the layers in this group
for layer_name in layer_names:
if layer_name not in self.kv_caches:
logger.warning(
"DecodeBenchConnector: Layer %s not found in KV caches", layer_name
)
continue
kv_cache = self.kv_caches[layer_name]
# Attention layers store KV as a single block-indexed tensor whose
# first dim is num_blocks; fill the requested block rows. Hybrid /
# linear-attention layers (e.g. Mamba, Kimi Delta Attention) store
# their state as a list/tuple of tensors that are NOT block-indexed
# — each tensor is a single state buffer with no num_blocks
# dimension — so fill each tensor in its entirety with the same
# dummy values.
if isinstance(kv_cache, torch.Tensor):
fill_dtype = self._fp8_dtypes.get(layer_name, kv_cache.dtype)
self._fill_block_tensor(
kv_cache, block_ids, fill_mean, fill_std, fill_dtype
)
elif isinstance(kv_cache, (list, tuple)) and all(
isinstance(t, torch.Tensor) for t in kv_cache
):
for state_tensor in kv_cache:
self._fill_tensor(
state_tensor, fill_mean, fill_std, state_tensor.dtype
)
else:
logger.warning_once(
"DecodeBenchConnector: skipping fill for layer %s whose KV "
"cache is %s, not a tensor or a list/tuple of tensors.",
layer_name,
type(kv_cache).__name__,
)
continue
logger.debug(
"DecodeBenchConnector: Filled %d blocks in group %d with %s values "
"(mean=%.3f, std=%.3f)",
len(block_ids),
group_idx,
"random" if fill_std > 0 else "constant",
fill_mean,
fill_std,
)
def _fill_block_tensor(
self,
kv_cache: torch.Tensor,
block_ids: list[int],
fill_mean: float,
fill_std: float,
fill_dtype: torch.dtype,
):
"""Fill the requested block rows of a block-indexed KV cache tensor.
Args:
kv_cache: A KV cache tensor whose first dim is num_blocks.
block_ids: Block IDs to fill. IDs that are out of range for this
tensor's first dim are ignored.
fill_mean: Mean value for the fill.
fill_std: Standard deviation for the fill.
fill_dtype: Dtype the values are encoded as, e.g. the fp8 dtype
of a uint8 fp8 cache.
"""
# Convert block_ids to tensor on device
block_ids_tensor = torch.tensor(
block_ids, dtype=torch.long, device=kv_cache.device
)
# Filter invalid block IDs
valid_mask = block_ids_tensor < kv_cache.shape[0]
valid_block_ids = block_ids_tensor[valid_mask]
if len(valid_block_ids) == 0:
return
fill_values = self._make_fill_values(
(len(valid_block_ids),) + kv_cache.shape[1:],
kv_cache.device,
fill_dtype,
fill_mean,
fill_std,
)
# Batch fill operation
kv_cache[valid_block_ids] = fill_values.view(kv_cache.dtype)
def _make_fill_values(
self,
size: tuple[int, ...],
device: torch.device,
dtype: torch.dtype,
fill_mean: float,
fill_std: float,
) -> torch.Tensor:
"""Create constant or random fill values of ``dtype``, clamped to its
finite range. Non-floating dtypes (packed layouts) are filled with
zeros.
"""
if not dtype.is_floating_point:
if fill_mean or fill_std:
logger.warning_once(
"DecodeBenchConnector: %s KV caches do not hold plain "
"floating-point values; filling them with zeros.",
dtype,
)
return torch.zeros(size, dtype=dtype, device=device)
finfo = torch.finfo(dtype)
if fill_std > 0:
sample_dtype = torch.float32
if dtype in (torch.float16, torch.bfloat16, torch.float32, torch.float64):
sample_dtype = dtype
values = torch.normal(
fill_mean, fill_std, size=size, device=device, dtype=sample_dtype
)
return values.clamp_(finfo.min, finfo.max).to(dtype)
fill_mean = min(max(fill_mean, finfo.min), finfo.max)
return torch.full(size, fill_mean, dtype=dtype, device=device)
def _fill_tensor(
self,
kv_cache: torch.Tensor,
fill_mean: float,
fill_std: float,
fill_dtype: torch.dtype,
):
"""Fill an entire tensor in place with dummy values.
Used for startup fills, and for hybrid / linear-attention layers (e.g.
Mamba, Kimi Delta Attention) whose per-layer state tensors are filled
in their entirety with the same constant or random values used for
block fills, rather than selected block rows.
Args:
kv_cache: A tensor to fill in its entirety.
fill_mean: Mean value for the fill.
fill_std: Standard deviation for the fill.
fill_dtype: Dtype the values are encoded as, e.g. the fp8 dtype
of a uint8 fp8 cache.
"""
view = kv_cache.view(fill_dtype)
if fill_std == 0 or not fill_dtype.is_floating_point:
view.fill_(
self._make_fill_values(
(), torch.device("cpu"), fill_dtype, fill_mean, fill_std
)
)
return
# Sample in chunks to bound the temporary memory.
rows = max(1, (1 << 24) // max(1, view[0].numel()))
for chunk in view.split(rows):
chunk.copy_(
self._make_fill_values(
chunk.shape, chunk.device, fill_dtype, fill_mean, fill_std
)
)
@@ -87,8 +87,6 @@ class _OffloadEventMetadata:
parent_block_hash: BlockHash | None
token_ids: tuple[int, ...]
block_size: int
lora_id: int | None
lora_name: str | None
# Deferred: needs the same incremental curr_mm_idx handling as GPU events.
extra_keys: tuple[tuple[Any, ...] | None, ...] | None
group_idx: int
@@ -218,15 +216,11 @@ class OffloadingEventsTracker:
)
assert first_hash_idx == 0 or parent_block_hash is not None
lora_id = req.lora_request.adapter_id if req.lora_request is not None else None
lora_name = req.lora_request.name if req.lora_request is not None else None
meta = _OffloadEventMetadata(
block_hashes=block_hashes,
parent_block_hash=parent_block_hash,
token_ids=tuple(req.all_token_ids[chunk_start:boundary_tokens]),
block_size=tokens_per_hash,
lora_id=lora_id,
lora_name=lora_name,
extra_keys=None,
group_idx=group_config.group_idx,
kv_cache_spec=group_config.kv_event_group_spec,
@@ -315,19 +309,11 @@ class OffloadingEventsTracker:
assert tok_end <= len(req.all_token_ids)
token_ids = tuple(req.all_token_ids[tok_start:tok_end])
lora_id: int | None = None
lora_name: str | None = None
if req.lora_request is not None:
lora_id = req.lora_request.adapter_id
lora_name = req.lora_request.name
return _OffloadEventMetadata(
block_hashes=tuple(chunk_hashes),
parent_block_hash=parent_block_hash,
token_ids=token_ids,
block_size=group_config.tokens_per_block,
lora_id=lora_id,
lora_name=lora_name,
extra_keys=None,
group_idx=group_config.group_idx,
kv_cache_spec=group_config.kv_event_group_spec,
@@ -347,10 +333,8 @@ class OffloadingEventsTracker:
],
parent_block_hash=None,
token_ids=[],
lora_id=None,
block_size=0,
medium=_MEDIUM_TO_EVENT_STR[medium],
lora_name=None,
group_idx=get_offload_group_idx(key),
locality=locality,
ownership=ownership,
@@ -391,9 +375,7 @@ class OffloadingEventsTracker:
),
token_ids=list(meta.token_ids),
block_size=meta.block_size,
lora_id=meta.lora_id,
medium=_MEDIUM_TO_EVENT_STR[event.medium],
lora_name=meta.lora_name,
extra_keys=to_event_extra_keys(meta.extra_keys),
group_idx=meta.group_idx,
kv_cache_spec_kind=meta.kv_cache_spec.kv_cache_spec_kind,