[3/N][Feat][Perf] Add new warmup infrastructure for JITs. Add provider registry and orchestration for JIT warmup (#50174)

Signed-off-by: LopezCastroRoberto <rocastro@redhat.com>
Co-authored-by: Codex <codex@openai.com>
This commit is contained in:
Roberto L. Castro
2026-08-17 10:11:14 -07:00
committed by GitHub
co-authored by Codex
parent 7075ddac28
commit 49905ad94d
12 changed files with 1064 additions and 165 deletions
+1
View File
@@ -71,6 +71,7 @@ uv pip install -e . --no-build-isolation
For more details about installing from source and installing for other hardware, check out the [installation instructions](../getting_started/installation/README.md) for your hardware and head to the "Build wheel from source" section.
For an optimized workflow when iterating on C++/CUDA kernels, see the [Incremental Compilation Workflow](./incremental_build.md) for recommendations.
For JIT kernel warmup conventions, see [JIT Kernel Warmup](./jit_kernel_warmup.md).
!!! tip
vLLM is compatible with Python versions 3.10 to 3.13. However, vLLM's default [Dockerfile](../../docker/Dockerfile) ships with Python 3.12 and tests in CI (except `mypy`) are run with Python 3.12.
+308
View File
@@ -0,0 +1,308 @@
# JIT Kernel Warmup
vLLM uses JIT-generated kernels from Triton, CuTeDSL, TileLang, and other backends. This contract makes their required specializations available during startup, before the first request, by warming the kernel's **compile-key space** without dummy runtime launches or real tensor allocation.
Use it when adding a warmable JIT kernel or migrating an existing warmup path.
## In This Guide
- [1. Quickstart](#1-quickstart): for contributors adding or migrating a warmable kernel.
- [2. Search-Space Reference](#2-search-space-reference): additional details regarding warmup input expansion and traced dispatch rules.
## 1. Quickstart
Each warmable kernel defines its compile-key mapping and compile-only entry point beside its normal runtime implementation. The startup registry then warms only the wrappers selected by the current engine configuration.
### Define the Kernel Wrapper
Here, a **kernel wrapper** (or just **wrapper**) is an instance of a concrete `VllmJitKernel` subclass.
Expose one wrapper near the kernel's normal runtime entry point. Prefer this shape:
```python
class MyKernel(VllmJitKernel["MyKernel.CompileKey"]):
@dataclass(frozen=True)
class CompileKey:
...
@staticmethod
def kernel(...):
...
def dispatch(self, ...) -> CompileKey:
return self.CompileKey(...)
def get_warmup_keys(self, ...) -> list[CompileKey]:
return self._trace_dispatch(self.dispatch)(...)
def compile(self, compile_key: CompileKey) -> None:
...
def __call__(self, ...):
return self.kernel(...)
MY_KERNEL = MyKernel()
```
`CompileKey`, `dispatch(...)`, and `get_warmup_keys(...)` are backend-agnostic. Backend-specific behavior belongs in `kernel(...)`, `compile(...)`, and `__call__(...)`.
The module-level singleton should be used by warmup and by the runtime call path. This keeps dispatch behavior shared instead of duplicated.
`VllmJitKernel.warmup(...)` compiles every key returned by `get_warmup_keys(...)`; wrappers should not reimplement it.
### Choose Compile-Key Fields
`CompileKey` must be frozen and hashable. Include only fields on which the backend specializes, such as tile sizes, head dimensions, dtypes, pointer alignment classes, or backend selectors; exclude runtime-only values. When unsure, inspect the backend cache key, specialization arguments, or verbose JIT-monitor output.
### Generate Warmup Keys
Use `_trace_dispatch(self.dispatch)` to describe representative inputs. The tracer maps them through the same specialization logic and deduplicates equal keys:
```python
def get_warmup_keys(self, vllm_config: VllmConfig) -> list[CompileKey]:
max_tokens = vllm_config.scheduler_config.max_num_batched_tokens
return self._trace_dispatch(self.dispatch)(
num_tokens=WarmupIntRange(1, max_tokens + 1),
)
```
Use independent ranges or alternatives for cartesian products, `zip_inputs(...)` for coupled rows, and `_when` for validity constraints. The complete syntax is documented in [Search-Space Reference](#2-search-space-reference).
### Compile Without Launching
`compile(compile_key)` means "make this specialization available". Depending on the backend, that may compile from source, call a compile-only API, load an already-built artifact, or compile on cache miss.
`compile(...)` should not launch a real inference workload or allocate real tensors. Each DSL should expose fake tensor/spec descriptors suitable for compilation only.
### Register the Selected Wrapper
Register the wrapper where the runtime implementation is selected:
```python
MY_KERNEL.register_warmup()
```
Registration records metadata only. It does not compile or launch the kernel. Repeated registrations from equivalent layers are allowed and deduplicated later.
### Review Checklist
- Warm actual compile keys rather than representative non-key inputs.
- Keep specialization mapping in `dispatch(...)` instead of duplicating it in warmup code.
- Use fake tensors or backend compile-only descriptors; never perform a dummy runtime launch.
- Keep registration metadata-only so model construction remains cheap and side-effect free.
- Compile registered kernels only through `kernel_warmup()`.
- Keep runtime execution and startup compilation separate and easy to review.
- Use one module-level wrapper instance for registration and runtime calls.
## 2. Search-Space Reference
### How Tracing Works
`_trace_dispatch(...)` expands the inputs declared by `get_warmup_keys(...)`. Each concrete combination becomes a `dispatch_values` mapping from input names to selected values. `_when` may reject that mapping; otherwise the tracer evaluates `dispatch(...)` to construct one `CompileKey`. Equal keys are deduplicated after all combinations are evaluated.
One call to `dispatch(...)` returns one key, but many input points may map to the same key. Prefer this traced mapping over manually reconstructing keys in warmup code; `dispatch(...)` should express the same specialization logic used by the runtime path.
### Define Input Spaces
Use ranges and alternatives for independent axes, `zip_inputs(...)` for coupled rows, and `_when` for validity constraints.
#### Integer Ranges
Use `WarmupIntRange` for integer ranges:
```python
return self._trace_dispatch(self.dispatch)(
num_prefills=WarmupIntRange(1, max_prefills + 1),
)
```
`WarmupIntRange(start, stop, step)` follows Python `range(...)` semantics: `start` is inclusive, `stop` is exclusive, and `step` defaults to 1.
For non-linear integer sequences, use `advance` to provide the action that computes each next value:
```python
return self._trace_dispatch(self.dispatch)(
num_tokens=WarmupIntRange(
1,
max_tokens + 1,
advance=lambda value: next_power_of_2(value) + 1,
),
)
```
This is useful for traversing specialization boundaries without enumerating every integer. `advance` cannot be combined with a non-default `step`, and it must return a value greater than its input so expansion always makes forward progress.
#### Independent Alternatives
Use tuples or lists for independent alternatives. Multiple expanded inputs form a cartesian product:
```python
return self._trace_dispatch(self.dispatch)(
query_slice_start=WarmupIntRange(0, 2),
query_slice_stop=(1, 2 * max_tokens - 1, 2 * max_tokens),
COMPRESS_RATIO=list(compress_ratios),
)
```
#### Coupled Inputs
Use `zip_inputs(...)` when values must vary together row-by-row:
```python
WARMUP_INPUTS = zip_inputs(
dict(compress_ratio=1, topk=0, topk_width=512),
dict(compress_ratio=4, topk=512, topk_width=512),
)
return self._trace_dispatch(self.dispatch)(
WARMUP_INPUTS,
WINDOW_SIZE=window_size,
)
```
Multiple `zip_inputs(...)` groups may be passed as positional arguments. The tracer forms the cartesian product across groups while preserving row-wise coupling inside each group.
Every row in a `zip_inputs(...)` group must use the same string keys. A `zip_inputs(...)` group cannot specify a field that is also specified as a keyword input to `_trace_dispatch(...)`.
#### Conditional Filtering
Use `_when=...` to filter generated input points before they are passed to `dispatch(...)`. This is useful when independent ranges contain invalid combinations, but the validity rule belongs with the kernel warmup definition.
```python
def _is_valid_warmup_input(
self,
*,
query_len: int,
num_reqs: int,
max_num_batched_tokens: int,
) -> bool:
return query_len + num_reqs - 1 <= max_num_batched_tokens
return self._trace_dispatch(self.dispatch)(
query_len=WarmupIntRange(1, max_tokens + 1),
num_reqs=WarmupIntRange(1, max_reqs + 1),
max_num_batched_tokens=max_tokens,
_when=self._is_valid_warmup_input,
)
```
`_when` accepts a function, bound method, or lambda and supports the same AST subset as `dispatch(...)`, including local assignments in function predicates.
The predicate is evaluated on the expanded warmup inputs. If it returns `False`, that input point is skipped and no `CompileKey` is produced for it.
### Write Dispatch Rules
#### Local Assignments
The traced body may contain local assignments, optionally annotated, followed by one `return self.CompileKey(...)` call. Local assignments let a kernel name intermediate specialization choices once and reuse them across fields:
```python
def dispatch(
self,
*,
num_tokens: int,
vectorized: bool,
) -> CompileKey:
block_size = next_power_of_2(num_tokens)
return self.CompileKey(
BLOCK_SIZE=block_size,
VECTOR_WIDTH=4 if vectorized and block_size >= 4 else 1,
)
```
#### Supported Expressions
The evaluator supports these expressions inside local assignments and `CompileKey(...)` fields:
| Feature | What It Allows |
| --- | --- |
| Names | Read dispatch inputs, local assignments, defaults, and module globals. |
| Constants | Use literals such as integers, strings, booleans, and `None`. |
| Attributes | Read structured values such as `cfg.block_size` or `mla_dims.v_head_dim`. |
| Subscriptions | Read sequence positions or mapping values such as `config[0]` and `config["block_size"]`. |
| Tuple/list literals | Build shapes, strides, and other small structured fields. |
| Conditional expressions | Select a field with `x if condition else y`. |
| Boolean expressions | Combine predicates with `and`, `or`, and `not`. |
| Comparisons | Use `==`, `!=`, `<`, `<=`, `>`, `>=`, `in`, `not in`, `is`, and `is not`. |
| Arithmetic | Use `+`, `-`, `*`, `//`, `%`, and `**`. |
| Unary minus | Build negative sentinel values or signed descriptors. |
| Helper calls | Call helpers with positional and explicit keyword arguments. |
Python builtins such as `min(...)`, `max(...)`, and `len(...)` are resolved unless the name is overridden locally or globally.
#### Helper Calls
Helpers are useful for small specialization rules:
```python
def dispatch(self, *, num_tokens: int, block_size: int) -> CompileKey:
return self.CompileKey(
PADDED_TOKENS=round_up(num_tokens, multiple=block_size),
)
```
`_trace_dispatch(...)` does not inspect helper bodies. It evaluates the call arguments and invokes the helper as ordinary Python, so control flow inside that helper is outside the AST interpreter's scope. Keep helpers deterministic and side-effect free.
#### Direct Keyword Forwarding
For many direct pass-through fields, the dispatch `**kwargs` parameter may be unpacked into `CompileKey(...)`:
```python
def dispatch(
self,
*,
num_tokens: int,
**compile_key_fields: int,
) -> CompileKey:
return self.CompileKey(
**compile_key_fields,
block_size=next_power_of_2(num_tokens),
)
```
Unmatched dispatch arguments become compile-key fields and warmup inputs. Keep transformed inputs named and explicit. The unpacking must use the dispatch method's own `**kwargs` parameter directly and exactly once; arbitrary mappings, repeated unpacking, and helper-call `**kwargs` are rejected. The fully explicit form remains supported and is often clearer for non-trivial mappings.
#### Unsupported Syntax
Conditional expressions (`x if condition else y`) are supported, but statement-level `if` blocks are not supported directly inside traced `dispatch(...)` or `_when` bodies. The tracer expects a straight-line sequence of local assignments followed by one return expression. Small, pure helpers called by traced expressions execute as normal Python with concrete values and may use ordinary control flow, including `if` blocks. Do not put loops, mutation, side effects, or backend imports directly inside traced functions. Put environment and model gating in `get_warmup_keys(...)` or the outer warmup entry point.
### Compile-Key Deduplication
`_trace_dispatch(...)` deduplicates the resulting keys while preserving order. This is important when many runtime-like inputs map to the same static bucket.
For example, this warmup range expands every token count, but the compile key only depends on the power-of-two bucket:
```python
def dispatch(
self,
*,
num_tokens: int,
) -> CompileKey:
return self.CompileKey(
BLOCK_SIZE=next_power_of_2(num_tokens),
)
def get_warmup_keys(self, vllm_config: VllmConfig) -> list[CompileKey]:
max_tokens = vllm_config.scheduler_config.max_num_batched_tokens
return self._trace_dispatch(self.dispatch)(
num_tokens=WarmupIntRange(1, max_tokens + 1),
)
```
For `max_tokens == 8`, the expanded inputs are `1, 2, 3, 4, 5, 6, 7, 8`, but the returned keys are:
```python
[
CompileKey(BLOCK_SIZE=1),
CompileKey(BLOCK_SIZE=2),
CompileKey(BLOCK_SIZE=4),
CompileKey(BLOCK_SIZE=8),
]
```
Deduplication happens after `dispatch(...)` is evaluated, so the warmup system removes duplicate compile keys, not duplicate input values. `CompileKey` must be hashable for this to work; using `@dataclass(frozen=True)` is the standard pattern.
+230 -1
View File
@@ -9,11 +9,15 @@ from typing import Any, cast
import pytest
from vllm.model_executor.warmup.jit_warmup import (
JitWarmupRegistry,
VllmJitKernel,
WarmupIntRange,
get_ast_full_name,
zip_inputs,
)
from vllm.model_executor.warmup.jit_warmup_triton_helper import (
triton_scalar_specialization_rep,
)
def _next_power_of_2(value: int) -> int:
@@ -94,6 +98,35 @@ class RecordingToyKernel(ToyKernel):
self.compiled.append(compile_key)
@pytest.mark.parametrize(
("value", "expected"),
[
(-(1 << 63), 1 << 31),
(-(1 << 31) - 1, (1 << 31) + 1),
(-(1 << 31), 16),
(0, 16),
(1, 1),
(2, 2),
(16, 16),
((1 << 31) - 1, 2),
(1 << 31, 1 << 31),
((1 << 31) + 1, (1 << 31) + 1),
((1 << 63) - 1, (1 << 31) + 1),
(1 << 63, 1 << 63),
((1 << 63) + 1, (1 << 63) + 1),
((1 << 64) - 1, (1 << 63) + 1),
],
)
def test_triton_scalar_specialization_rep(value: int, expected: int) -> None:
assert triton_scalar_specialization_rep(value) == expected
@pytest.mark.parametrize("value", [-(1 << 63) - 1, 1 << 64])
def test_triton_scalar_specialization_rep_rejects_out_of_range(value: int) -> None:
with pytest.raises(OverflowError, match="outside Triton's scalar range"):
triton_scalar_specialization_rep(value)
def test_trace_dispatch_expands_ranges_dedupes_and_ignores_unused_inputs() -> None:
cfg = _config()
@@ -287,6 +320,31 @@ def test_helper_calls_support_keywords_and_reject_star_kwargs() -> None:
StarKwargsKernel().compile_key({"tokens": 5, "block_size": 4})
def test_dispatch_helper_calls_resolve_python_builtins() -> None:
class BuiltinKernel(VllmJitKernel["BuiltinKernel.CompileKey"]):
@dataclass(frozen=True)
class CompileKey:
value: int
def dispatch( # type: ignore[override]
self,
*,
tokens: int,
limit: int,
) -> CompileKey:
return self.CompileKey(value=max(1, min(tokens, limit)))
def get_warmup_keys(self) -> list[CompileKey]:
return []
def compile(self, compile_key: CompileKey) -> None:
pass
assert BuiltinKernel().compile_key({"tokens": 8, "limit": 4}) == (
BuiltinKernel.CompileKey(value=4)
)
def test_dispatch_body_must_be_local_assignments_then_compile_key_return() -> None:
class BranchKernel(VllmJitKernel["BranchKernel.CompileKey"]):
@dataclass(frozen=True)
@@ -320,10 +378,85 @@ def test_dispatch_body_must_be_local_assignments_then_compile_key_return() -> No
with pytest.raises(ValueError, match="local assignments"):
BranchKernel()
with pytest.raises(ValueError, match=r"cannot use \*\*kwargs in CompileKey"):
with pytest.raises(
ValueError,
match=r"may unpack only its own \*\*kwargs parameter once",
):
KwargsReturnKernel()
def test_dispatch_can_forward_compile_key_fields() -> None:
class ForwardingKernel(VllmJitKernel["ForwardingKernel.CompileKey"]):
@dataclass(frozen=True)
class CompileKey:
mode: int
block_size: int
def dispatch( # type: ignore[override]
self,
*,
tokens: int,
**compile_key_fields: int,
) -> CompileKey:
return self.CompileKey(
**compile_key_fields,
block_size=_round_up(tokens, multiple=4),
)
def get_warmup_keys(self) -> list[CompileKey]:
return self._trace_dispatch(self.dispatch)(
tokens=(1, 5),
mode=(2, 3),
)
def compile(self, compile_key: CompileKey) -> None:
pass
kernel = ForwardingKernel()
expected = kernel.CompileKey(
mode=2,
block_size=8,
)
assert kernel.dispatch(tokens=5, mode=2) == expected
assert kernel.compile_key({"tokens": 5, "mode": 2}) == expected
assert kernel.get_warmup_keys() == [
kernel.CompileKey(mode=2, block_size=4),
kernel.CompileKey(mode=3, block_size=4),
kernel.CompileKey(mode=2, block_size=8),
kernel.CompileKey(mode=3, block_size=8),
]
with pytest.raises(TypeError, match="field 'block_size' is specified twice"):
kernel.compile_key({"tokens": 5, "mode": 2, "block_size": 4})
with pytest.raises(TypeError, match="unexpected keyword argument 'extra'"):
kernel.compile_key({"tokens": 5, "mode": 2, "extra": 1})
def test_dispatch_supports_tuple_and_mapping_subscriptions() -> None:
class SubscriptKernel(VllmJitKernel["SubscriptKernel.CompileKey"]):
@dataclass(frozen=True)
class CompileKey:
first: int
named: int
def dispatch( # type: ignore[override]
self,
*,
values: tuple[int, ...],
config: dict[str, int],
) -> CompileKey:
return self.CompileKey(first=values[0], named=config["named"])
def get_warmup_keys(self) -> list[CompileKey]:
return []
def compile(self, compile_key: CompileKey) -> None:
pass
assert SubscriptKernel().compile_key(
{"values": (3, 5), "config": {"named": 7}}
) == SubscriptKernel.CompileKey(first=3, named=7)
def test_dispatch_reports_unsupported_expression_with_context() -> None:
class UnsupportedKernel(VllmJitKernel["UnsupportedKernel.CompileKey"]):
@dataclass(frozen=True)
@@ -361,6 +494,102 @@ def test_warmup_compiles_all_returned_keys_in_order() -> None:
]
def test_runtime_cache_miss_compiles_and_caches_executor() -> None:
class CachedKernel(VllmJitKernel["CachedKernel.CompileKey"]):
@dataclass(frozen=True)
class CompileKey:
value: int
def __init__(self) -> None:
self.compiled: list[CachedKernel.CompileKey] = []
super().__init__()
def dispatch(self, *, value: int) -> CompileKey: # type: ignore[override]
return self.CompileKey(value=value)
def get_warmup_keys(self) -> list[CompileKey]:
return self._trace_dispatch(self.dispatch)(value=1)
def compile(self, compile_key: CompileKey) -> None:
self.compiled.append(compile_key)
self._compiled_cache[compile_key] = object()
def __call__(self, value: int) -> None:
compile_key = self.dispatch(value=value)
self._get_or_compile(compile_key)
kernel = CachedKernel()
kernel.warmup()
kernel(1)
kernel(2)
assert kernel.compiled == [
CachedKernel.CompileKey(value=1),
CachedKernel.CompileKey(value=2),
]
def test_registry_records_only_inside_model_setup_context() -> None:
registry = JitWarmupRegistry(_config())
kernel = RecordingToyKernel()
kernel.register_warmup(3, _config())
with registry.activate():
kernel.register_warmup(3, _config())
assert len(registry) == 1
assert kernel.compiled == []
def test_registry_expands_requests_and_deduplicates_owner_keys() -> None:
registry = JitWarmupRegistry(_config())
kernel = RecordingToyKernel()
with registry.activate():
kernel.register_warmup(3, _config())
kernel.register_warmup(5, _config())
registry.warmup()
assert kernel.compiled == [
ToyKernel.CompileKey(1, 1, 1, ("base", "default", -1, 1, 1), True),
ToyKernel.CompileKey(2, 2, 1, ("base", "default", -2, 2, 4), True),
ToyKernel.CompileKey(4, 4, 1, ("base", "default", -4, 1, 16), True),
ToyKernel.CompileKey(8, 8, 1, ("base", "default", -8, 2, 64), True),
]
def test_registry_passes_vllm_config_to_default_requests() -> None:
class ConfigKernel(VllmJitKernel["ConfigKernel.CompileKey"]):
@dataclass(frozen=True)
class CompileKey:
value: int
def __init__(self) -> None:
self.compiled: list[ConfigKernel.CompileKey] = []
super().__init__()
def dispatch(self, *, value: int) -> CompileKey: # type: ignore[override]
return self.CompileKey(value=value)
def get_warmup_keys(self, vllm_config: Any) -> list[CompileKey]:
return [self.dispatch(value=vllm_config.bias)]
def compile(self, compile_key: CompileKey) -> None:
self.compiled.append(compile_key)
registry = JitWarmupRegistry(_config(bias=7))
kernel = ConfigKernel()
with registry.activate():
kernel.register_warmup()
kernel.register_warmup()
assert len(registry) == 1
registry.warmup()
assert kernel.compiled == [ConfigKernel.CompileKey(value=7)]
def test_get_ast_full_name_handles_names_attributes_and_other_nodes() -> None:
dotted_expr = ast.parse("foo.bar.baz").body[0]
call_expr = ast.parse("foo()").body[0]
@@ -0,0 +1,52 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Validate the registry reference kernel against runtime dispatch."""
import pytest
from vllm.platforms import current_platform
if not current_platform.is_cuda_alike():
pytest.skip("NVIDIA dispatch tests require CUDA", allow_module_level=True)
from vllm.v1.worker.block_table import ComputeSlotMappingKernel
@pytest.mark.parametrize(
("kv_cache_block_size", "blocks_per_kv_block", "block_size", "block_size_rep"),
[
(256, 1, 256, 16),
(256, 4, 64, 16),
(64, 1, 64, 16),
(8, 1, 8, 2),
(4, 1, 4, 2),
],
)
def test_compute_slot_mapping_warmup_matches_runtime_specializations(
kv_cache_block_size: int,
blocks_per_kv_block: int,
block_size: int,
block_size_rep: int,
) -> None:
kernel = ComputeSlotMappingKernel()
kwargs = dict(
kv_cache_block_size=kv_cache_block_size,
blocks_per_kv_block=blocks_per_kv_block,
total_cp_world_size=1,
total_cp_rank=0,
cp_kv_cache_interleave_size=1,
block_table_stride=32768,
block_size=block_size,
)
expected = kernel.CompileKey(
kv_cache_block_size=kv_cache_block_size,
blocks_per_kv_block=blocks_per_kv_block,
total_cp_world_size=1,
total_cp_rank=0,
cp_kv_cache_interleave_size=1,
block_table_stride=16,
block_size=block_size_rep,
)
assert kernel.dispatch(**kwargs) == expected
assert kernel.get_warmup_keys(**kwargs) == [expected]
+211 -18
View File
@@ -5,16 +5,20 @@
from __future__ import annotations
import ast
import builtins
import inspect
import itertools
import operator
import textwrap
from abc import ABC, abstractmethod
from collections.abc import Callable, Iterable, Mapping
from collections.abc import Callable, Iterable, Iterator, Mapping
from contextlib import contextmanager
from contextvars import ContextVar
from dataclasses import dataclass
from typing import Any, Generic, TypeVar
__all__ = [
"JitWarmupRegistry",
"VllmJitKernel",
"WarmupIntRange",
"get_ast_full_name",
@@ -136,8 +140,15 @@ class _CompileKeyDispatchTrace:
field_exprs: tuple[tuple[str, ast.AST], ...]
globals: Mapping[str, Any]
input_names: frozenset[str]
# Named parameters are excluded from direct **kwargs forwarding.
named_parameters: frozenset[str] | None
defaults: Mapping[str, Any]
def input_names_for(self, available_names: set[str]) -> frozenset[str]:
if self.named_parameters is None:
return self.input_names
return self.input_names | (available_names - self.named_parameters)
def compile_key(
self,
compile_key_type: type[CompileKeyT],
@@ -146,12 +157,20 @@ class _CompileKeyDispatchTrace:
dispatch_values = _eval_local_exprs(
self.local_exprs, {**self.defaults, **kwargs}, self.globals
)
return compile_key_type(
**{
field: _eval_dispatch_expr(expr, dispatch_values, self.globals)
for field, expr in self.field_exprs
named_parameters = self.named_parameters
# Materialize direct fields before evaluating named AST expressions.
fields: dict[str, Any] = {}
if named_parameters is not None:
fields = {
name: value
for name, value in kwargs.items()
if name not in named_parameters
}
)
for field, expr in self.field_exprs:
if field in fields:
raise TypeError(f"CompileKey field '{field}' is specified twice")
fields[field] = _eval_dispatch_expr(expr, dispatch_values, self.globals)
return compile_key_type(**fields)
@dataclass(frozen=True)
@@ -186,6 +205,10 @@ _CMP_OPS: dict[type[ast.cmpop], Callable[[Any, Any], bool]] = {
ast.LtE: operator.le,
ast.Gt: operator.gt,
ast.GtE: operator.ge,
ast.In: lambda left, right: left in right,
ast.NotIn: lambda left, right: left not in right,
ast.Is: operator.is_,
ast.IsNot: operator.is_not,
}
@@ -200,8 +223,9 @@ def _dispatch_expr_error(node: ast.AST, reason: str) -> ValueError:
return ValueError(
f"{reason}: {_dispatch_expr_source(node)}. "
"Supported dispatch expressions are names, constants, attributes, "
"tuple/list literals, conditional expressions, comparisons, boolean "
"operators, unary not/minus, arithmetic, and calls without **kwargs."
"subscriptions, tuple/list literals, conditional expressions, "
"comparisons, boolean operators, unary not/minus, arithmetic, and "
"calls without **kwargs."
)
@@ -225,6 +249,8 @@ class _DispatchExprEvaluator(ast.NodeVisitor):
return self.values[node.id]
if node.id in self.globals:
return self.globals[node.id]
if hasattr(builtins, node.id):
return getattr(builtins, node.id)
raise _dispatch_expr_error(node, f"Unknown dispatch name '{node.id}'")
def visit_Constant(self, node: ast.Constant) -> Any:
@@ -299,6 +325,9 @@ class _DispatchExprEvaluator(ast.NodeVisitor):
def visit_Attribute(self, node: ast.Attribute) -> Any:
return getattr(self.visit(node.value), node.attr)
def visit_Subscript(self, node: ast.Subscript) -> Any:
return self.visit(node.value)[self.visit(node.slice)]
def get_ast_full_name(node: ast.AST) -> str | None:
if isinstance(node, ast.Name):
@@ -310,15 +339,23 @@ def get_ast_full_name(node: ast.AST) -> str | None:
return None
def get_function_source_node(fn: Callable[..., Any]) -> ast.FunctionDef:
def get_function_source_node(fn: Callable[..., Any]) -> ast.FunctionDef | ast.Lambda:
source_fn = getattr(fn, "fn", fn)
source = textwrap.dedent(inspect.getsource(source_fn))
tree = ast.parse(source)
function_defs = [node for node in tree.body if isinstance(node, ast.FunctionDef)]
if len(function_defs) != 1:
name = getattr(source_fn, "__name__", type(source_fn).__name__)
raise ValueError(f"Expected one function in {name}, found {len(function_defs)}")
return function_defs[0]
if len(function_defs) == 1:
return function_defs[0]
lambdas = [node for node in ast.walk(tree) if isinstance(node, ast.Lambda)]
if len(lambdas) == 1:
return lambdas[0]
name = getattr(source_fn, "__name__", type(source_fn).__name__)
raise ValueError(
f"Expected one function or lambda in {name}, found "
f"{len(function_defs)} functions and {len(lambdas)} lambdas"
)
def _eval_dispatch_expr(
@@ -349,8 +386,11 @@ def _collect_input_names(
def _collect_expression_body(
fn: Callable[..., Any],
function_def: ast.FunctionDef,
function_def: ast.FunctionDef | ast.Lambda,
) -> tuple[list[tuple[str, ast.AST]], ast.AST]:
if isinstance(function_def, ast.Lambda):
return [], function_def.body
local_exprs: list[tuple[str, ast.AST]] = []
for statement in function_def.body:
if (
@@ -421,6 +461,10 @@ def _trace_compile_key_dispatch(
source_fn = getattr(fn, "__func__", fn)
globals_ = source_fn.__globals__
function_def = get_function_source_node(fn)
if isinstance(function_def, ast.Lambda):
raise _dispatch_expr_error(
function_def, "Dispatch must be a function definition"
)
local_exprs, return_expr = _collect_expression_body(fn, function_def)
if not isinstance(return_expr, ast.Call):
@@ -431,13 +475,37 @@ def _trace_compile_key_dispatch(
field_exprs: list[tuple[str, ast.AST]] = []
defaults, candidate_names = _function_trace_inputs(fn)
# Fields captured by dispatch **kwargs are forwarded, not AST-evaluated.
signature = inspect.signature(fn)
variadic_keyword = next(
(
name
for name, parameter in signature.parameters.items()
if parameter.kind is inspect.Parameter.VAR_KEYWORD
),
None,
)
candidate_names.discard(variadic_keyword)
input_names: set[str] = set()
local_names = {name for name, _ in local_exprs}
for _, expr in local_exprs:
input_names.update(_collect_input_names(expr, candidate_names))
named_parameters: frozenset[str] | None = None
for keyword in return_expr.keywords:
# CompileKey may unpack only that **kwargs parameter, once.
if keyword.arg is None:
raise ValueError(f"{fn.__name__} cannot use **kwargs in CompileKey")
if (
named_parameters is not None
or variadic_keyword is None
or not isinstance(keyword.value, ast.Name)
or keyword.value.id != variadic_keyword
):
raise ValueError(
f"{fn.__name__} may unpack only its own **kwargs parameter "
"once in CompileKey"
)
named_parameters = frozenset(candidate_names)
continue
field_exprs.append((keyword.arg, keyword.value))
input_names.update(
_collect_input_names(keyword.value, candidate_names, local_names)
@@ -448,6 +516,7 @@ def _trace_compile_key_dispatch(
tuple(field_exprs),
globals_,
frozenset(input_names),
named_parameters,
defaults,
)
@@ -482,10 +551,32 @@ class VllmJitKernel(Generic[CompileKeyT], ABC):
CompileKey: type[CompileKeyT]
def __init__(self) -> None:
self.compile_key_dispatch_trace = _trace_compile_key_dispatch(self.dispatch)
self._dispatch_trace = _trace_compile_key_dispatch(self.dispatch)
self._compiled_cache: dict[Any, Any] = {}
def compile_key(self, kwargs: Mapping[str, Any]) -> CompileKeyT:
return self.compile_key_dispatch_trace.compile_key(self.CompileKey, kwargs)
return self._dispatch_trace.compile_key(self.CompileKey, kwargs)
def _get_or_compile(
self,
compile_key: CompileKeyT,
*,
runtime_context: Mapping[str, Any] | None = None,
) -> Any:
"""Return a cached executor, compiling it on a monitored cache miss."""
if compile_key not in self._compiled_cache:
self.compile(compile_key)
try:
return self._compiled_cache[compile_key]
except KeyError as exc:
details = [f"compile_key={compile_key!r}"]
if runtime_context:
details.append(f"runtime_context={dict(runtime_context)!r}")
raise RuntimeError(
f"{type(self).__name__}.compile(...) did not cache its JIT "
f"executor ({', '.join(details)})"
) from exc
def _trace_dispatch(
self, dispatch: CompileKeyDispatchFn[CompileKeyT]
@@ -506,7 +597,11 @@ class VllmJitKernel(Generic[CompileKeyT], ABC):
predicate_trace = (
_trace_warmup_predicate(_when) if _when is not None else None
)
input_names = compile_key_dispatch_trace.input_names
# Unmatched **kwargs fields also belong to the expansion space.
available_names = set(kwargs).union(
*(group.rows[0] for group in input_groups)
)
input_names = compile_key_dispatch_trace.input_names_for(available_names)
if predicate_trace is not None:
input_names = input_names | predicate_trace.input_names
expanded_input_groups = tuple(
@@ -565,7 +660,105 @@ class VllmJitKernel(Generic[CompileKeyT], ABC):
"""Compile one warmup key."""
raise NotImplementedError
def register_warmup(self, *args: Any, **kwargs: Any) -> None:
"""Register this kernel with the active runner's warmup registry."""
JitWarmupRegistry.register(self, *args, **kwargs)
def warmup(self, *args: Any, **kwargs: Any) -> None:
"""Compile this kernel's warmup keys."""
for compile_key in self.get_warmup_keys(*args, **kwargs):
self.compile(compile_key)
class JitWarmupRegistry:
"""Collect and compile JIT kernels selected during runner setup."""
_active: ContextVar[JitWarmupRegistry | None] = ContextVar(
"active_jit_warmup_registry",
default=None,
)
def __init__(self, vllm_config: Any) -> None:
self.vllm_config = vllm_config
self._registrations: dict[
VllmJitKernel[Any],
list[tuple[tuple[Any, ...], dict[str, Any]]],
] = {}
@contextmanager
def activate(self) -> Iterator[None]:
"""Collect registrations made in this context."""
token = self._active.set(self)
try:
yield
finally:
self._active.reset(token)
@classmethod
def register(
cls,
kernel: VllmJitKernel[Any],
*args: Any,
**kwargs: Any,
) -> None:
"""Register a kernel with the active registry, if one exists."""
registry = cls._active.get()
if registry is not None:
registry._add(kernel, args, kwargs)
def _add(
self,
kernel: VllmJitKernel[Any],
args: tuple[Any, ...],
kwargs: dict[str, Any],
) -> None:
registrations = self._registrations.setdefault(kernel, [])
if (
not args
and not kwargs
and any(
not registered_args and not registered_kwargs
for registered_args, registered_kwargs in registrations
)
):
return
registrations.append((args, kwargs))
def __len__(self) -> int:
return sum(len(registrations) for registrations in self._registrations.values())
def warmup(self) -> None:
"""Expand registrations and compile each wrapper/key pair once."""
from tqdm import tqdm
from vllm.distributed import is_global_first_rank
kernel_items: list[tuple[VllmJitKernel[Any], dict[Any, None]]] = []
for kernel, registrations in self._registrations.items():
compile_keys: dict[Any, None] = {}
for args, kwargs in registrations:
if not args and not kwargs:
args = (self.vllm_config,)
for compile_key in kernel.get_warmup_keys(*args, **kwargs):
compile_keys[compile_key] = None
if compile_keys:
kernel_items.append((kernel, compile_keys))
if not kernel_items:
return
total_keys = sum(len(compile_keys) for _, compile_keys in kernel_items)
with tqdm(
kernel_items,
desc=f"JIT kernel warmup ({total_keys} compile keys)",
disable=not is_global_first_rank(),
dynamic_ncols=True,
unit="kernel",
) as progress:
for kernel, compile_keys in progress:
progress.set_postfix_str(
f"{kernel.__class__.__name__} ({len(compile_keys)} keys)",
refresh=False,
)
for compile_key in compile_keys:
kernel.compile(compile_key)
@@ -12,6 +12,42 @@ from vllm.model_executor.warmup.jit_warmup import (
)
def triton_scalar_specialization_rep(value: int) -> int:
"""Return an integer with the same default Triton JIT specialization.
For an ordinary integer argument, Triton's cache key contains its inferred
type (``i32``, ``i64``, or ``u64``) and one of three value classes:
* ``1`` is specialized as the exact constant ``1``.
* Multiples of 16 receive a ``tt.divisibility = 16`` attribute.
* All other values have no value specialization.
Warmup only needs one concrete value for each cache-key class. This helper
returns ``1`` for the exact-one class and otherwise returns a divisible or
generic representative while preserving the inferred integer type.
This applies only to non-``constexpr`` integer arguments using Triton's
default specialization. Do not use it for arguments listed in
``do_not_specialize`` or ``do_not_specialize_on_alignment``.
"""
if value == 1:
return 1
if -(1 << 31) <= value < (1 << 31):
divisible_rep = 16
generic_rep = 2
elif -(1 << 63) <= value < (1 << 63):
divisible_rep = 1 << 31
generic_rep = (1 << 31) + 1
elif 0 <= value < (1 << 64):
divisible_rep = 1 << 63
generic_rep = (1 << 63) + 1
else:
raise OverflowError(f"Integer {value} is outside Triton's scalar range")
return divisible_rep if value % 16 == 0 else generic_rep
@dataclass(frozen=True)
class TritonWarmupTensor:
# Compile-only tensor descriptor for Triton pointer specialization.
@@ -167,6 +203,8 @@ def trace_triton_kernel_specialization_args(
kernel: Callable[..., Any],
) -> tuple[str, ...]:
function_def = get_function_source_node(kernel)
if not isinstance(function_def, ast.FunctionDef):
raise ValueError("Expected Triton kernel to be defined as a function")
source_fn = getattr(kernel, "fn", kernel)
arg_names = tuple(inspect.signature(source_fn).parameters)
constexpr_args = _triton_constexpr_arg_names(kernel, function_def, arg_names)
+17 -6
View File
@@ -6,6 +6,7 @@ This is useful specifically for JIT'ed kernels as we don't want JIT'ing to
happen during model execution.
"""
import time
from typing import TYPE_CHECKING
import torch
@@ -36,9 +37,6 @@ from vllm.model_executor.warmup.qwen_triton_warmup import qwen_triton_warmup
from vllm.model_executor.warmup.sparse_mla_triton_warmup import (
sparse_mla_triton_warmup,
)
from vllm.model_executor.warmup.v1_block_table_warmup import (
warm_v1_block_table_kernels,
)
from vllm.platforms import current_platform
from vllm.utils.deep_gemm import is_deep_gemm_supported
from vllm.utils.flashinfer import has_flashinfer
@@ -103,15 +101,28 @@ def kernel_warmup(worker: "Worker", *, process_local_only: bool = False):
)
if not worker.use_v2_model_runner:
# Pooling models do not use the generation slot-mapping path.
if not worker.model_runner.is_pooling_model:
warm_v1_block_table_kernels(worker.model_runner)
# The KV-block zeroing kernel is driven by the scheduler's
# `new_block_ids_to_zero`, so no dummy run ever reaches it.
zeroer = getattr(worker.model_runner, "_kv_block_zeroer", None)
if zeroer is not None:
zeroer.warmup(worker.model_runner.kv_cache_config.num_blocks)
if worker.vllm_config.kernel_config.enable_jit_warmup:
logger.info("JIT kernel warmup starting.")
jit_warmup_start = time.perf_counter()
try:
worker.model_runner.jit_warmup_registry.warmup()
except Exception:
logger.exception(
"JIT kernel warmup failed after %.2fs.",
time.perf_counter() - jit_warmup_start,
)
raise
logger.info(
"JIT kernel warmup finished in %.2fs.",
time.perf_counter() - jit_warmup_start,
)
qwen_triton_warmup(worker.model_runner, worker.vllm_config.model_config)
compilation_config = worker.vllm_config.compilation_config
@@ -1,29 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Warm up v1 block-table Triton kernels."""
from typing import TYPE_CHECKING
import torch
if TYPE_CHECKING:
from vllm.v1.worker.gpu_model_runner import GPUModelRunner
_SLOT_MAPPING_WARMUP_TOKENS = 8
def warm_v1_block_table_kernels(runner: "GPUModelRunner") -> None:
"""JIT-compile ``_compute_slot_mapping_kernel`` for the real block tables."""
device = runner.device
block_table = runner.input_batch.block_table
num_tokens = min(
_SLOT_MAPPING_WARMUP_TOKENS,
runner.scheduler_config.max_num_batched_tokens,
)
if num_tokens <= 0:
return
query_start_loc = torch.tensor([0, num_tokens], dtype=torch.int32, device=device)
positions = torch.arange(num_tokens, dtype=torch.int64, device=device)
block_table.compute_slot_mapping(1, query_start_loc, positions)
+151 -64
View File
@@ -2,13 +2,22 @@
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import math
from dataclasses import dataclass
from enum import Enum
from typing import Any
import numpy as np
import torch
from vllm.distributed import get_dcp_group, get_pcp_group
from vllm.logger import init_logger
from vllm.model_executor.warmup.jit_warmup import (
VllmJitKernel,
)
from vllm.model_executor.warmup.jit_warmup_triton_helper import (
TritonWarmupTensor,
triton_scalar_specialization_rep,
)
from vllm.triton_utils import tl, triton
from vllm.utils.math_utils import cdiv
from vllm.v1.attention.backends.utils import PAD_SLOT_ID
@@ -134,6 +143,16 @@ class BlockTable:
self.dcp_rank = 0
self.cp_kv_cache_interleave_size = cp_kv_cache_interleave_size
self.slot_mapping_mode = slot_mapping_mode
if self.slot_mapping_mode == SlotMappingMode.TOKEN_TO_KV_SLOT:
_COMPUTE_SLOT_MAPPING_KERNEL.register_warmup(
kv_cache_block_size=self.kv_cache_block_size,
blocks_per_kv_block=self.blocks_per_kv_block,
total_cp_world_size=self.dcp_world_size,
total_cp_rank=self.dcp_rank,
cp_kv_cache_interleave_size=self.cp_kv_cache_interleave_size,
block_table_stride=self.block_table.gpu.stride(0),
block_size=self.block_size,
)
def append_row(
self,
@@ -192,7 +211,8 @@ class BlockTable:
return
assert self.slot_mapping_mode == SlotMappingMode.TOKEN_TO_KV_SLOT
_compute_slot_mapping_kernel[(num_reqs + 1,)](
_COMPUTE_SLOT_MAPPING_KERNEL(
num_reqs,
num_tokens,
self.max_num_batched_tokens,
query_start_loc,
@@ -201,13 +221,11 @@ class BlockTable:
self.block_table.gpu.stride(0),
self.block_size,
self.slot_mapping.gpu,
KV_CACHE_BLOCK_SIZE=self.kv_cache_block_size,
BLOCKS_PER_KV_BLOCK=self.blocks_per_kv_block,
TOTAL_CP_WORLD_SIZE=self.dcp_world_size,
TOTAL_CP_RANK=self.dcp_rank,
CP_KV_CACHE_INTERLEAVE_SIZE=self.cp_kv_cache_interleave_size,
PAD_ID=PAD_SLOT_ID,
BLOCK_SIZE=1024,
self.kv_cache_block_size,
self.blocks_per_kv_block,
self.dcp_world_size,
self.dcp_rank,
self.cp_kv_cache_interleave_size,
)
def commit_block_table(self, num_reqs: int) -> None:
@@ -376,67 +394,136 @@ class MultiGroupBlockTable:
return self.block_tables[idx]
@triton.jit(do_not_specialize=["num_tokens", "max_num_tokens"])
def _compute_slot_mapping_kernel(
num_tokens,
max_num_tokens,
query_start_loc_ptr, # [num_reqs + 1], int32
positions_ptr, # [num_tokens], int64
block_table_ptr, # [max_num_reqs, max_num_blocks_per_req], int32 (flat)
block_table_stride, # max_num_blocks_per_req
block_size,
slot_mapping_ptr, # [max_num_tokens], int64
KV_CACHE_BLOCK_SIZE: tl.constexpr,
BLOCKS_PER_KV_BLOCK: tl.constexpr,
TOTAL_CP_WORLD_SIZE: tl.constexpr,
TOTAL_CP_RANK: tl.constexpr,
CP_KV_CACHE_INTERLEAVE_SIZE: tl.constexpr,
PAD_ID: tl.constexpr,
BLOCK_SIZE: tl.constexpr,
):
req_idx = tl.program_id(0)
class ComputeSlotMappingKernel(VllmJitKernel["ComputeSlotMappingKernel.CompileKey"]):
triton_block_size = 1024
if req_idx == tl.num_programs(0) - 1:
# Pad remaining slots for CUDA graph compatibility.
for i in range(num_tokens, max_num_tokens, BLOCK_SIZE):
@dataclass(frozen=True)
class CompileKey:
kv_cache_block_size: int
blocks_per_kv_block: int
total_cp_world_size: int
total_cp_rank: int
cp_kv_cache_interleave_size: int
block_table_stride: int
block_size: int
@staticmethod
@triton.jit(do_not_specialize=["num_tokens", "max_num_tokens"])
def kernel(
num_tokens,
max_num_tokens,
query_start_loc_ptr, # [num_reqs + 1], int32
positions_ptr, # [num_tokens], int64
block_table_ptr, # [max_num_reqs, max_num_blocks_per_req], int32 (flat)
block_table_stride, # max_num_blocks_per_req
block_size,
slot_mapping_ptr, # [max_num_tokens], int64
KV_CACHE_BLOCK_SIZE: tl.constexpr,
BLOCKS_PER_KV_BLOCK: tl.constexpr,
TOTAL_CP_WORLD_SIZE: tl.constexpr,
TOTAL_CP_RANK: tl.constexpr,
CP_KV_CACHE_INTERLEAVE_SIZE: tl.constexpr,
PAD_ID: tl.constexpr,
BLOCK_SIZE: tl.constexpr,
):
req_idx = tl.program_id(0)
if req_idx == tl.num_programs(0) - 1:
# Pad remaining slots for CUDA graph compatibility.
for i in range(num_tokens, max_num_tokens, BLOCK_SIZE):
offsets = i + tl.arange(0, BLOCK_SIZE)
tl.store(
slot_mapping_ptr + offsets,
PAD_ID,
mask=offsets < max_num_tokens,
)
return
start_idx = tl.load(query_start_loc_ptr + req_idx).to(tl.int64)
end_idx = tl.load(query_start_loc_ptr + req_idx + 1).to(tl.int64)
virtual_block_size = KV_CACHE_BLOCK_SIZE * TOTAL_CP_WORLD_SIZE
row_offset = req_idx * block_table_stride
for i in range(start_idx, end_idx, BLOCK_SIZE):
offsets = i + tl.arange(0, BLOCK_SIZE)
tl.store(
slot_mapping_ptr + offsets,
PAD_ID,
mask=offsets < max_num_tokens,
mask = offsets < end_idx
pos = tl.load(positions_ptr + offsets, mask=mask, other=0)
virtual_block_indices = pos // virtual_block_size
virtual_block_offsets = pos - virtual_block_indices * virtual_block_size
is_local = (
virtual_block_offsets // CP_KV_CACHE_INTERLEAVE_SIZE
) % TOTAL_CP_WORLD_SIZE == TOTAL_CP_RANK
local_block_offsets = (
virtual_block_offsets
// (TOTAL_CP_WORLD_SIZE * CP_KV_CACHE_INTERLEAVE_SIZE)
) * CP_KV_CACHE_INTERLEAVE_SIZE + (
virtual_block_offsets % CP_KV_CACHE_INTERLEAVE_SIZE
)
return
start_idx = tl.load(query_start_loc_ptr + req_idx).to(tl.int64)
end_idx = tl.load(query_start_loc_ptr + req_idx + 1).to(tl.int64)
block_indices = (
virtual_block_indices * BLOCKS_PER_KV_BLOCK
+ local_block_offsets // block_size
)
block_numbers = tl.load(
block_table_ptr + row_offset + block_indices,
mask=mask & is_local,
other=0,
).to(tl.int64)
slot_offsets = local_block_offsets % block_size
slot_ids = block_numbers * block_size + slot_offsets
slot_ids = tl.where(is_local, slot_ids, PAD_ID)
tl.store(slot_mapping_ptr + offsets, slot_ids, mask=mask)
virtual_block_size = KV_CACHE_BLOCK_SIZE * TOTAL_CP_WORLD_SIZE
row_offset = req_idx * block_table_stride
for i in range(start_idx, end_idx, BLOCK_SIZE):
offsets = i + tl.arange(0, BLOCK_SIZE)
mask = offsets < end_idx
pos = tl.load(positions_ptr + offsets, mask=mask, other=0)
virtual_block_indices = pos // virtual_block_size
virtual_block_offsets = pos - virtual_block_indices * virtual_block_size
is_local = (
virtual_block_offsets // CP_KV_CACHE_INTERLEAVE_SIZE
) % TOTAL_CP_WORLD_SIZE == TOTAL_CP_RANK
local_block_offsets = (
virtual_block_offsets // (TOTAL_CP_WORLD_SIZE * CP_KV_CACHE_INTERLEAVE_SIZE)
) * CP_KV_CACHE_INTERLEAVE_SIZE + (
virtual_block_offsets % CP_KV_CACHE_INTERLEAVE_SIZE
def dispatch( # type: ignore[override]
self,
*,
block_table_stride: int,
block_size: int,
**compile_key_fields: int,
) -> CompileKey:
return self.CompileKey(
**compile_key_fields,
block_table_stride=triton_scalar_specialization_rep(block_table_stride),
block_size=triton_scalar_specialization_rep(block_size),
)
block_indices = (
virtual_block_indices * BLOCKS_PER_KV_BLOCK
+ local_block_offsets // block_size
def get_warmup_keys(self, **dispatch_kwargs: int) -> list[CompileKey]:
return self._trace_dispatch(self.dispatch)(**dispatch_kwargs)
def compile(self, compile_key: CompileKey) -> None:
warmup = getattr(self.kernel, "warmup", None)
assert warmup is not None
int32_ptr = TritonWarmupTensor(torch.int32)
int64_ptr = TritonWarmupTensor(torch.int64)
warmup(
2, # arbitrary, num_tokens in do_not_specialize
2, # arbitrary, max_num_tokens in do_not_specialize
int32_ptr,
int64_ptr,
int32_ptr,
compile_key.block_table_stride,
compile_key.block_size,
int64_ptr,
KV_CACHE_BLOCK_SIZE=compile_key.kv_cache_block_size,
BLOCKS_PER_KV_BLOCK=compile_key.blocks_per_kv_block,
TOTAL_CP_WORLD_SIZE=compile_key.total_cp_world_size,
TOTAL_CP_RANK=compile_key.total_cp_rank,
CP_KV_CACHE_INTERLEAVE_SIZE=compile_key.cp_kv_cache_interleave_size,
PAD_ID=PAD_SLOT_ID,
BLOCK_SIZE=self.triton_block_size,
grid=(2,),
)
block_numbers = tl.load(
block_table_ptr + row_offset + block_indices,
mask=mask & is_local,
other=0,
).to(tl.int64)
slot_offsets = local_block_offsets % block_size
slot_ids = block_numbers * block_size + slot_offsets
slot_ids = tl.where(is_local, slot_ids, PAD_ID)
tl.store(slot_mapping_ptr + offsets, slot_ids, mask=mask)
def __call__(
self,
num_reqs: int,
*args: Any,
) -> None:
self.kernel[(num_reqs + 1,)](
*args,
PAD_ID=PAD_SLOT_ID,
BLOCK_SIZE=self.triton_block_size,
)
_COMPUTE_SLOT_MAPPING_KERNEL = ComputeSlotMappingKernel()
+1 -1
View File
@@ -74,7 +74,7 @@ class CPUModelRunner(GPUModelRunner):
import vllm.v1.worker.block_table
vllm.v1.worker.block_table._compute_slot_mapping_kernel = (
vllm.v1.worker.block_table._COMPUTE_SLOT_MAPPING_KERNEL.kernel = (
cpu_tl.compute_slot_mapping_kernel
)
+2
View File
@@ -51,6 +51,7 @@ from vllm.model_executor.offloader import (
get_offloader,
set_offloader,
)
from vllm.model_executor.warmup.jit_warmup import JitWarmupRegistry
from vllm.multimodal import MULTIMODAL_REGISTRY
from vllm.multimodal.encoder_budget import (
MultiModalBudget,
@@ -170,6 +171,7 @@ class GPUModelRunner(LoRAModelRunnerMixin):
self.speculative_config is not None and self.speculative_config.use_dspark()
)
self.observability_config = vllm_config.observability_config
self.jit_warmup_registry = JitWarmupRegistry(vllm_config)
self.device = device
self.dtype = self.model_config.dtype
+53 -46
View File
@@ -100,6 +100,7 @@ from vllm.model_executor.offloader import (
get_offloader,
set_offloader,
)
from vllm.model_executor.warmup.jit_warmup import JitWarmupRegistry
from vllm.multimodal import MULTIMODAL_REGISTRY
from vllm.multimodal.encoder_budget import MultiModalBudget
from vllm.multimodal.inputs import (
@@ -516,6 +517,7 @@ class GPUModelRunner(
self.scheduler_config = vllm_config.scheduler_config
self.speculative_config = vllm_config.speculative_config
self.observability_config = vllm_config.observability_config
self.jit_warmup_registry = JitWarmupRegistry(vllm_config)
model_config = self.model_config
cache_config = self.cache_config
@@ -750,34 +752,37 @@ class GPUModelRunner(
self._init_kernel_block_sizes = [placeholder_block_size]
self._init_max_num_blocks = [placeholder_max_num_blocks]
self._init_slot_mapping_modes = [SlotMappingMode.TOKEN_TO_KV_SLOT]
self.input_batch = InputBatch(
max_num_reqs=self.max_num_reqs,
# We need to use the encoder length for encoder-decoder
# because of KV cache for cross-attention.
max_model_len=max(self.max_model_len, self.max_encoder_len),
max_num_batched_tokens=self.max_num_tokens,
device=self.device,
vocab_size=self.model_config.get_vocab_size(),
block_sizes=[placeholder_block_size],
kernel_block_sizes=[placeholder_block_size],
max_num_blocks_per_req=[placeholder_max_num_blocks],
num_spec_tokens=self.num_spec_tokens,
logitsprocs=build_logitsprocs(
self.vllm_config,
self.device,
PIN_MEMORY,
self.is_pooling_model,
custom_logitsprocs,
),
# We currently don't know whether a particular custom logits processor
# uses output token ids so we set this conservatively. Thinking-budget
# tracking is requested dynamically when a budgeted request is in the batch.
logitsprocs_need_output_token_ids=bool(custom_logitsprocs),
is_pooling_model=self.is_pooling_model,
cp_kv_cache_interleave_size=self.parallel_config.cp_kv_cache_interleave_size,
reasoning_config=self.vllm_config.reasoning_config,
use_replayssm=self.cache_config.use_replayssm,
)
# Capture warmup providers registered by the initial placeholder InputBatch
with self.jit_warmup_registry.activate():
self.input_batch = InputBatch(
max_num_reqs=self.max_num_reqs,
# We need to use the encoder length for encoder-decoder
# because of KV cache for cross-attention.
max_model_len=max(self.max_model_len, self.max_encoder_len),
max_num_batched_tokens=self.max_num_tokens,
device=self.device,
vocab_size=self.model_config.get_vocab_size(),
block_sizes=[placeholder_block_size],
kernel_block_sizes=[placeholder_block_size],
max_num_blocks_per_req=[placeholder_max_num_blocks],
num_spec_tokens=self.num_spec_tokens,
logitsprocs=build_logitsprocs(
self.vllm_config,
self.device,
PIN_MEMORY,
self.is_pooling_model,
custom_logitsprocs,
),
# We currently don't know whether a particular custom logits processor
# uses output token ids so we set this conservatively. Thinking-budget
# tracking is requested dynamically when a budgeted request is in the
# batch.
logitsprocs_need_output_token_ids=bool(custom_logitsprocs),
is_pooling_model=self.is_pooling_model,
cp_kv_cache_interleave_size=self.parallel_config.cp_kv_cache_interleave_size,
reasoning_config=self.vllm_config.reasoning_config,
use_replayssm=self.cache_config.use_replayssm,
)
# Separate cuda stream for overlapping transfer of sampled token ids from
# GPU to CPU when async scheduling is enabled.
@@ -7416,24 +7421,26 @@ class GPUModelRunner(
self._init_kernel_block_sizes = kernel_block_sizes
self._init_max_num_blocks = max_num_blocks
self._init_slot_mapping_modes = slot_mapping_modes
self.input_batch = InputBatch(
max_num_reqs=self.max_num_reqs,
max_model_len=max_model_len,
max_num_batched_tokens=self.max_num_tokens,
device=self.device,
vocab_size=self.model_config.get_vocab_size(),
block_sizes=block_sizes,
kernel_block_sizes=kernel_block_sizes,
max_num_blocks_per_req=max_num_blocks,
num_spec_tokens=self.num_spec_tokens,
logitsprocs=self.input_batch.logitsprocs,
logitsprocs_need_output_token_ids=self.input_batch.logitsprocs_need_output_token_ids,
is_pooling_model=self.is_pooling_model,
cp_kv_cache_interleave_size=self.parallel_config.cp_kv_cache_interleave_size,
reasoning_config=self.vllm_config.reasoning_config,
use_replayssm=self.cache_config.use_replayssm,
slot_mapping_modes=slot_mapping_modes,
)
# Capture warmup providers registered after final KV-cache geometry is known
with self.jit_warmup_registry.activate():
self.input_batch = InputBatch(
max_num_reqs=self.max_num_reqs,
max_model_len=max_model_len,
max_num_batched_tokens=self.max_num_tokens,
device=self.device,
vocab_size=self.model_config.get_vocab_size(),
block_sizes=block_sizes,
kernel_block_sizes=kernel_block_sizes,
max_num_blocks_per_req=max_num_blocks,
num_spec_tokens=self.num_spec_tokens,
logitsprocs=self.input_batch.logitsprocs,
logitsprocs_need_output_token_ids=self.input_batch.logitsprocs_need_output_token_ids,
is_pooling_model=self.is_pooling_model,
cp_kv_cache_interleave_size=self.parallel_config.cp_kv_cache_interleave_size,
reasoning_config=self.vllm_config.reasoning_config,
use_replayssm=self.cache_config.use_replayssm,
slot_mapping_modes=slot_mapping_modes,
)
assert self._init_block_sizes == block_sizes, (
f"InputBatch block_sizes {self._init_block_sizes} != "