[3/N][Feat][Perf] Add new warmup infrastructure for JITs. Add provider registry and orchestration for JIT warmup (#50174)
Signed-off-by: LopezCastroRoberto <rocastro@redhat.com> Co-authored-by: Codex <codex@openai.com>
This commit is contained in:
co-authored by
Codex
parent
7075ddac28
commit
49905ad94d
@@ -71,6 +71,7 @@ uv pip install -e . --no-build-isolation
|
||||
For more details about installing from source and installing for other hardware, check out the [installation instructions](../getting_started/installation/README.md) for your hardware and head to the "Build wheel from source" section.
|
||||
|
||||
For an optimized workflow when iterating on C++/CUDA kernels, see the [Incremental Compilation Workflow](./incremental_build.md) for recommendations.
|
||||
For JIT kernel warmup conventions, see [JIT Kernel Warmup](./jit_kernel_warmup.md).
|
||||
|
||||
!!! tip
|
||||
vLLM is compatible with Python versions 3.10 to 3.13. However, vLLM's default [Dockerfile](../../docker/Dockerfile) ships with Python 3.12 and tests in CI (except `mypy`) are run with Python 3.12.
|
||||
|
||||
@@ -0,0 +1,308 @@
|
||||
# JIT Kernel Warmup
|
||||
|
||||
vLLM uses JIT-generated kernels from Triton, CuTeDSL, TileLang, and other backends. This contract makes their required specializations available during startup, before the first request, by warming the kernel's **compile-key space** without dummy runtime launches or real tensor allocation.
|
||||
|
||||
Use it when adding a warmable JIT kernel or migrating an existing warmup path.
|
||||
|
||||
## In This Guide
|
||||
|
||||
- [1. Quickstart](#1-quickstart): for contributors adding or migrating a warmable kernel.
|
||||
- [2. Search-Space Reference](#2-search-space-reference): additional details regarding warmup input expansion and traced dispatch rules.
|
||||
|
||||
## 1. Quickstart
|
||||
|
||||
Each warmable kernel defines its compile-key mapping and compile-only entry point beside its normal runtime implementation. The startup registry then warms only the wrappers selected by the current engine configuration.
|
||||
|
||||
### Define the Kernel Wrapper
|
||||
|
||||
Here, a **kernel wrapper** (or just **wrapper**) is an instance of a concrete `VllmJitKernel` subclass.
|
||||
|
||||
Expose one wrapper near the kernel's normal runtime entry point. Prefer this shape:
|
||||
|
||||
```python
|
||||
class MyKernel(VllmJitKernel["MyKernel.CompileKey"]):
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CompileKey:
|
||||
...
|
||||
|
||||
@staticmethod
|
||||
def kernel(...):
|
||||
...
|
||||
|
||||
def dispatch(self, ...) -> CompileKey:
|
||||
return self.CompileKey(...)
|
||||
|
||||
def get_warmup_keys(self, ...) -> list[CompileKey]:
|
||||
return self._trace_dispatch(self.dispatch)(...)
|
||||
|
||||
def compile(self, compile_key: CompileKey) -> None:
|
||||
...
|
||||
|
||||
def __call__(self, ...):
|
||||
return self.kernel(...)
|
||||
|
||||
|
||||
MY_KERNEL = MyKernel()
|
||||
```
|
||||
|
||||
`CompileKey`, `dispatch(...)`, and `get_warmup_keys(...)` are backend-agnostic. Backend-specific behavior belongs in `kernel(...)`, `compile(...)`, and `__call__(...)`.
|
||||
|
||||
The module-level singleton should be used by warmup and by the runtime call path. This keeps dispatch behavior shared instead of duplicated.
|
||||
|
||||
`VllmJitKernel.warmup(...)` compiles every key returned by `get_warmup_keys(...)`; wrappers should not reimplement it.
|
||||
|
||||
### Choose Compile-Key Fields
|
||||
|
||||
`CompileKey` must be frozen and hashable. Include only fields on which the backend specializes, such as tile sizes, head dimensions, dtypes, pointer alignment classes, or backend selectors; exclude runtime-only values. When unsure, inspect the backend cache key, specialization arguments, or verbose JIT-monitor output.
|
||||
|
||||
### Generate Warmup Keys
|
||||
|
||||
Use `_trace_dispatch(self.dispatch)` to describe representative inputs. The tracer maps them through the same specialization logic and deduplicates equal keys:
|
||||
|
||||
```python
|
||||
def get_warmup_keys(self, vllm_config: VllmConfig) -> list[CompileKey]:
|
||||
max_tokens = vllm_config.scheduler_config.max_num_batched_tokens
|
||||
return self._trace_dispatch(self.dispatch)(
|
||||
num_tokens=WarmupIntRange(1, max_tokens + 1),
|
||||
)
|
||||
```
|
||||
|
||||
Use independent ranges or alternatives for cartesian products, `zip_inputs(...)` for coupled rows, and `_when` for validity constraints. The complete syntax is documented in [Search-Space Reference](#2-search-space-reference).
|
||||
|
||||
### Compile Without Launching
|
||||
|
||||
`compile(compile_key)` means "make this specialization available". Depending on the backend, that may compile from source, call a compile-only API, load an already-built artifact, or compile on cache miss.
|
||||
|
||||
`compile(...)` should not launch a real inference workload or allocate real tensors. Each DSL should expose fake tensor/spec descriptors suitable for compilation only.
|
||||
|
||||
### Register the Selected Wrapper
|
||||
|
||||
Register the wrapper where the runtime implementation is selected:
|
||||
|
||||
```python
|
||||
MY_KERNEL.register_warmup()
|
||||
```
|
||||
|
||||
Registration records metadata only. It does not compile or launch the kernel. Repeated registrations from equivalent layers are allowed and deduplicated later.
|
||||
|
||||
### Review Checklist
|
||||
|
||||
- Warm actual compile keys rather than representative non-key inputs.
|
||||
- Keep specialization mapping in `dispatch(...)` instead of duplicating it in warmup code.
|
||||
- Use fake tensors or backend compile-only descriptors; never perform a dummy runtime launch.
|
||||
- Keep registration metadata-only so model construction remains cheap and side-effect free.
|
||||
- Compile registered kernels only through `kernel_warmup()`.
|
||||
- Keep runtime execution and startup compilation separate and easy to review.
|
||||
- Use one module-level wrapper instance for registration and runtime calls.
|
||||
|
||||
## 2. Search-Space Reference
|
||||
|
||||
### How Tracing Works
|
||||
|
||||
`_trace_dispatch(...)` expands the inputs declared by `get_warmup_keys(...)`. Each concrete combination becomes a `dispatch_values` mapping from input names to selected values. `_when` may reject that mapping; otherwise the tracer evaluates `dispatch(...)` to construct one `CompileKey`. Equal keys are deduplicated after all combinations are evaluated.
|
||||
|
||||
One call to `dispatch(...)` returns one key, but many input points may map to the same key. Prefer this traced mapping over manually reconstructing keys in warmup code; `dispatch(...)` should express the same specialization logic used by the runtime path.
|
||||
|
||||
### Define Input Spaces
|
||||
|
||||
Use ranges and alternatives for independent axes, `zip_inputs(...)` for coupled rows, and `_when` for validity constraints.
|
||||
|
||||
#### Integer Ranges
|
||||
|
||||
Use `WarmupIntRange` for integer ranges:
|
||||
|
||||
```python
|
||||
return self._trace_dispatch(self.dispatch)(
|
||||
num_prefills=WarmupIntRange(1, max_prefills + 1),
|
||||
)
|
||||
```
|
||||
|
||||
`WarmupIntRange(start, stop, step)` follows Python `range(...)` semantics: `start` is inclusive, `stop` is exclusive, and `step` defaults to 1.
|
||||
|
||||
For non-linear integer sequences, use `advance` to provide the action that computes each next value:
|
||||
|
||||
```python
|
||||
return self._trace_dispatch(self.dispatch)(
|
||||
num_tokens=WarmupIntRange(
|
||||
1,
|
||||
max_tokens + 1,
|
||||
advance=lambda value: next_power_of_2(value) + 1,
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
This is useful for traversing specialization boundaries without enumerating every integer. `advance` cannot be combined with a non-default `step`, and it must return a value greater than its input so expansion always makes forward progress.
|
||||
|
||||
#### Independent Alternatives
|
||||
|
||||
Use tuples or lists for independent alternatives. Multiple expanded inputs form a cartesian product:
|
||||
|
||||
```python
|
||||
return self._trace_dispatch(self.dispatch)(
|
||||
query_slice_start=WarmupIntRange(0, 2),
|
||||
query_slice_stop=(1, 2 * max_tokens - 1, 2 * max_tokens),
|
||||
COMPRESS_RATIO=list(compress_ratios),
|
||||
)
|
||||
```
|
||||
|
||||
#### Coupled Inputs
|
||||
|
||||
Use `zip_inputs(...)` when values must vary together row-by-row:
|
||||
|
||||
```python
|
||||
WARMUP_INPUTS = zip_inputs(
|
||||
dict(compress_ratio=1, topk=0, topk_width=512),
|
||||
dict(compress_ratio=4, topk=512, topk_width=512),
|
||||
)
|
||||
|
||||
|
||||
return self._trace_dispatch(self.dispatch)(
|
||||
WARMUP_INPUTS,
|
||||
WINDOW_SIZE=window_size,
|
||||
)
|
||||
```
|
||||
|
||||
Multiple `zip_inputs(...)` groups may be passed as positional arguments. The tracer forms the cartesian product across groups while preserving row-wise coupling inside each group.
|
||||
|
||||
Every row in a `zip_inputs(...)` group must use the same string keys. A `zip_inputs(...)` group cannot specify a field that is also specified as a keyword input to `_trace_dispatch(...)`.
|
||||
|
||||
#### Conditional Filtering
|
||||
|
||||
Use `_when=...` to filter generated input points before they are passed to `dispatch(...)`. This is useful when independent ranges contain invalid combinations, but the validity rule belongs with the kernel warmup definition.
|
||||
|
||||
```python
|
||||
def _is_valid_warmup_input(
|
||||
self,
|
||||
*,
|
||||
query_len: int,
|
||||
num_reqs: int,
|
||||
max_num_batched_tokens: int,
|
||||
) -> bool:
|
||||
return query_len + num_reqs - 1 <= max_num_batched_tokens
|
||||
|
||||
|
||||
return self._trace_dispatch(self.dispatch)(
|
||||
query_len=WarmupIntRange(1, max_tokens + 1),
|
||||
num_reqs=WarmupIntRange(1, max_reqs + 1),
|
||||
max_num_batched_tokens=max_tokens,
|
||||
_when=self._is_valid_warmup_input,
|
||||
)
|
||||
```
|
||||
|
||||
`_when` accepts a function, bound method, or lambda and supports the same AST subset as `dispatch(...)`, including local assignments in function predicates.
|
||||
|
||||
The predicate is evaluated on the expanded warmup inputs. If it returns `False`, that input point is skipped and no `CompileKey` is produced for it.
|
||||
|
||||
### Write Dispatch Rules
|
||||
|
||||
#### Local Assignments
|
||||
|
||||
The traced body may contain local assignments, optionally annotated, followed by one `return self.CompileKey(...)` call. Local assignments let a kernel name intermediate specialization choices once and reuse them across fields:
|
||||
|
||||
```python
|
||||
def dispatch(
|
||||
self,
|
||||
*,
|
||||
num_tokens: int,
|
||||
vectorized: bool,
|
||||
) -> CompileKey:
|
||||
block_size = next_power_of_2(num_tokens)
|
||||
return self.CompileKey(
|
||||
BLOCK_SIZE=block_size,
|
||||
VECTOR_WIDTH=4 if vectorized and block_size >= 4 else 1,
|
||||
)
|
||||
```
|
||||
|
||||
#### Supported Expressions
|
||||
|
||||
The evaluator supports these expressions inside local assignments and `CompileKey(...)` fields:
|
||||
|
||||
| Feature | What It Allows |
|
||||
| --- | --- |
|
||||
| Names | Read dispatch inputs, local assignments, defaults, and module globals. |
|
||||
| Constants | Use literals such as integers, strings, booleans, and `None`. |
|
||||
| Attributes | Read structured values such as `cfg.block_size` or `mla_dims.v_head_dim`. |
|
||||
| Subscriptions | Read sequence positions or mapping values such as `config[0]` and `config["block_size"]`. |
|
||||
| Tuple/list literals | Build shapes, strides, and other small structured fields. |
|
||||
| Conditional expressions | Select a field with `x if condition else y`. |
|
||||
| Boolean expressions | Combine predicates with `and`, `or`, and `not`. |
|
||||
| Comparisons | Use `==`, `!=`, `<`, `<=`, `>`, `>=`, `in`, `not in`, `is`, and `is not`. |
|
||||
| Arithmetic | Use `+`, `-`, `*`, `//`, `%`, and `**`. |
|
||||
| Unary minus | Build negative sentinel values or signed descriptors. |
|
||||
| Helper calls | Call helpers with positional and explicit keyword arguments. |
|
||||
|
||||
Python builtins such as `min(...)`, `max(...)`, and `len(...)` are resolved unless the name is overridden locally or globally.
|
||||
|
||||
#### Helper Calls
|
||||
|
||||
Helpers are useful for small specialization rules:
|
||||
|
||||
```python
|
||||
def dispatch(self, *, num_tokens: int, block_size: int) -> CompileKey:
|
||||
return self.CompileKey(
|
||||
PADDED_TOKENS=round_up(num_tokens, multiple=block_size),
|
||||
)
|
||||
```
|
||||
|
||||
`_trace_dispatch(...)` does not inspect helper bodies. It evaluates the call arguments and invokes the helper as ordinary Python, so control flow inside that helper is outside the AST interpreter's scope. Keep helpers deterministic and side-effect free.
|
||||
|
||||
#### Direct Keyword Forwarding
|
||||
|
||||
For many direct pass-through fields, the dispatch `**kwargs` parameter may be unpacked into `CompileKey(...)`:
|
||||
|
||||
```python
|
||||
def dispatch(
|
||||
self,
|
||||
*,
|
||||
num_tokens: int,
|
||||
**compile_key_fields: int,
|
||||
) -> CompileKey:
|
||||
return self.CompileKey(
|
||||
**compile_key_fields,
|
||||
block_size=next_power_of_2(num_tokens),
|
||||
)
|
||||
```
|
||||
|
||||
Unmatched dispatch arguments become compile-key fields and warmup inputs. Keep transformed inputs named and explicit. The unpacking must use the dispatch method's own `**kwargs` parameter directly and exactly once; arbitrary mappings, repeated unpacking, and helper-call `**kwargs` are rejected. The fully explicit form remains supported and is often clearer for non-trivial mappings.
|
||||
|
||||
#### Unsupported Syntax
|
||||
|
||||
Conditional expressions (`x if condition else y`) are supported, but statement-level `if` blocks are not supported directly inside traced `dispatch(...)` or `_when` bodies. The tracer expects a straight-line sequence of local assignments followed by one return expression. Small, pure helpers called by traced expressions execute as normal Python with concrete values and may use ordinary control flow, including `if` blocks. Do not put loops, mutation, side effects, or backend imports directly inside traced functions. Put environment and model gating in `get_warmup_keys(...)` or the outer warmup entry point.
|
||||
|
||||
### Compile-Key Deduplication
|
||||
|
||||
`_trace_dispatch(...)` deduplicates the resulting keys while preserving order. This is important when many runtime-like inputs map to the same static bucket.
|
||||
|
||||
For example, this warmup range expands every token count, but the compile key only depends on the power-of-two bucket:
|
||||
|
||||
```python
|
||||
def dispatch(
|
||||
self,
|
||||
*,
|
||||
num_tokens: int,
|
||||
) -> CompileKey:
|
||||
return self.CompileKey(
|
||||
BLOCK_SIZE=next_power_of_2(num_tokens),
|
||||
)
|
||||
|
||||
|
||||
def get_warmup_keys(self, vllm_config: VllmConfig) -> list[CompileKey]:
|
||||
max_tokens = vllm_config.scheduler_config.max_num_batched_tokens
|
||||
return self._trace_dispatch(self.dispatch)(
|
||||
num_tokens=WarmupIntRange(1, max_tokens + 1),
|
||||
)
|
||||
```
|
||||
|
||||
For `max_tokens == 8`, the expanded inputs are `1, 2, 3, 4, 5, 6, 7, 8`, but the returned keys are:
|
||||
|
||||
```python
|
||||
[
|
||||
CompileKey(BLOCK_SIZE=1),
|
||||
CompileKey(BLOCK_SIZE=2),
|
||||
CompileKey(BLOCK_SIZE=4),
|
||||
CompileKey(BLOCK_SIZE=8),
|
||||
]
|
||||
```
|
||||
|
||||
Deduplication happens after `dispatch(...)` is evaluated, so the warmup system removes duplicate compile keys, not duplicate input values. `CompileKey` must be hashable for this to work; using `@dataclass(frozen=True)` is the standard pattern.
|
||||
@@ -9,11 +9,15 @@ from typing import Any, cast
|
||||
import pytest
|
||||
|
||||
from vllm.model_executor.warmup.jit_warmup import (
|
||||
JitWarmupRegistry,
|
||||
VllmJitKernel,
|
||||
WarmupIntRange,
|
||||
get_ast_full_name,
|
||||
zip_inputs,
|
||||
)
|
||||
from vllm.model_executor.warmup.jit_warmup_triton_helper import (
|
||||
triton_scalar_specialization_rep,
|
||||
)
|
||||
|
||||
|
||||
def _next_power_of_2(value: int) -> int:
|
||||
@@ -94,6 +98,35 @@ class RecordingToyKernel(ToyKernel):
|
||||
self.compiled.append(compile_key)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("value", "expected"),
|
||||
[
|
||||
(-(1 << 63), 1 << 31),
|
||||
(-(1 << 31) - 1, (1 << 31) + 1),
|
||||
(-(1 << 31), 16),
|
||||
(0, 16),
|
||||
(1, 1),
|
||||
(2, 2),
|
||||
(16, 16),
|
||||
((1 << 31) - 1, 2),
|
||||
(1 << 31, 1 << 31),
|
||||
((1 << 31) + 1, (1 << 31) + 1),
|
||||
((1 << 63) - 1, (1 << 31) + 1),
|
||||
(1 << 63, 1 << 63),
|
||||
((1 << 63) + 1, (1 << 63) + 1),
|
||||
((1 << 64) - 1, (1 << 63) + 1),
|
||||
],
|
||||
)
|
||||
def test_triton_scalar_specialization_rep(value: int, expected: int) -> None:
|
||||
assert triton_scalar_specialization_rep(value) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize("value", [-(1 << 63) - 1, 1 << 64])
|
||||
def test_triton_scalar_specialization_rep_rejects_out_of_range(value: int) -> None:
|
||||
with pytest.raises(OverflowError, match="outside Triton's scalar range"):
|
||||
triton_scalar_specialization_rep(value)
|
||||
|
||||
|
||||
def test_trace_dispatch_expands_ranges_dedupes_and_ignores_unused_inputs() -> None:
|
||||
cfg = _config()
|
||||
|
||||
@@ -287,6 +320,31 @@ def test_helper_calls_support_keywords_and_reject_star_kwargs() -> None:
|
||||
StarKwargsKernel().compile_key({"tokens": 5, "block_size": 4})
|
||||
|
||||
|
||||
def test_dispatch_helper_calls_resolve_python_builtins() -> None:
|
||||
class BuiltinKernel(VllmJitKernel["BuiltinKernel.CompileKey"]):
|
||||
@dataclass(frozen=True)
|
||||
class CompileKey:
|
||||
value: int
|
||||
|
||||
def dispatch( # type: ignore[override]
|
||||
self,
|
||||
*,
|
||||
tokens: int,
|
||||
limit: int,
|
||||
) -> CompileKey:
|
||||
return self.CompileKey(value=max(1, min(tokens, limit)))
|
||||
|
||||
def get_warmup_keys(self) -> list[CompileKey]:
|
||||
return []
|
||||
|
||||
def compile(self, compile_key: CompileKey) -> None:
|
||||
pass
|
||||
|
||||
assert BuiltinKernel().compile_key({"tokens": 8, "limit": 4}) == (
|
||||
BuiltinKernel.CompileKey(value=4)
|
||||
)
|
||||
|
||||
|
||||
def test_dispatch_body_must_be_local_assignments_then_compile_key_return() -> None:
|
||||
class BranchKernel(VllmJitKernel["BranchKernel.CompileKey"]):
|
||||
@dataclass(frozen=True)
|
||||
@@ -320,10 +378,85 @@ def test_dispatch_body_must_be_local_assignments_then_compile_key_return() -> No
|
||||
|
||||
with pytest.raises(ValueError, match="local assignments"):
|
||||
BranchKernel()
|
||||
with pytest.raises(ValueError, match=r"cannot use \*\*kwargs in CompileKey"):
|
||||
with pytest.raises(
|
||||
ValueError,
|
||||
match=r"may unpack only its own \*\*kwargs parameter once",
|
||||
):
|
||||
KwargsReturnKernel()
|
||||
|
||||
|
||||
def test_dispatch_can_forward_compile_key_fields() -> None:
|
||||
class ForwardingKernel(VllmJitKernel["ForwardingKernel.CompileKey"]):
|
||||
@dataclass(frozen=True)
|
||||
class CompileKey:
|
||||
mode: int
|
||||
block_size: int
|
||||
|
||||
def dispatch( # type: ignore[override]
|
||||
self,
|
||||
*,
|
||||
tokens: int,
|
||||
**compile_key_fields: int,
|
||||
) -> CompileKey:
|
||||
return self.CompileKey(
|
||||
**compile_key_fields,
|
||||
block_size=_round_up(tokens, multiple=4),
|
||||
)
|
||||
|
||||
def get_warmup_keys(self) -> list[CompileKey]:
|
||||
return self._trace_dispatch(self.dispatch)(
|
||||
tokens=(1, 5),
|
||||
mode=(2, 3),
|
||||
)
|
||||
|
||||
def compile(self, compile_key: CompileKey) -> None:
|
||||
pass
|
||||
|
||||
kernel = ForwardingKernel()
|
||||
expected = kernel.CompileKey(
|
||||
mode=2,
|
||||
block_size=8,
|
||||
)
|
||||
assert kernel.dispatch(tokens=5, mode=2) == expected
|
||||
assert kernel.compile_key({"tokens": 5, "mode": 2}) == expected
|
||||
assert kernel.get_warmup_keys() == [
|
||||
kernel.CompileKey(mode=2, block_size=4),
|
||||
kernel.CompileKey(mode=3, block_size=4),
|
||||
kernel.CompileKey(mode=2, block_size=8),
|
||||
kernel.CompileKey(mode=3, block_size=8),
|
||||
]
|
||||
with pytest.raises(TypeError, match="field 'block_size' is specified twice"):
|
||||
kernel.compile_key({"tokens": 5, "mode": 2, "block_size": 4})
|
||||
with pytest.raises(TypeError, match="unexpected keyword argument 'extra'"):
|
||||
kernel.compile_key({"tokens": 5, "mode": 2, "extra": 1})
|
||||
|
||||
|
||||
def test_dispatch_supports_tuple_and_mapping_subscriptions() -> None:
|
||||
class SubscriptKernel(VllmJitKernel["SubscriptKernel.CompileKey"]):
|
||||
@dataclass(frozen=True)
|
||||
class CompileKey:
|
||||
first: int
|
||||
named: int
|
||||
|
||||
def dispatch( # type: ignore[override]
|
||||
self,
|
||||
*,
|
||||
values: tuple[int, ...],
|
||||
config: dict[str, int],
|
||||
) -> CompileKey:
|
||||
return self.CompileKey(first=values[0], named=config["named"])
|
||||
|
||||
def get_warmup_keys(self) -> list[CompileKey]:
|
||||
return []
|
||||
|
||||
def compile(self, compile_key: CompileKey) -> None:
|
||||
pass
|
||||
|
||||
assert SubscriptKernel().compile_key(
|
||||
{"values": (3, 5), "config": {"named": 7}}
|
||||
) == SubscriptKernel.CompileKey(first=3, named=7)
|
||||
|
||||
|
||||
def test_dispatch_reports_unsupported_expression_with_context() -> None:
|
||||
class UnsupportedKernel(VllmJitKernel["UnsupportedKernel.CompileKey"]):
|
||||
@dataclass(frozen=True)
|
||||
@@ -361,6 +494,102 @@ def test_warmup_compiles_all_returned_keys_in_order() -> None:
|
||||
]
|
||||
|
||||
|
||||
def test_runtime_cache_miss_compiles_and_caches_executor() -> None:
|
||||
class CachedKernel(VllmJitKernel["CachedKernel.CompileKey"]):
|
||||
@dataclass(frozen=True)
|
||||
class CompileKey:
|
||||
value: int
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.compiled: list[CachedKernel.CompileKey] = []
|
||||
super().__init__()
|
||||
|
||||
def dispatch(self, *, value: int) -> CompileKey: # type: ignore[override]
|
||||
return self.CompileKey(value=value)
|
||||
|
||||
def get_warmup_keys(self) -> list[CompileKey]:
|
||||
return self._trace_dispatch(self.dispatch)(value=1)
|
||||
|
||||
def compile(self, compile_key: CompileKey) -> None:
|
||||
self.compiled.append(compile_key)
|
||||
self._compiled_cache[compile_key] = object()
|
||||
|
||||
def __call__(self, value: int) -> None:
|
||||
compile_key = self.dispatch(value=value)
|
||||
self._get_or_compile(compile_key)
|
||||
|
||||
kernel = CachedKernel()
|
||||
kernel.warmup()
|
||||
kernel(1)
|
||||
kernel(2)
|
||||
|
||||
assert kernel.compiled == [
|
||||
CachedKernel.CompileKey(value=1),
|
||||
CachedKernel.CompileKey(value=2),
|
||||
]
|
||||
|
||||
|
||||
def test_registry_records_only_inside_model_setup_context() -> None:
|
||||
registry = JitWarmupRegistry(_config())
|
||||
kernel = RecordingToyKernel()
|
||||
|
||||
kernel.register_warmup(3, _config())
|
||||
with registry.activate():
|
||||
kernel.register_warmup(3, _config())
|
||||
|
||||
assert len(registry) == 1
|
||||
assert kernel.compiled == []
|
||||
|
||||
|
||||
def test_registry_expands_requests_and_deduplicates_owner_keys() -> None:
|
||||
registry = JitWarmupRegistry(_config())
|
||||
kernel = RecordingToyKernel()
|
||||
|
||||
with registry.activate():
|
||||
kernel.register_warmup(3, _config())
|
||||
kernel.register_warmup(5, _config())
|
||||
|
||||
registry.warmup()
|
||||
|
||||
assert kernel.compiled == [
|
||||
ToyKernel.CompileKey(1, 1, 1, ("base", "default", -1, 1, 1), True),
|
||||
ToyKernel.CompileKey(2, 2, 1, ("base", "default", -2, 2, 4), True),
|
||||
ToyKernel.CompileKey(4, 4, 1, ("base", "default", -4, 1, 16), True),
|
||||
ToyKernel.CompileKey(8, 8, 1, ("base", "default", -8, 2, 64), True),
|
||||
]
|
||||
|
||||
|
||||
def test_registry_passes_vllm_config_to_default_requests() -> None:
|
||||
class ConfigKernel(VllmJitKernel["ConfigKernel.CompileKey"]):
|
||||
@dataclass(frozen=True)
|
||||
class CompileKey:
|
||||
value: int
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.compiled: list[ConfigKernel.CompileKey] = []
|
||||
super().__init__()
|
||||
|
||||
def dispatch(self, *, value: int) -> CompileKey: # type: ignore[override]
|
||||
return self.CompileKey(value=value)
|
||||
|
||||
def get_warmup_keys(self, vllm_config: Any) -> list[CompileKey]:
|
||||
return [self.dispatch(value=vllm_config.bias)]
|
||||
|
||||
def compile(self, compile_key: CompileKey) -> None:
|
||||
self.compiled.append(compile_key)
|
||||
|
||||
registry = JitWarmupRegistry(_config(bias=7))
|
||||
kernel = ConfigKernel()
|
||||
|
||||
with registry.activate():
|
||||
kernel.register_warmup()
|
||||
kernel.register_warmup()
|
||||
assert len(registry) == 1
|
||||
registry.warmup()
|
||||
|
||||
assert kernel.compiled == [ConfigKernel.CompileKey(value=7)]
|
||||
|
||||
|
||||
def test_get_ast_full_name_handles_names_attributes_and_other_nodes() -> None:
|
||||
dotted_expr = ast.parse("foo.bar.baz").body[0]
|
||||
call_expr = ast.parse("foo()").body[0]
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Validate the registry reference kernel against runtime dispatch."""
|
||||
|
||||
import pytest
|
||||
|
||||
from vllm.platforms import current_platform
|
||||
|
||||
if not current_platform.is_cuda_alike():
|
||||
pytest.skip("NVIDIA dispatch tests require CUDA", allow_module_level=True)
|
||||
|
||||
from vllm.v1.worker.block_table import ComputeSlotMappingKernel
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("kv_cache_block_size", "blocks_per_kv_block", "block_size", "block_size_rep"),
|
||||
[
|
||||
(256, 1, 256, 16),
|
||||
(256, 4, 64, 16),
|
||||
(64, 1, 64, 16),
|
||||
(8, 1, 8, 2),
|
||||
(4, 1, 4, 2),
|
||||
],
|
||||
)
|
||||
def test_compute_slot_mapping_warmup_matches_runtime_specializations(
|
||||
kv_cache_block_size: int,
|
||||
blocks_per_kv_block: int,
|
||||
block_size: int,
|
||||
block_size_rep: int,
|
||||
) -> None:
|
||||
kernel = ComputeSlotMappingKernel()
|
||||
kwargs = dict(
|
||||
kv_cache_block_size=kv_cache_block_size,
|
||||
blocks_per_kv_block=blocks_per_kv_block,
|
||||
total_cp_world_size=1,
|
||||
total_cp_rank=0,
|
||||
cp_kv_cache_interleave_size=1,
|
||||
block_table_stride=32768,
|
||||
block_size=block_size,
|
||||
)
|
||||
expected = kernel.CompileKey(
|
||||
kv_cache_block_size=kv_cache_block_size,
|
||||
blocks_per_kv_block=blocks_per_kv_block,
|
||||
total_cp_world_size=1,
|
||||
total_cp_rank=0,
|
||||
cp_kv_cache_interleave_size=1,
|
||||
block_table_stride=16,
|
||||
block_size=block_size_rep,
|
||||
)
|
||||
|
||||
assert kernel.dispatch(**kwargs) == expected
|
||||
assert kernel.get_warmup_keys(**kwargs) == [expected]
|
||||
@@ -5,16 +5,20 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import builtins
|
||||
import inspect
|
||||
import itertools
|
||||
import operator
|
||||
import textwrap
|
||||
from abc import ABC, abstractmethod
|
||||
from collections.abc import Callable, Iterable, Mapping
|
||||
from collections.abc import Callable, Iterable, Iterator, Mapping
|
||||
from contextlib import contextmanager
|
||||
from contextvars import ContextVar
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Generic, TypeVar
|
||||
|
||||
__all__ = [
|
||||
"JitWarmupRegistry",
|
||||
"VllmJitKernel",
|
||||
"WarmupIntRange",
|
||||
"get_ast_full_name",
|
||||
@@ -136,8 +140,15 @@ class _CompileKeyDispatchTrace:
|
||||
field_exprs: tuple[tuple[str, ast.AST], ...]
|
||||
globals: Mapping[str, Any]
|
||||
input_names: frozenset[str]
|
||||
# Named parameters are excluded from direct **kwargs forwarding.
|
||||
named_parameters: frozenset[str] | None
|
||||
defaults: Mapping[str, Any]
|
||||
|
||||
def input_names_for(self, available_names: set[str]) -> frozenset[str]:
|
||||
if self.named_parameters is None:
|
||||
return self.input_names
|
||||
return self.input_names | (available_names - self.named_parameters)
|
||||
|
||||
def compile_key(
|
||||
self,
|
||||
compile_key_type: type[CompileKeyT],
|
||||
@@ -146,12 +157,20 @@ class _CompileKeyDispatchTrace:
|
||||
dispatch_values = _eval_local_exprs(
|
||||
self.local_exprs, {**self.defaults, **kwargs}, self.globals
|
||||
)
|
||||
return compile_key_type(
|
||||
**{
|
||||
field: _eval_dispatch_expr(expr, dispatch_values, self.globals)
|
||||
for field, expr in self.field_exprs
|
||||
named_parameters = self.named_parameters
|
||||
# Materialize direct fields before evaluating named AST expressions.
|
||||
fields: dict[str, Any] = {}
|
||||
if named_parameters is not None:
|
||||
fields = {
|
||||
name: value
|
||||
for name, value in kwargs.items()
|
||||
if name not in named_parameters
|
||||
}
|
||||
)
|
||||
for field, expr in self.field_exprs:
|
||||
if field in fields:
|
||||
raise TypeError(f"CompileKey field '{field}' is specified twice")
|
||||
fields[field] = _eval_dispatch_expr(expr, dispatch_values, self.globals)
|
||||
return compile_key_type(**fields)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -186,6 +205,10 @@ _CMP_OPS: dict[type[ast.cmpop], Callable[[Any, Any], bool]] = {
|
||||
ast.LtE: operator.le,
|
||||
ast.Gt: operator.gt,
|
||||
ast.GtE: operator.ge,
|
||||
ast.In: lambda left, right: left in right,
|
||||
ast.NotIn: lambda left, right: left not in right,
|
||||
ast.Is: operator.is_,
|
||||
ast.IsNot: operator.is_not,
|
||||
}
|
||||
|
||||
|
||||
@@ -200,8 +223,9 @@ def _dispatch_expr_error(node: ast.AST, reason: str) -> ValueError:
|
||||
return ValueError(
|
||||
f"{reason}: {_dispatch_expr_source(node)}. "
|
||||
"Supported dispatch expressions are names, constants, attributes, "
|
||||
"tuple/list literals, conditional expressions, comparisons, boolean "
|
||||
"operators, unary not/minus, arithmetic, and calls without **kwargs."
|
||||
"subscriptions, tuple/list literals, conditional expressions, "
|
||||
"comparisons, boolean operators, unary not/minus, arithmetic, and "
|
||||
"calls without **kwargs."
|
||||
)
|
||||
|
||||
|
||||
@@ -225,6 +249,8 @@ class _DispatchExprEvaluator(ast.NodeVisitor):
|
||||
return self.values[node.id]
|
||||
if node.id in self.globals:
|
||||
return self.globals[node.id]
|
||||
if hasattr(builtins, node.id):
|
||||
return getattr(builtins, node.id)
|
||||
raise _dispatch_expr_error(node, f"Unknown dispatch name '{node.id}'")
|
||||
|
||||
def visit_Constant(self, node: ast.Constant) -> Any:
|
||||
@@ -299,6 +325,9 @@ class _DispatchExprEvaluator(ast.NodeVisitor):
|
||||
def visit_Attribute(self, node: ast.Attribute) -> Any:
|
||||
return getattr(self.visit(node.value), node.attr)
|
||||
|
||||
def visit_Subscript(self, node: ast.Subscript) -> Any:
|
||||
return self.visit(node.value)[self.visit(node.slice)]
|
||||
|
||||
|
||||
def get_ast_full_name(node: ast.AST) -> str | None:
|
||||
if isinstance(node, ast.Name):
|
||||
@@ -310,15 +339,23 @@ def get_ast_full_name(node: ast.AST) -> str | None:
|
||||
return None
|
||||
|
||||
|
||||
def get_function_source_node(fn: Callable[..., Any]) -> ast.FunctionDef:
|
||||
def get_function_source_node(fn: Callable[..., Any]) -> ast.FunctionDef | ast.Lambda:
|
||||
source_fn = getattr(fn, "fn", fn)
|
||||
source = textwrap.dedent(inspect.getsource(source_fn))
|
||||
tree = ast.parse(source)
|
||||
function_defs = [node for node in tree.body if isinstance(node, ast.FunctionDef)]
|
||||
if len(function_defs) != 1:
|
||||
name = getattr(source_fn, "__name__", type(source_fn).__name__)
|
||||
raise ValueError(f"Expected one function in {name}, found {len(function_defs)}")
|
||||
return function_defs[0]
|
||||
if len(function_defs) == 1:
|
||||
return function_defs[0]
|
||||
|
||||
lambdas = [node for node in ast.walk(tree) if isinstance(node, ast.Lambda)]
|
||||
if len(lambdas) == 1:
|
||||
return lambdas[0]
|
||||
|
||||
name = getattr(source_fn, "__name__", type(source_fn).__name__)
|
||||
raise ValueError(
|
||||
f"Expected one function or lambda in {name}, found "
|
||||
f"{len(function_defs)} functions and {len(lambdas)} lambdas"
|
||||
)
|
||||
|
||||
|
||||
def _eval_dispatch_expr(
|
||||
@@ -349,8 +386,11 @@ def _collect_input_names(
|
||||
|
||||
def _collect_expression_body(
|
||||
fn: Callable[..., Any],
|
||||
function_def: ast.FunctionDef,
|
||||
function_def: ast.FunctionDef | ast.Lambda,
|
||||
) -> tuple[list[tuple[str, ast.AST]], ast.AST]:
|
||||
if isinstance(function_def, ast.Lambda):
|
||||
return [], function_def.body
|
||||
|
||||
local_exprs: list[tuple[str, ast.AST]] = []
|
||||
for statement in function_def.body:
|
||||
if (
|
||||
@@ -421,6 +461,10 @@ def _trace_compile_key_dispatch(
|
||||
source_fn = getattr(fn, "__func__", fn)
|
||||
globals_ = source_fn.__globals__
|
||||
function_def = get_function_source_node(fn)
|
||||
if isinstance(function_def, ast.Lambda):
|
||||
raise _dispatch_expr_error(
|
||||
function_def, "Dispatch must be a function definition"
|
||||
)
|
||||
|
||||
local_exprs, return_expr = _collect_expression_body(fn, function_def)
|
||||
if not isinstance(return_expr, ast.Call):
|
||||
@@ -431,13 +475,37 @@ def _trace_compile_key_dispatch(
|
||||
|
||||
field_exprs: list[tuple[str, ast.AST]] = []
|
||||
defaults, candidate_names = _function_trace_inputs(fn)
|
||||
# Fields captured by dispatch **kwargs are forwarded, not AST-evaluated.
|
||||
signature = inspect.signature(fn)
|
||||
variadic_keyword = next(
|
||||
(
|
||||
name
|
||||
for name, parameter in signature.parameters.items()
|
||||
if parameter.kind is inspect.Parameter.VAR_KEYWORD
|
||||
),
|
||||
None,
|
||||
)
|
||||
candidate_names.discard(variadic_keyword)
|
||||
input_names: set[str] = set()
|
||||
local_names = {name for name, _ in local_exprs}
|
||||
for _, expr in local_exprs:
|
||||
input_names.update(_collect_input_names(expr, candidate_names))
|
||||
named_parameters: frozenset[str] | None = None
|
||||
for keyword in return_expr.keywords:
|
||||
# CompileKey may unpack only that **kwargs parameter, once.
|
||||
if keyword.arg is None:
|
||||
raise ValueError(f"{fn.__name__} cannot use **kwargs in CompileKey")
|
||||
if (
|
||||
named_parameters is not None
|
||||
or variadic_keyword is None
|
||||
or not isinstance(keyword.value, ast.Name)
|
||||
or keyword.value.id != variadic_keyword
|
||||
):
|
||||
raise ValueError(
|
||||
f"{fn.__name__} may unpack only its own **kwargs parameter "
|
||||
"once in CompileKey"
|
||||
)
|
||||
named_parameters = frozenset(candidate_names)
|
||||
continue
|
||||
field_exprs.append((keyword.arg, keyword.value))
|
||||
input_names.update(
|
||||
_collect_input_names(keyword.value, candidate_names, local_names)
|
||||
@@ -448,6 +516,7 @@ def _trace_compile_key_dispatch(
|
||||
tuple(field_exprs),
|
||||
globals_,
|
||||
frozenset(input_names),
|
||||
named_parameters,
|
||||
defaults,
|
||||
)
|
||||
|
||||
@@ -482,10 +551,32 @@ class VllmJitKernel(Generic[CompileKeyT], ABC):
|
||||
CompileKey: type[CompileKeyT]
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.compile_key_dispatch_trace = _trace_compile_key_dispatch(self.dispatch)
|
||||
self._dispatch_trace = _trace_compile_key_dispatch(self.dispatch)
|
||||
self._compiled_cache: dict[Any, Any] = {}
|
||||
|
||||
def compile_key(self, kwargs: Mapping[str, Any]) -> CompileKeyT:
|
||||
return self.compile_key_dispatch_trace.compile_key(self.CompileKey, kwargs)
|
||||
return self._dispatch_trace.compile_key(self.CompileKey, kwargs)
|
||||
|
||||
def _get_or_compile(
|
||||
self,
|
||||
compile_key: CompileKeyT,
|
||||
*,
|
||||
runtime_context: Mapping[str, Any] | None = None,
|
||||
) -> Any:
|
||||
"""Return a cached executor, compiling it on a monitored cache miss."""
|
||||
if compile_key not in self._compiled_cache:
|
||||
self.compile(compile_key)
|
||||
|
||||
try:
|
||||
return self._compiled_cache[compile_key]
|
||||
except KeyError as exc:
|
||||
details = [f"compile_key={compile_key!r}"]
|
||||
if runtime_context:
|
||||
details.append(f"runtime_context={dict(runtime_context)!r}")
|
||||
raise RuntimeError(
|
||||
f"{type(self).__name__}.compile(...) did not cache its JIT "
|
||||
f"executor ({', '.join(details)})"
|
||||
) from exc
|
||||
|
||||
def _trace_dispatch(
|
||||
self, dispatch: CompileKeyDispatchFn[CompileKeyT]
|
||||
@@ -506,7 +597,11 @@ class VllmJitKernel(Generic[CompileKeyT], ABC):
|
||||
predicate_trace = (
|
||||
_trace_warmup_predicate(_when) if _when is not None else None
|
||||
)
|
||||
input_names = compile_key_dispatch_trace.input_names
|
||||
# Unmatched **kwargs fields also belong to the expansion space.
|
||||
available_names = set(kwargs).union(
|
||||
*(group.rows[0] for group in input_groups)
|
||||
)
|
||||
input_names = compile_key_dispatch_trace.input_names_for(available_names)
|
||||
if predicate_trace is not None:
|
||||
input_names = input_names | predicate_trace.input_names
|
||||
expanded_input_groups = tuple(
|
||||
@@ -565,7 +660,105 @@ class VllmJitKernel(Generic[CompileKeyT], ABC):
|
||||
"""Compile one warmup key."""
|
||||
raise NotImplementedError
|
||||
|
||||
def register_warmup(self, *args: Any, **kwargs: Any) -> None:
|
||||
"""Register this kernel with the active runner's warmup registry."""
|
||||
JitWarmupRegistry.register(self, *args, **kwargs)
|
||||
|
||||
def warmup(self, *args: Any, **kwargs: Any) -> None:
|
||||
"""Compile this kernel's warmup keys."""
|
||||
for compile_key in self.get_warmup_keys(*args, **kwargs):
|
||||
self.compile(compile_key)
|
||||
|
||||
|
||||
class JitWarmupRegistry:
|
||||
"""Collect and compile JIT kernels selected during runner setup."""
|
||||
|
||||
_active: ContextVar[JitWarmupRegistry | None] = ContextVar(
|
||||
"active_jit_warmup_registry",
|
||||
default=None,
|
||||
)
|
||||
|
||||
def __init__(self, vllm_config: Any) -> None:
|
||||
self.vllm_config = vllm_config
|
||||
self._registrations: dict[
|
||||
VllmJitKernel[Any],
|
||||
list[tuple[tuple[Any, ...], dict[str, Any]]],
|
||||
] = {}
|
||||
|
||||
@contextmanager
|
||||
def activate(self) -> Iterator[None]:
|
||||
"""Collect registrations made in this context."""
|
||||
token = self._active.set(self)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
self._active.reset(token)
|
||||
|
||||
@classmethod
|
||||
def register(
|
||||
cls,
|
||||
kernel: VllmJitKernel[Any],
|
||||
*args: Any,
|
||||
**kwargs: Any,
|
||||
) -> None:
|
||||
"""Register a kernel with the active registry, if one exists."""
|
||||
registry = cls._active.get()
|
||||
if registry is not None:
|
||||
registry._add(kernel, args, kwargs)
|
||||
|
||||
def _add(
|
||||
self,
|
||||
kernel: VllmJitKernel[Any],
|
||||
args: tuple[Any, ...],
|
||||
kwargs: dict[str, Any],
|
||||
) -> None:
|
||||
registrations = self._registrations.setdefault(kernel, [])
|
||||
if (
|
||||
not args
|
||||
and not kwargs
|
||||
and any(
|
||||
not registered_args and not registered_kwargs
|
||||
for registered_args, registered_kwargs in registrations
|
||||
)
|
||||
):
|
||||
return
|
||||
registrations.append((args, kwargs))
|
||||
|
||||
def __len__(self) -> int:
|
||||
return sum(len(registrations) for registrations in self._registrations.values())
|
||||
|
||||
def warmup(self) -> None:
|
||||
"""Expand registrations and compile each wrapper/key pair once."""
|
||||
from tqdm import tqdm
|
||||
|
||||
from vllm.distributed import is_global_first_rank
|
||||
|
||||
kernel_items: list[tuple[VllmJitKernel[Any], dict[Any, None]]] = []
|
||||
for kernel, registrations in self._registrations.items():
|
||||
compile_keys: dict[Any, None] = {}
|
||||
for args, kwargs in registrations:
|
||||
if not args and not kwargs:
|
||||
args = (self.vllm_config,)
|
||||
for compile_key in kernel.get_warmup_keys(*args, **kwargs):
|
||||
compile_keys[compile_key] = None
|
||||
if compile_keys:
|
||||
kernel_items.append((kernel, compile_keys))
|
||||
|
||||
if not kernel_items:
|
||||
return
|
||||
|
||||
total_keys = sum(len(compile_keys) for _, compile_keys in kernel_items)
|
||||
with tqdm(
|
||||
kernel_items,
|
||||
desc=f"JIT kernel warmup ({total_keys} compile keys)",
|
||||
disable=not is_global_first_rank(),
|
||||
dynamic_ncols=True,
|
||||
unit="kernel",
|
||||
) as progress:
|
||||
for kernel, compile_keys in progress:
|
||||
progress.set_postfix_str(
|
||||
f"{kernel.__class__.__name__} ({len(compile_keys)} keys)",
|
||||
refresh=False,
|
||||
)
|
||||
for compile_key in compile_keys:
|
||||
kernel.compile(compile_key)
|
||||
|
||||
@@ -12,6 +12,42 @@ from vllm.model_executor.warmup.jit_warmup import (
|
||||
)
|
||||
|
||||
|
||||
def triton_scalar_specialization_rep(value: int) -> int:
|
||||
"""Return an integer with the same default Triton JIT specialization.
|
||||
|
||||
For an ordinary integer argument, Triton's cache key contains its inferred
|
||||
type (``i32``, ``i64``, or ``u64``) and one of three value classes:
|
||||
|
||||
* ``1`` is specialized as the exact constant ``1``.
|
||||
* Multiples of 16 receive a ``tt.divisibility = 16`` attribute.
|
||||
* All other values have no value specialization.
|
||||
|
||||
Warmup only needs one concrete value for each cache-key class. This helper
|
||||
returns ``1`` for the exact-one class and otherwise returns a divisible or
|
||||
generic representative while preserving the inferred integer type.
|
||||
|
||||
This applies only to non-``constexpr`` integer arguments using Triton's
|
||||
default specialization. Do not use it for arguments listed in
|
||||
``do_not_specialize`` or ``do_not_specialize_on_alignment``.
|
||||
"""
|
||||
if value == 1:
|
||||
return 1
|
||||
|
||||
if -(1 << 31) <= value < (1 << 31):
|
||||
divisible_rep = 16
|
||||
generic_rep = 2
|
||||
elif -(1 << 63) <= value < (1 << 63):
|
||||
divisible_rep = 1 << 31
|
||||
generic_rep = (1 << 31) + 1
|
||||
elif 0 <= value < (1 << 64):
|
||||
divisible_rep = 1 << 63
|
||||
generic_rep = (1 << 63) + 1
|
||||
else:
|
||||
raise OverflowError(f"Integer {value} is outside Triton's scalar range")
|
||||
|
||||
return divisible_rep if value % 16 == 0 else generic_rep
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TritonWarmupTensor:
|
||||
# Compile-only tensor descriptor for Triton pointer specialization.
|
||||
@@ -167,6 +203,8 @@ def trace_triton_kernel_specialization_args(
|
||||
kernel: Callable[..., Any],
|
||||
) -> tuple[str, ...]:
|
||||
function_def = get_function_source_node(kernel)
|
||||
if not isinstance(function_def, ast.FunctionDef):
|
||||
raise ValueError("Expected Triton kernel to be defined as a function")
|
||||
source_fn = getattr(kernel, "fn", kernel)
|
||||
arg_names = tuple(inspect.signature(source_fn).parameters)
|
||||
constexpr_args = _triton_constexpr_arg_names(kernel, function_def, arg_names)
|
||||
|
||||
@@ -6,6 +6,7 @@ This is useful specifically for JIT'ed kernels as we don't want JIT'ing to
|
||||
happen during model execution.
|
||||
"""
|
||||
|
||||
import time
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import torch
|
||||
@@ -36,9 +37,6 @@ from vllm.model_executor.warmup.qwen_triton_warmup import qwen_triton_warmup
|
||||
from vllm.model_executor.warmup.sparse_mla_triton_warmup import (
|
||||
sparse_mla_triton_warmup,
|
||||
)
|
||||
from vllm.model_executor.warmup.v1_block_table_warmup import (
|
||||
warm_v1_block_table_kernels,
|
||||
)
|
||||
from vllm.platforms import current_platform
|
||||
from vllm.utils.deep_gemm import is_deep_gemm_supported
|
||||
from vllm.utils.flashinfer import has_flashinfer
|
||||
@@ -103,15 +101,28 @@ def kernel_warmup(worker: "Worker", *, process_local_only: bool = False):
|
||||
)
|
||||
|
||||
if not worker.use_v2_model_runner:
|
||||
# Pooling models do not use the generation slot-mapping path.
|
||||
if not worker.model_runner.is_pooling_model:
|
||||
warm_v1_block_table_kernels(worker.model_runner)
|
||||
# The KV-block zeroing kernel is driven by the scheduler's
|
||||
# `new_block_ids_to_zero`, so no dummy run ever reaches it.
|
||||
zeroer = getattr(worker.model_runner, "_kv_block_zeroer", None)
|
||||
if zeroer is not None:
|
||||
zeroer.warmup(worker.model_runner.kv_cache_config.num_blocks)
|
||||
|
||||
if worker.vllm_config.kernel_config.enable_jit_warmup:
|
||||
logger.info("JIT kernel warmup starting.")
|
||||
jit_warmup_start = time.perf_counter()
|
||||
try:
|
||||
worker.model_runner.jit_warmup_registry.warmup()
|
||||
except Exception:
|
||||
logger.exception(
|
||||
"JIT kernel warmup failed after %.2fs.",
|
||||
time.perf_counter() - jit_warmup_start,
|
||||
)
|
||||
raise
|
||||
logger.info(
|
||||
"JIT kernel warmup finished in %.2fs.",
|
||||
time.perf_counter() - jit_warmup_start,
|
||||
)
|
||||
|
||||
qwen_triton_warmup(worker.model_runner, worker.vllm_config.model_config)
|
||||
|
||||
compilation_config = worker.vllm_config.compilation_config
|
||||
|
||||
@@ -1,29 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Warm up v1 block-table Triton kernels."""
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import torch
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from vllm.v1.worker.gpu_model_runner import GPUModelRunner
|
||||
|
||||
_SLOT_MAPPING_WARMUP_TOKENS = 8
|
||||
|
||||
|
||||
def warm_v1_block_table_kernels(runner: "GPUModelRunner") -> None:
|
||||
"""JIT-compile ``_compute_slot_mapping_kernel`` for the real block tables."""
|
||||
|
||||
device = runner.device
|
||||
block_table = runner.input_batch.block_table
|
||||
num_tokens = min(
|
||||
_SLOT_MAPPING_WARMUP_TOKENS,
|
||||
runner.scheduler_config.max_num_batched_tokens,
|
||||
)
|
||||
if num_tokens <= 0:
|
||||
return
|
||||
|
||||
query_start_loc = torch.tensor([0, num_tokens], dtype=torch.int32, device=device)
|
||||
positions = torch.arange(num_tokens, dtype=torch.int64, device=device)
|
||||
block_table.compute_slot_mapping(1, query_start_loc, positions)
|
||||
+151
-64
@@ -2,13 +2,22 @@
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
from vllm.distributed import get_dcp_group, get_pcp_group
|
||||
from vllm.logger import init_logger
|
||||
from vllm.model_executor.warmup.jit_warmup import (
|
||||
VllmJitKernel,
|
||||
)
|
||||
from vllm.model_executor.warmup.jit_warmup_triton_helper import (
|
||||
TritonWarmupTensor,
|
||||
triton_scalar_specialization_rep,
|
||||
)
|
||||
from vllm.triton_utils import tl, triton
|
||||
from vllm.utils.math_utils import cdiv
|
||||
from vllm.v1.attention.backends.utils import PAD_SLOT_ID
|
||||
@@ -134,6 +143,16 @@ class BlockTable:
|
||||
self.dcp_rank = 0
|
||||
self.cp_kv_cache_interleave_size = cp_kv_cache_interleave_size
|
||||
self.slot_mapping_mode = slot_mapping_mode
|
||||
if self.slot_mapping_mode == SlotMappingMode.TOKEN_TO_KV_SLOT:
|
||||
_COMPUTE_SLOT_MAPPING_KERNEL.register_warmup(
|
||||
kv_cache_block_size=self.kv_cache_block_size,
|
||||
blocks_per_kv_block=self.blocks_per_kv_block,
|
||||
total_cp_world_size=self.dcp_world_size,
|
||||
total_cp_rank=self.dcp_rank,
|
||||
cp_kv_cache_interleave_size=self.cp_kv_cache_interleave_size,
|
||||
block_table_stride=self.block_table.gpu.stride(0),
|
||||
block_size=self.block_size,
|
||||
)
|
||||
|
||||
def append_row(
|
||||
self,
|
||||
@@ -192,7 +211,8 @@ class BlockTable:
|
||||
return
|
||||
assert self.slot_mapping_mode == SlotMappingMode.TOKEN_TO_KV_SLOT
|
||||
|
||||
_compute_slot_mapping_kernel[(num_reqs + 1,)](
|
||||
_COMPUTE_SLOT_MAPPING_KERNEL(
|
||||
num_reqs,
|
||||
num_tokens,
|
||||
self.max_num_batched_tokens,
|
||||
query_start_loc,
|
||||
@@ -201,13 +221,11 @@ class BlockTable:
|
||||
self.block_table.gpu.stride(0),
|
||||
self.block_size,
|
||||
self.slot_mapping.gpu,
|
||||
KV_CACHE_BLOCK_SIZE=self.kv_cache_block_size,
|
||||
BLOCKS_PER_KV_BLOCK=self.blocks_per_kv_block,
|
||||
TOTAL_CP_WORLD_SIZE=self.dcp_world_size,
|
||||
TOTAL_CP_RANK=self.dcp_rank,
|
||||
CP_KV_CACHE_INTERLEAVE_SIZE=self.cp_kv_cache_interleave_size,
|
||||
PAD_ID=PAD_SLOT_ID,
|
||||
BLOCK_SIZE=1024,
|
||||
self.kv_cache_block_size,
|
||||
self.blocks_per_kv_block,
|
||||
self.dcp_world_size,
|
||||
self.dcp_rank,
|
||||
self.cp_kv_cache_interleave_size,
|
||||
)
|
||||
|
||||
def commit_block_table(self, num_reqs: int) -> None:
|
||||
@@ -376,67 +394,136 @@ class MultiGroupBlockTable:
|
||||
return self.block_tables[idx]
|
||||
|
||||
|
||||
@triton.jit(do_not_specialize=["num_tokens", "max_num_tokens"])
|
||||
def _compute_slot_mapping_kernel(
|
||||
num_tokens,
|
||||
max_num_tokens,
|
||||
query_start_loc_ptr, # [num_reqs + 1], int32
|
||||
positions_ptr, # [num_tokens], int64
|
||||
block_table_ptr, # [max_num_reqs, max_num_blocks_per_req], int32 (flat)
|
||||
block_table_stride, # max_num_blocks_per_req
|
||||
block_size,
|
||||
slot_mapping_ptr, # [max_num_tokens], int64
|
||||
KV_CACHE_BLOCK_SIZE: tl.constexpr,
|
||||
BLOCKS_PER_KV_BLOCK: tl.constexpr,
|
||||
TOTAL_CP_WORLD_SIZE: tl.constexpr,
|
||||
TOTAL_CP_RANK: tl.constexpr,
|
||||
CP_KV_CACHE_INTERLEAVE_SIZE: tl.constexpr,
|
||||
PAD_ID: tl.constexpr,
|
||||
BLOCK_SIZE: tl.constexpr,
|
||||
):
|
||||
req_idx = tl.program_id(0)
|
||||
class ComputeSlotMappingKernel(VllmJitKernel["ComputeSlotMappingKernel.CompileKey"]):
|
||||
triton_block_size = 1024
|
||||
|
||||
if req_idx == tl.num_programs(0) - 1:
|
||||
# Pad remaining slots for CUDA graph compatibility.
|
||||
for i in range(num_tokens, max_num_tokens, BLOCK_SIZE):
|
||||
@dataclass(frozen=True)
|
||||
class CompileKey:
|
||||
kv_cache_block_size: int
|
||||
blocks_per_kv_block: int
|
||||
total_cp_world_size: int
|
||||
total_cp_rank: int
|
||||
cp_kv_cache_interleave_size: int
|
||||
block_table_stride: int
|
||||
block_size: int
|
||||
|
||||
@staticmethod
|
||||
@triton.jit(do_not_specialize=["num_tokens", "max_num_tokens"])
|
||||
def kernel(
|
||||
num_tokens,
|
||||
max_num_tokens,
|
||||
query_start_loc_ptr, # [num_reqs + 1], int32
|
||||
positions_ptr, # [num_tokens], int64
|
||||
block_table_ptr, # [max_num_reqs, max_num_blocks_per_req], int32 (flat)
|
||||
block_table_stride, # max_num_blocks_per_req
|
||||
block_size,
|
||||
slot_mapping_ptr, # [max_num_tokens], int64
|
||||
KV_CACHE_BLOCK_SIZE: tl.constexpr,
|
||||
BLOCKS_PER_KV_BLOCK: tl.constexpr,
|
||||
TOTAL_CP_WORLD_SIZE: tl.constexpr,
|
||||
TOTAL_CP_RANK: tl.constexpr,
|
||||
CP_KV_CACHE_INTERLEAVE_SIZE: tl.constexpr,
|
||||
PAD_ID: tl.constexpr,
|
||||
BLOCK_SIZE: tl.constexpr,
|
||||
):
|
||||
req_idx = tl.program_id(0)
|
||||
|
||||
if req_idx == tl.num_programs(0) - 1:
|
||||
# Pad remaining slots for CUDA graph compatibility.
|
||||
for i in range(num_tokens, max_num_tokens, BLOCK_SIZE):
|
||||
offsets = i + tl.arange(0, BLOCK_SIZE)
|
||||
tl.store(
|
||||
slot_mapping_ptr + offsets,
|
||||
PAD_ID,
|
||||
mask=offsets < max_num_tokens,
|
||||
)
|
||||
return
|
||||
|
||||
start_idx = tl.load(query_start_loc_ptr + req_idx).to(tl.int64)
|
||||
end_idx = tl.load(query_start_loc_ptr + req_idx + 1).to(tl.int64)
|
||||
|
||||
virtual_block_size = KV_CACHE_BLOCK_SIZE * TOTAL_CP_WORLD_SIZE
|
||||
row_offset = req_idx * block_table_stride
|
||||
for i in range(start_idx, end_idx, BLOCK_SIZE):
|
||||
offsets = i + tl.arange(0, BLOCK_SIZE)
|
||||
tl.store(
|
||||
slot_mapping_ptr + offsets,
|
||||
PAD_ID,
|
||||
mask=offsets < max_num_tokens,
|
||||
mask = offsets < end_idx
|
||||
pos = tl.load(positions_ptr + offsets, mask=mask, other=0)
|
||||
virtual_block_indices = pos // virtual_block_size
|
||||
virtual_block_offsets = pos - virtual_block_indices * virtual_block_size
|
||||
is_local = (
|
||||
virtual_block_offsets // CP_KV_CACHE_INTERLEAVE_SIZE
|
||||
) % TOTAL_CP_WORLD_SIZE == TOTAL_CP_RANK
|
||||
local_block_offsets = (
|
||||
virtual_block_offsets
|
||||
// (TOTAL_CP_WORLD_SIZE * CP_KV_CACHE_INTERLEAVE_SIZE)
|
||||
) * CP_KV_CACHE_INTERLEAVE_SIZE + (
|
||||
virtual_block_offsets % CP_KV_CACHE_INTERLEAVE_SIZE
|
||||
)
|
||||
return
|
||||
|
||||
start_idx = tl.load(query_start_loc_ptr + req_idx).to(tl.int64)
|
||||
end_idx = tl.load(query_start_loc_ptr + req_idx + 1).to(tl.int64)
|
||||
block_indices = (
|
||||
virtual_block_indices * BLOCKS_PER_KV_BLOCK
|
||||
+ local_block_offsets // block_size
|
||||
)
|
||||
block_numbers = tl.load(
|
||||
block_table_ptr + row_offset + block_indices,
|
||||
mask=mask & is_local,
|
||||
other=0,
|
||||
).to(tl.int64)
|
||||
slot_offsets = local_block_offsets % block_size
|
||||
slot_ids = block_numbers * block_size + slot_offsets
|
||||
slot_ids = tl.where(is_local, slot_ids, PAD_ID)
|
||||
tl.store(slot_mapping_ptr + offsets, slot_ids, mask=mask)
|
||||
|
||||
virtual_block_size = KV_CACHE_BLOCK_SIZE * TOTAL_CP_WORLD_SIZE
|
||||
row_offset = req_idx * block_table_stride
|
||||
for i in range(start_idx, end_idx, BLOCK_SIZE):
|
||||
offsets = i + tl.arange(0, BLOCK_SIZE)
|
||||
mask = offsets < end_idx
|
||||
pos = tl.load(positions_ptr + offsets, mask=mask, other=0)
|
||||
virtual_block_indices = pos // virtual_block_size
|
||||
virtual_block_offsets = pos - virtual_block_indices * virtual_block_size
|
||||
is_local = (
|
||||
virtual_block_offsets // CP_KV_CACHE_INTERLEAVE_SIZE
|
||||
) % TOTAL_CP_WORLD_SIZE == TOTAL_CP_RANK
|
||||
local_block_offsets = (
|
||||
virtual_block_offsets // (TOTAL_CP_WORLD_SIZE * CP_KV_CACHE_INTERLEAVE_SIZE)
|
||||
) * CP_KV_CACHE_INTERLEAVE_SIZE + (
|
||||
virtual_block_offsets % CP_KV_CACHE_INTERLEAVE_SIZE
|
||||
def dispatch( # type: ignore[override]
|
||||
self,
|
||||
*,
|
||||
block_table_stride: int,
|
||||
block_size: int,
|
||||
**compile_key_fields: int,
|
||||
) -> CompileKey:
|
||||
return self.CompileKey(
|
||||
**compile_key_fields,
|
||||
block_table_stride=triton_scalar_specialization_rep(block_table_stride),
|
||||
block_size=triton_scalar_specialization_rep(block_size),
|
||||
)
|
||||
|
||||
block_indices = (
|
||||
virtual_block_indices * BLOCKS_PER_KV_BLOCK
|
||||
+ local_block_offsets // block_size
|
||||
def get_warmup_keys(self, **dispatch_kwargs: int) -> list[CompileKey]:
|
||||
return self._trace_dispatch(self.dispatch)(**dispatch_kwargs)
|
||||
|
||||
def compile(self, compile_key: CompileKey) -> None:
|
||||
warmup = getattr(self.kernel, "warmup", None)
|
||||
assert warmup is not None
|
||||
int32_ptr = TritonWarmupTensor(torch.int32)
|
||||
int64_ptr = TritonWarmupTensor(torch.int64)
|
||||
warmup(
|
||||
2, # arbitrary, num_tokens in do_not_specialize
|
||||
2, # arbitrary, max_num_tokens in do_not_specialize
|
||||
int32_ptr,
|
||||
int64_ptr,
|
||||
int32_ptr,
|
||||
compile_key.block_table_stride,
|
||||
compile_key.block_size,
|
||||
int64_ptr,
|
||||
KV_CACHE_BLOCK_SIZE=compile_key.kv_cache_block_size,
|
||||
BLOCKS_PER_KV_BLOCK=compile_key.blocks_per_kv_block,
|
||||
TOTAL_CP_WORLD_SIZE=compile_key.total_cp_world_size,
|
||||
TOTAL_CP_RANK=compile_key.total_cp_rank,
|
||||
CP_KV_CACHE_INTERLEAVE_SIZE=compile_key.cp_kv_cache_interleave_size,
|
||||
PAD_ID=PAD_SLOT_ID,
|
||||
BLOCK_SIZE=self.triton_block_size,
|
||||
grid=(2,),
|
||||
)
|
||||
block_numbers = tl.load(
|
||||
block_table_ptr + row_offset + block_indices,
|
||||
mask=mask & is_local,
|
||||
other=0,
|
||||
).to(tl.int64)
|
||||
slot_offsets = local_block_offsets % block_size
|
||||
slot_ids = block_numbers * block_size + slot_offsets
|
||||
slot_ids = tl.where(is_local, slot_ids, PAD_ID)
|
||||
tl.store(slot_mapping_ptr + offsets, slot_ids, mask=mask)
|
||||
|
||||
def __call__(
|
||||
self,
|
||||
num_reqs: int,
|
||||
*args: Any,
|
||||
) -> None:
|
||||
self.kernel[(num_reqs + 1,)](
|
||||
*args,
|
||||
PAD_ID=PAD_SLOT_ID,
|
||||
BLOCK_SIZE=self.triton_block_size,
|
||||
)
|
||||
|
||||
|
||||
_COMPUTE_SLOT_MAPPING_KERNEL = ComputeSlotMappingKernel()
|
||||
|
||||
@@ -74,7 +74,7 @@ class CPUModelRunner(GPUModelRunner):
|
||||
|
||||
import vllm.v1.worker.block_table
|
||||
|
||||
vllm.v1.worker.block_table._compute_slot_mapping_kernel = (
|
||||
vllm.v1.worker.block_table._COMPUTE_SLOT_MAPPING_KERNEL.kernel = (
|
||||
cpu_tl.compute_slot_mapping_kernel
|
||||
)
|
||||
|
||||
|
||||
@@ -51,6 +51,7 @@ from vllm.model_executor.offloader import (
|
||||
get_offloader,
|
||||
set_offloader,
|
||||
)
|
||||
from vllm.model_executor.warmup.jit_warmup import JitWarmupRegistry
|
||||
from vllm.multimodal import MULTIMODAL_REGISTRY
|
||||
from vllm.multimodal.encoder_budget import (
|
||||
MultiModalBudget,
|
||||
@@ -170,6 +171,7 @@ class GPUModelRunner(LoRAModelRunnerMixin):
|
||||
self.speculative_config is not None and self.speculative_config.use_dspark()
|
||||
)
|
||||
self.observability_config = vllm_config.observability_config
|
||||
self.jit_warmup_registry = JitWarmupRegistry(vllm_config)
|
||||
|
||||
self.device = device
|
||||
self.dtype = self.model_config.dtype
|
||||
|
||||
@@ -100,6 +100,7 @@ from vllm.model_executor.offloader import (
|
||||
get_offloader,
|
||||
set_offloader,
|
||||
)
|
||||
from vllm.model_executor.warmup.jit_warmup import JitWarmupRegistry
|
||||
from vllm.multimodal import MULTIMODAL_REGISTRY
|
||||
from vllm.multimodal.encoder_budget import MultiModalBudget
|
||||
from vllm.multimodal.inputs import (
|
||||
@@ -516,6 +517,7 @@ class GPUModelRunner(
|
||||
self.scheduler_config = vllm_config.scheduler_config
|
||||
self.speculative_config = vllm_config.speculative_config
|
||||
self.observability_config = vllm_config.observability_config
|
||||
self.jit_warmup_registry = JitWarmupRegistry(vllm_config)
|
||||
|
||||
model_config = self.model_config
|
||||
cache_config = self.cache_config
|
||||
@@ -750,34 +752,37 @@ class GPUModelRunner(
|
||||
self._init_kernel_block_sizes = [placeholder_block_size]
|
||||
self._init_max_num_blocks = [placeholder_max_num_blocks]
|
||||
self._init_slot_mapping_modes = [SlotMappingMode.TOKEN_TO_KV_SLOT]
|
||||
self.input_batch = InputBatch(
|
||||
max_num_reqs=self.max_num_reqs,
|
||||
# We need to use the encoder length for encoder-decoder
|
||||
# because of KV cache for cross-attention.
|
||||
max_model_len=max(self.max_model_len, self.max_encoder_len),
|
||||
max_num_batched_tokens=self.max_num_tokens,
|
||||
device=self.device,
|
||||
vocab_size=self.model_config.get_vocab_size(),
|
||||
block_sizes=[placeholder_block_size],
|
||||
kernel_block_sizes=[placeholder_block_size],
|
||||
max_num_blocks_per_req=[placeholder_max_num_blocks],
|
||||
num_spec_tokens=self.num_spec_tokens,
|
||||
logitsprocs=build_logitsprocs(
|
||||
self.vllm_config,
|
||||
self.device,
|
||||
PIN_MEMORY,
|
||||
self.is_pooling_model,
|
||||
custom_logitsprocs,
|
||||
),
|
||||
# We currently don't know whether a particular custom logits processor
|
||||
# uses output token ids so we set this conservatively. Thinking-budget
|
||||
# tracking is requested dynamically when a budgeted request is in the batch.
|
||||
logitsprocs_need_output_token_ids=bool(custom_logitsprocs),
|
||||
is_pooling_model=self.is_pooling_model,
|
||||
cp_kv_cache_interleave_size=self.parallel_config.cp_kv_cache_interleave_size,
|
||||
reasoning_config=self.vllm_config.reasoning_config,
|
||||
use_replayssm=self.cache_config.use_replayssm,
|
||||
)
|
||||
# Capture warmup providers registered by the initial placeholder InputBatch
|
||||
with self.jit_warmup_registry.activate():
|
||||
self.input_batch = InputBatch(
|
||||
max_num_reqs=self.max_num_reqs,
|
||||
# We need to use the encoder length for encoder-decoder
|
||||
# because of KV cache for cross-attention.
|
||||
max_model_len=max(self.max_model_len, self.max_encoder_len),
|
||||
max_num_batched_tokens=self.max_num_tokens,
|
||||
device=self.device,
|
||||
vocab_size=self.model_config.get_vocab_size(),
|
||||
block_sizes=[placeholder_block_size],
|
||||
kernel_block_sizes=[placeholder_block_size],
|
||||
max_num_blocks_per_req=[placeholder_max_num_blocks],
|
||||
num_spec_tokens=self.num_spec_tokens,
|
||||
logitsprocs=build_logitsprocs(
|
||||
self.vllm_config,
|
||||
self.device,
|
||||
PIN_MEMORY,
|
||||
self.is_pooling_model,
|
||||
custom_logitsprocs,
|
||||
),
|
||||
# We currently don't know whether a particular custom logits processor
|
||||
# uses output token ids so we set this conservatively. Thinking-budget
|
||||
# tracking is requested dynamically when a budgeted request is in the
|
||||
# batch.
|
||||
logitsprocs_need_output_token_ids=bool(custom_logitsprocs),
|
||||
is_pooling_model=self.is_pooling_model,
|
||||
cp_kv_cache_interleave_size=self.parallel_config.cp_kv_cache_interleave_size,
|
||||
reasoning_config=self.vllm_config.reasoning_config,
|
||||
use_replayssm=self.cache_config.use_replayssm,
|
||||
)
|
||||
|
||||
# Separate cuda stream for overlapping transfer of sampled token ids from
|
||||
# GPU to CPU when async scheduling is enabled.
|
||||
@@ -7416,24 +7421,26 @@ class GPUModelRunner(
|
||||
self._init_kernel_block_sizes = kernel_block_sizes
|
||||
self._init_max_num_blocks = max_num_blocks
|
||||
self._init_slot_mapping_modes = slot_mapping_modes
|
||||
self.input_batch = InputBatch(
|
||||
max_num_reqs=self.max_num_reqs,
|
||||
max_model_len=max_model_len,
|
||||
max_num_batched_tokens=self.max_num_tokens,
|
||||
device=self.device,
|
||||
vocab_size=self.model_config.get_vocab_size(),
|
||||
block_sizes=block_sizes,
|
||||
kernel_block_sizes=kernel_block_sizes,
|
||||
max_num_blocks_per_req=max_num_blocks,
|
||||
num_spec_tokens=self.num_spec_tokens,
|
||||
logitsprocs=self.input_batch.logitsprocs,
|
||||
logitsprocs_need_output_token_ids=self.input_batch.logitsprocs_need_output_token_ids,
|
||||
is_pooling_model=self.is_pooling_model,
|
||||
cp_kv_cache_interleave_size=self.parallel_config.cp_kv_cache_interleave_size,
|
||||
reasoning_config=self.vllm_config.reasoning_config,
|
||||
use_replayssm=self.cache_config.use_replayssm,
|
||||
slot_mapping_modes=slot_mapping_modes,
|
||||
)
|
||||
# Capture warmup providers registered after final KV-cache geometry is known
|
||||
with self.jit_warmup_registry.activate():
|
||||
self.input_batch = InputBatch(
|
||||
max_num_reqs=self.max_num_reqs,
|
||||
max_model_len=max_model_len,
|
||||
max_num_batched_tokens=self.max_num_tokens,
|
||||
device=self.device,
|
||||
vocab_size=self.model_config.get_vocab_size(),
|
||||
block_sizes=block_sizes,
|
||||
kernel_block_sizes=kernel_block_sizes,
|
||||
max_num_blocks_per_req=max_num_blocks,
|
||||
num_spec_tokens=self.num_spec_tokens,
|
||||
logitsprocs=self.input_batch.logitsprocs,
|
||||
logitsprocs_need_output_token_ids=self.input_batch.logitsprocs_need_output_token_ids,
|
||||
is_pooling_model=self.is_pooling_model,
|
||||
cp_kv_cache_interleave_size=self.parallel_config.cp_kv_cache_interleave_size,
|
||||
reasoning_config=self.vllm_config.reasoning_config,
|
||||
use_replayssm=self.cache_config.use_replayssm,
|
||||
slot_mapping_modes=slot_mapping_modes,
|
||||
)
|
||||
|
||||
assert self._init_block_sizes == block_sizes, (
|
||||
f"InputBatch block_sizes {self._init_block_sizes} != "
|
||||
|
||||
Reference in New Issue
Block a user