[Core][V1] Support trace_decode_token_ids for deterministic decode replay (#46701)

Signed-off-by: ianzxliu <ianzxliu@tencent.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: Zhixia Liu <liuzhixia.nju@gmail.com>
Co-authored-by: ianzxliu <ianzxliu@tencent.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
This commit is contained in:
Zhixia Liu
2026-08-20 15:12:43 +08:00
committed by GitHub
co-authored by ianzxliu Claude aoshen02 Codex Nick Hill
parent 754e1c3de4
commit a1c5b1fd9f
13 changed files with 931 additions and 9 deletions
@@ -0,0 +1,60 @@
# Trace Replay
Trace replay forces the engine to emit a predetermined token sequence during decoding while computing real logprobs from the model's unmodified logit distribution. The primary use case is comparing logprob distributions across different configurations:
- **Inference config diff**: compare logprobs between different quantization schemes, tensor parallelism layouts, or attention backends for the same model.
- **Train vs inference diff**: replay a training-time token sequence through the inference engine to detect logprob divergence caused by numerical differences between training and serving frameworks.
## Requirements
Trace replay reserves a per-request trace buffer, so it is off by default. Enable it with `--enable-trace-replay` (or `LLM(..., enable_trace_replay=True)`). It is only supported by model runner V2; requests are rejected when either condition is not met.
## Usage
```python
from vllm import LLM, SamplingParams
llm = LLM(model="Qwen/Qwen3-0.6B", enable_trace_replay=True)
# Token sequence captured from a previous run or training log
trace_tokens = [15, 284, 1026, 374]
params = SamplingParams(
trace_decode_token_ids=trace_tokens,
logprobs=5,
)
outputs = llm.generate(["Once upon a time"], sampling_params=params)
for token, logprob in zip(
outputs[0].outputs[0].token_ids,
outputs[0].outputs[0].logprobs,
):
print(f"token={token} logprob={logprob[token].logprob:.4f}")
```
The output tokens will always be `[15, 284, 1026, 374]`. The logprobs reflect the model's true probability for each forced token under the current inference configuration.
To compare configurations, run the same trace against two engine setups and diff the per-token logprobs.
## Behavior
When `trace_decode_token_ids` is set:
- `max_tokens` is automatically set to the trace length.
- The trace is truncated if it does not fit within `max_model_len` given the prompt length.
- All stop conditions (EOS, stop strings, stop token IDs) are disabled.
- Generation produces exactly the trace tokens, then stops.
## Limitations
`trace_decode_token_ids` is incompatible with the following features (raises `ValueError`):
| Feature | Reason |
| --- | --- |
| `n > 1` | Trace replay produces a single deterministic sequence |
| `prompt_logprobs` | Expanded logit layout conflicts with trace kernel indexing |
| Speculative decoding | Multi-token speculation conflicts with single-token trace stepping |
| Structured outputs | Grammar constraints conflict with forced token injection |
| `repetition_detection` | Would terminate the request on repeated patterns in the trace |
| `thinking_token_budget` | Logit masking corrupts logprobs for trace tokens |
| `bad_words` | Logit masking corrupts logprobs for trace tokens |
+195
View File
@@ -0,0 +1,195 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""
Trace-replay with vLLM offline inference.
Trace-replay lets you supply a known sequence of decode token IDs alongside
the prompt. Instead of sampling from the model distribution, the engine
injects each decode token deterministically, step by step. All other
outputs — logprobs, token ranks, text decoding — are computed faithfully
from the real logit distribution for that token.
How it works:
1. Start the engine with ``--enable-trace-replay`` (or
``LLM(..., enable_trace_replay=True)``).
2. Set ``SamplingParams.trace_decode_token_ids`` to the list of decode
token IDs you want to force.
3. The engine will output exactly those tokens and stop. ``max_tokens``
is overwritten with the trace length, and the trace is truncated if it
does not fit within ``max_model_len``. EOS tokens inside the trace
sequence do **not** halt generation early.
Requires model runner V2. Requests are rejected with ``ValueError`` when
combined with any of:
* n > 1 * Speculative decoding
* prompt_logprobs * Structured outputs
* repetition_detection * thinking_token_budget
* bad_words
Typical use-cases:
* Reproduce exact outputs from a previous run for benchmarking.
* Compute logprobs for an already-known output (e.g. reference answers).
* Dataset annotation: given (prompt, response) pairs, obtain per-token
logprob scores without altering the response.
Usage:
python examples/generate/trace_replay_offline.py
python examples/generate/trace_replay_offline.py --model facebook/opt-125m
"""
import argparse
from vllm import LLM, SamplingParams
DEFAULT_PROMPT = "Hello, my name is"
def build_llm(args: argparse.Namespace) -> LLM:
"""Construct an LLM from common CLI args."""
llm_kwargs: dict = {
"model": args.model,
"trust_remote_code": args.trust_remote_code,
"tensor_parallel_size": args.tensor_parallel_size,
"enforce_eager": args.enforce_eager,
"gpu_memory_utilization": args.gpu_memory_utilization,
"max_num_seqs": args.max_num_seqs,
"enable_trace_replay": True,
}
if args.max_model_len is not None:
llm_kwargs["max_model_len"] = args.max_model_len
return LLM(**llm_kwargs)
def run_normal_generation(llm: LLM, prompt: str, max_tokens: int) -> list[int]:
"""Run a standard greedy generation and return the output token IDs."""
sampling_params = SamplingParams(
temperature=0.0,
max_tokens=max_tokens,
# Request logprobs for the greedy token at each step so we can
# compare them against the trace-replay logprobs below.
logprobs=1,
)
outputs = llm.generate([prompt], sampling_params=sampling_params)
result = outputs[0].outputs[0]
output_token_ids = list(result.token_ids)
print("[Normal generation]")
print(f" Prompt : {prompt!r}")
print(f" Output token IDs : {output_token_ids}")
print(f" Output text : {result.text!r}")
return output_token_ids
def run_trace_replay(llm: LLM, prompt: str, decode_token_ids: list[int]) -> None:
"""Replay a known decode sequence and print per-token logprobs."""
sampling_params = SamplingParams(
# Provide the decode tokens to replay.
trace_decode_token_ids=decode_token_ids,
# Request top-5 logprobs so we can inspect the distribution.
logprobs=5,
)
outputs = llm.generate([prompt], sampling_params=sampling_params)
result = outputs[0].outputs[0]
replayed_ids = list(result.token_ids)
print("\n[Trace-replay]")
print(f" Requested decode token IDs : {decode_token_ids}")
print(f" Replayed output token IDs : {replayed_ids}")
print(f" Replayed output text : {result.text!r}")
# Verify the replayed tokens match exactly.
assert replayed_ids == decode_token_ids, (
f"Mismatch!\n expected: {decode_token_ids}\n got: {replayed_ids}"
)
print(" Replayed tokens match the requested trace exactly.")
# Show per-token logprobs (computed from the real distribution).
if result.logprobs:
print("\n Per-token logprobs (trace token):")
for step, (token_id, logprob_dict) in enumerate(
zip(replayed_ids, result.logprobs)
):
sampled_lp = logprob_dict.get(token_id)
lp_value = f"{sampled_lp.logprob:.4f}" if sampled_lp is not None else "n/a"
rank = sampled_lp.rank if sampled_lp is not None else "n/a"
print(
f" step {step:2d}: token_id={token_id:6d} "
f"logprob={lp_value} rank={rank}"
)
def run_demo(args: argparse.Namespace) -> None:
print(f"Loading model: {args.model}\n")
llm = build_llm(args)
prompt = args.prompt
print("=" * 60)
print("Step 1 — Normal greedy generation (captures decode tokens)")
print("=" * 60)
decode_token_ids = run_normal_generation(llm, prompt, max_tokens=8)
print("\n" + "=" * 60)
print("Step 2 — Trace-replay with the captured tokens")
print("=" * 60)
run_trace_replay(llm, prompt, decode_token_ids)
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Trace-replay with vLLM offline inference"
)
parser.add_argument(
"--prompt",
type=str,
default=DEFAULT_PROMPT,
help="Text prompt to use for generation (default: %(default)r)",
)
parser.add_argument(
"--model",
type=str,
default="facebook/opt-125m",
help="Name or path of the HuggingFace model to use",
)
parser.add_argument(
"--trust-remote-code",
action="store_true",
help="Trust remote code from HuggingFace",
)
parser.add_argument(
"--tensor-parallel-size",
type=int,
default=1,
help="Number of tensor parallel replicas",
)
parser.add_argument(
"--enforce-eager",
action="store_true",
help="Always use eager-mode PyTorch (disable CUDA graph)",
)
parser.add_argument(
"--gpu-memory-utilization",
type=float,
default=0.9,
help="Fraction of GPU memory to use",
)
parser.add_argument(
"--max-model-len",
type=int,
default=None,
help="Model context length",
)
parser.add_argument(
"--max-num-seqs",
type=int,
default=256,
help="Maximum number of sequences per iteration",
)
return parser.parse_args()
def main() -> None:
args = parse_args()
run_demo(args)
if __name__ == "__main__":
main()
@@ -0,0 +1,110 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import time
from types import SimpleNamespace
from unittest.mock import patch
import pytest
from vllm import SamplingParams
from vllm.config import VllmConfig
from vllm.exceptions import VLLMValidationError
from vllm.v1.core.sched.utils import check_stop
from vllm.v1.engine.input_processor import InputProcessor
from vllm.v1.request import Request, RequestStatus
pytestmark = pytest.mark.skip_global_cleanup
def _make_request(params: SamplingParams) -> Request:
return Request(
request_id="req-0",
client_index=0,
prompt_token_ids=[1, 2, 3],
sampling_params=params,
pooling_params=None,
arrival_time=time.time(),
)
def _normalize(params: SamplingParams, prompt_len: int, max_model_len: int) -> None:
processor = SimpleNamespace(
model_config=SimpleNamespace(max_model_len=max_model_len)
)
InputProcessor._normalize_trace_replay_params(processor, params, prompt_len)
def test_normalize_trace_replay_params():
params = SamplingParams(
max_tokens=16,
min_tokens=10,
stop_token_ids=[20],
trace_decode_token_ids=[10, 20, 30],
)
params.update_from_generation_config({}, eos_token_id=20)
_normalize(params, prompt_len=3, max_model_len=128)
assert params.max_tokens == 3
assert params.min_tokens == 0
assert params.ignore_eos is True
assert params.eos_token_id is None
assert params.stop_token_ids == []
assert params.all_stop_token_ids == set()
request = _make_request(params)
assert request.max_tokens == 3
request.append_output_token_ids([10, 20, 30])
assert check_stop(request, max_model_len=128)
assert request.status == RequestStatus.FINISHED_LENGTH_CAPPED
def test_trace_longer_than_remaining_context_is_truncated():
"""The trace is staged into a max_model_len-wide row, so it must be cut.
An explicitly set max_tokens is never clamped to max_model_len, so it cannot
bound the staged write.
"""
params = SamplingParams(max_tokens=100, trace_decode_token_ids=list(range(20)))
_normalize(params, prompt_len=6, max_model_len=10)
assert params.trace_decode_token_ids == [0, 1, 2, 3]
assert params.max_tokens == 4
def _validate(enable_trace_replay: bool) -> None:
"""Run _validate_params' trace gating with the rest of verify() stubbed."""
processor = SimpleNamespace(
model_config=SimpleNamespace(
return_sampling_mask=False,
enable_trace_replay=enable_trace_replay,
),
vllm_config=SimpleNamespace(reasoning_config=None),
speculative_config=None,
structured_outputs_config=None,
tokenizer=None,
)
params = SamplingParams(trace_decode_token_ids=[1, 2, 3])
with patch.object(SamplingParams, "verify"):
InputProcessor._validate_params(processor, params, ("generate",))
def test_trace_request_rejected_when_feature_disabled():
with pytest.raises(VLLMValidationError, match="--enable-trace-replay"):
_validate(enable_trace_replay=False)
def test_trace_request_accepted_when_feature_enabled():
_validate(enable_trace_replay=True)
def test_trace_replay_requires_v2_model_runner():
config = SimpleNamespace(
model_config=SimpleNamespace(enable_trace_replay=True),
use_v2_model_runner=False,
)
with pytest.raises(ValueError, match="trace replay requires Model Runner V2"):
VllmConfig._verify_trace_replay_config(config)
+111
View File
@@ -0,0 +1,111 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import pytest
from vllm import SamplingParams
from vllm.exceptions import VLLMValidationError
from vllm.sampling_params import StructuredOutputsParams
pytestmark = pytest.mark.skip_global_cleanup
# ---------------------------------------------------------------------------
# SamplingParams field
# ---------------------------------------------------------------------------
def test_sampling_params_trace_field_defaults_to_none():
params = SamplingParams(max_tokens=10)
assert params.trace_decode_token_ids is None
def test_sampling_params_trace_field_accepts_list():
ids = [100, 200, 300]
params = SamplingParams(trace_decode_token_ids=ids)
assert params.trace_decode_token_ids == ids
def test_sampling_params_trace_field_preserved_by_clone():
ids = [1, 2, 3]
params = SamplingParams(trace_decode_token_ids=ids)
cloned = params.clone()
assert cloned.trace_decode_token_ids == ids
assert cloned.trace_decode_token_ids is not params.trace_decode_token_ids
def test_sampling_params_trace_field_rejects_empty_list():
params = SamplingParams(trace_decode_token_ids=[])
with pytest.raises(ValueError, match="non-empty"):
params._validate_trace_replay(
_make_model_config(vocab_size=100), speculative_config=None
)
def test_sampling_params_trace_field_requires_single_output():
params = SamplingParams(n=2, trace_decode_token_ids=[1])
with pytest.raises(ValueError, match="requires n=1"):
params._validate_trace_replay(
_make_model_config(vocab_size=100), speculative_config=None
)
@pytest.mark.parametrize("invalid_ids", [[-1, 5], [1, "2"]])
def test_sampling_params_trace_field_rejects_invalid_token_ids(invalid_ids):
params = SamplingParams(trace_decode_token_ids=invalid_ids)
with pytest.raises(ValueError, match="non-negative integers"):
params._validate_trace_replay(
_make_model_config(vocab_size=100), speculative_config=None
)
def _make_model_config(vocab_size: int):
from unittest.mock import Mock
model_config = Mock()
model_config.get_vocab_size = lambda: vocab_size
return model_config
def test_validate_trace_replay_accepts_in_vocab():
params = SamplingParams(trace_decode_token_ids=[0, 50, 99])
# Should not raise.
params._validate_trace_replay(
_make_model_config(vocab_size=100), speculative_config=None
)
def test_validate_trace_replay_rejects_out_of_vocab():
# The non-negative check passes at construction, but the token id exceeds
# the vocabulary; verify() must reject it before it reaches the sampler.
params = SamplingParams(trace_decode_token_ids=[0, 100])
with pytest.raises(VLLMValidationError, match="out-of-vocab"):
params._validate_trace_replay(
_make_model_config(vocab_size=100), speculative_config=None
)
def test_validate_trace_replay_noop_when_unset():
params = SamplingParams(max_tokens=4)
# Should not raise when the field is unset.
params._validate_trace_replay(
_make_model_config(vocab_size=100), speculative_config=None
)
def test_trace_decode_token_ids_rejects_speculative_decoding():
params = SamplingParams(trace_decode_token_ids=[1])
with pytest.raises(ValueError, match="not supported with speculative decoding"):
params._validate_trace_replay(
_make_model_config(vocab_size=100), speculative_config=object()
)
def test_trace_decode_token_ids_rejects_structured_outputs():
params = SamplingParams(
trace_decode_token_ids=[1],
structured_outputs=StructuredOutputsParams(json_object=True),
)
with pytest.raises(ValueError, match="not supported with structured outputs"):
params._validate_trace_replay(
_make_model_config(vocab_size=100), speculative_config=None
)
+195
View File
@@ -0,0 +1,195 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Tests for Model Runner V2 inference trace-replay.
Trace-replay forces the sampler to emit a predetermined sequence of decode
token IDs. The replay step for a request is derived from GPU state as
``total_len - prompt_len`` (output tokens produced so far), and the trace token
overwrites the sampled token in place before logprobs are computed.
"""
from types import SimpleNamespace
from typing import cast
import pytest
import torch
pytest.importorskip("triton")
if not torch.cuda.is_available():
pytest.skip("CUDA required for trace-replay tests", allow_module_level=True)
from vllm.sampling_params import SamplingParams
from vllm.v1.worker.gpu.sample.trace_replay import (
TraceReplayState,
apply_trace_tokens,
)
from vllm.v1.worker.gpu.states import RequestState
DEVICE = "cuda"
TEST_MAX_MODEL_LEN = 2048
def _i32(x) -> torch.Tensor:
return torch.tensor(x, dtype=torch.int32, device=DEVICE)
def _i64(x) -> torch.Tensor:
return torch.tensor(x, dtype=torch.int64, device=DEVICE)
def _trace_state(max_num_reqs: int) -> TraceReplayState:
"""Build a state whose req_states exposes the buffers apply_trace reads.
total_len/prompt_len are written by the caller to position each request at
the desired replay step.
"""
req_states = cast(
RequestState,
SimpleNamespace(
max_num_reqs=max_num_reqs,
max_model_len=TEST_MAX_MODEL_LEN,
device=torch.device(DEVICE),
total_len=SimpleNamespace(
gpu=torch.zeros(max_num_reqs, dtype=torch.int32, device=DEVICE)
),
prompt_len=SimpleNamespace(
gpu=torch.zeros(max_num_reqs, dtype=torch.int32, device=DEVICE)
),
),
)
return TraceReplayState(req_states)
def _set_lens(state: TraceReplayState, total_len, prompt_len) -> None:
"""Position each request at a replay step via total_len - prompt_len."""
state.req_states.total_len.gpu[: len(total_len)] = _i32(total_len)
state.req_states.prompt_len.gpu[: len(prompt_len)] = _i32(prompt_len)
# ------------------------------ Kernel ------------------------------------
def test_replay_overwrites_sampled_at_each_step():
"""The trace token for the current step replaces the sampled token."""
trace = [[100, 101, 102], [200, 201, 202]]
trace_len = _i32([3, 3])
trace_token_ids = torch.zeros(
2, TEST_MAX_MODEL_LEN, dtype=torch.int32, device=DEVICE
)
for i, t in enumerate(trace):
trace_token_ids[i, : len(t)] = _i32(t)
prompt_len = _i32([5, 8])
idx_mapping = _i32([0, 1])
for step in range(3):
sampled = _i64([-7, -7]) # sentinel that must be overwritten
total_len = _i32([5 + step, 8 + step])
apply_trace_tokens(
sampled, idx_mapping, trace_token_ids, trace_len, total_len, prompt_len
)
assert sampled.tolist() == [trace[0][step], trace[1][step]]
def test_past_end_of_trace_leaves_sampled_untouched():
"""Once step >= trace_len, the sampler's own token is kept."""
trace_token_ids = torch.zeros(
1, TEST_MAX_MODEL_LEN, dtype=torch.int32, device=DEVICE
)
trace_token_ids[0, :2] = _i32([100, 101])
trace_len = _i32([2])
prompt_len = _i32([4])
idx_mapping = _i32([0])
sampled = _i64([999])
total_len = _i32([4 + 2]) # step == 2 == trace_len -> out of range
apply_trace_tokens(
sampled, idx_mapping, trace_token_ids, trace_len, total_len, prompt_len
)
assert sampled.tolist() == [999]
def test_non_trace_request_untouched():
"""trace_len == 0 means the request never uses replay."""
trace_token_ids = torch.zeros(
1, TEST_MAX_MODEL_LEN, dtype=torch.int32, device=DEVICE
)
trace_len = _i32([0])
prompt_len = _i32([3])
idx_mapping = _i32([0])
sampled = _i64([42])
total_len = _i32([3])
apply_trace_tokens(
sampled, idx_mapping, trace_token_ids, trace_len, total_len, prompt_len
)
assert sampled.tolist() == [42]
def test_idx_mapping_indirection_and_negative_skip():
"""batch_idx -> req_state_idx indirection, and negative entries are skipped."""
# req_state 0: no trace; req_state 1: trace [500, 501].
trace_token_ids = torch.zeros(
2, TEST_MAX_MODEL_LEN, dtype=torch.int32, device=DEVICE
)
trace_token_ids[1, :2] = _i32([500, 501])
trace_len = _i32([0, 2])
prompt_len = _i32([10, 6])
# batch: [maps to state 1, masked (-1), maps to state 0].
idx_mapping = _i32([1, -1, 0])
sampled = _i64([1, 2, 3])
# Indexed by req_state_idx: state 0 at total_len=10, state 1 at step 1.
total_len = _i32([10, 6 + 1])
apply_trace_tokens(
sampled, idx_mapping, trace_token_ids, trace_len, total_len, prompt_len
)
# batch 0 -> state 1 step 1 -> 501; batch 1 masked; batch 2 -> state 0 no trace.
assert sampled.tolist() == [501, 2, 3]
# --------------------------- TraceReplayState -----------------------------
def test_state_end_to_end():
"""add_request -> apply_staged_writes -> apply_trace overwrites correctly."""
state = _trace_state(4)
state.add_request(0, SamplingParams(trace_decode_token_ids=[11, 22, 33]))
state.add_request(1, SamplingParams()) # no trace
state.apply_staged_writes()
idx_mapping = _i32([0, 1])
sampled = _i64([-1, -1])
_set_lens(state, [7 + 1, 7, 0, 0], [7, 7, 0, 0]) # req 0 at step 1
state.apply_trace(sampled, idx_mapping)
assert sampled.tolist() == [22, -1]
def test_state_leaves_non_trace_batch_unchanged():
"""Requests with trace_len == 0 remain unchanged after a trace was seen."""
state = _trace_state(4)
state.add_request(0, SamplingParams(trace_decode_token_ids=[11, 22]))
state.add_request(1, SamplingParams())
state.apply_staged_writes()
# Only the non-trace request (state 1) is in this batch.
idx_mapping = _i32([1])
sampled = _i64([555])
_set_lens(state, [0, 7, 0, 0], [7, 7, 0, 0])
state.apply_trace(sampled, idx_mapping)
assert sampled.tolist() == [555]
def test_slot_reuse_clears_trace():
"""Reusing a slot for a non-trace request must not replay stale tokens."""
state = _trace_state(2)
state.add_request(0, SamplingParams(trace_decode_token_ids=[11, 22]))
state.apply_staged_writes()
# Slot 0 reused by a request without a trace.
state.add_request(0, SamplingParams())
state.apply_staged_writes()
idx_mapping = _i32([0])
sampled = _i64([888])
_set_lens(state, [3, 0], [3, 0])
state.apply_trace(sampled, idx_mapping)
assert sampled.tolist() == [888]
+7
View File
@@ -262,6 +262,12 @@ class ModelConfig:
equivalent exponential-race sampling. FP64 preserves lower-tail sampling
events that fp32 uniform/exponential draws can truncate, at the cost of
significantly lower throughput on most GPUs."""
enable_trace_replay: bool = False
"""Whether to allow requests to set
`SamplingParams.trace_decode_token_ids`, which forces decoding to follow a
predetermined token sequence while still computing real logprobs. Reserved
for debugging and RL workflows: enabling it reserves a per-request trace
buffer, so it is off by default."""
disable_sliding_window: bool = False
"""Whether to disable sliding window. If True, we will disable the sliding
window functionality of the model, capping to sliding window size. If the
@@ -424,6 +430,7 @@ class ModelConfig:
"return_sampling_mask",
"logprobs_mode",
"use_fp64_gumbel",
"enable_trace_replay",
"disable_cascade_attn",
"skip_tokenizer_init",
"served_model_name",
+8
View File
@@ -1091,6 +1091,13 @@ class VllmConfig:
"are normalized over the same nucleus as the sampling mask"
)
def _verify_trace_replay_config(self) -> None:
model_config = self.model_config
if model_config is None or not model_config.enable_trace_replay:
return
if not self.use_v2_model_runner:
raise ValueError("trace replay requires Model Runner V2")
def __post_init__(self):
"""Verify configs are valid & consistent with each other."""
@@ -1141,6 +1148,7 @@ class VllmConfig:
)
self._verify_sampling_replay_config()
self._verify_trace_replay_config()
if self.lora_config is not None:
self.lora_config.verify_with_model_config(self.model_config)
+5
View File
@@ -543,6 +543,7 @@ class EngineArgs:
max_logprobs: int = ModelConfig.max_logprobs
logprobs_mode: LogprobsMode = ModelConfig.logprobs_mode
use_fp64_gumbel: bool = ModelConfig.use_fp64_gumbel
enable_trace_replay: bool = ModelConfig.enable_trace_replay
disable_log_stats: bool = False
aggregate_engine_logging: bool = False
revision: str | None = ModelConfig.revision
@@ -884,6 +885,9 @@ class EngineArgs:
model_group.add_argument("--max-logprobs", **model_kwargs["max_logprobs"])
model_group.add_argument("--logprobs-mode", **model_kwargs["logprobs_mode"])
model_group.add_argument("--use-fp64-gumbel", **model_kwargs["use_fp64_gumbel"])
model_group.add_argument(
"--enable-trace-replay", **model_kwargs["enable_trace_replay"]
)
model_group.add_argument(
"--disable-sliding-window", **model_kwargs["disable_sliding_window"]
)
@@ -1760,6 +1764,7 @@ class EngineArgs:
max_logprobs=self.max_logprobs,
logprobs_mode=self.logprobs_mode,
use_fp64_gumbel=self.use_fp64_gumbel,
enable_trace_replay=self.enable_trace_replay,
disable_sliding_window=self.disable_sliding_window,
disable_cascade_attn=self.disable_cascade_attn,
skip_tokenizer_init=self.skip_tokenizer_init,
+72 -8
View File
@@ -346,14 +346,6 @@ class SamplingParams(
"""Arbitrary additional args, that can be used by custom sampling
implementations, plugins, etc. Not used by any in-tree sampling
implementations."""
routed_experts_prompt_start: int = 0
"""When enable_return_routed_experts is active, skip the first
routed_experts_prompt_start prompt tokens from the returned routing
data. In multi-turn agent scenarios, set this to the length of the
already-returned prefix to avoid duplicating routing for prompt tokens
covered by earlier turns. Default 0 returns routing for all prompt
tokens."""
# Fields used for bad words
bad_words: list[str] | None = None
"""Words that are not allowed to be generated. More precisely, only the
@@ -373,6 +365,19 @@ class SamplingParams(
'\\emoji \\emoji \\emoji ...'). This feature can detect such behavior
and terminate early, saving time and tokens."""
# Debugging / RL-specific parameters. Not intended for production serving.
routed_experts_prompt_start: int = 0
"""When enable_return_routed_experts is active, skip the first
routed_experts_prompt_start prompt tokens from the returned routing
data. In multi-turn agent scenarios, set this to the length of the
already-returned prefix to avoid duplicating routing for prompt tokens
covered by earlier turns. Default 0 returns routing for all prompt
tokens."""
trace_decode_token_ids: list[int] | None = None
"""If provided, forces the engine to emit this predetermined sequence of
token IDs during decoding instead of sampling randomly. Real logprobs are
still computed. Conflict checking is performed at the engine level."""
@staticmethod
def from_optional(
n: int | None = 1,
@@ -407,6 +412,8 @@ class SamplingParams(
repetition_detection: RepetitionDetectionParams | None = None,
logprob_token_ids: list[int] | None = None,
routed_experts_prompt_start: int = 0,
# Debugging / RL-specific parameters.
trace_decode_token_ids: list[int] | None = None,
) -> "SamplingParams":
if logit_bias is not None:
# Fast path uses a dict comprehension; on failure we iterate once
@@ -470,6 +477,7 @@ class SamplingParams(
skip_clone=skip_clone,
repetition_detection=repetition_detection,
routed_experts_prompt_start=routed_experts_prompt_start,
trace_decode_token_ids=trace_decode_token_ids,
)
def __post_init__(self) -> None:
@@ -780,6 +788,7 @@ class SamplingParams(
) -> None:
self._validate_logprobs(model_config)
self._validate_logit_bias(model_config)
self._validate_trace_replay(model_config, speculative_config)
self._validate_logits_processors(model_config)
self._validate_allowed_token_ids(tokenizer)
self._validate_spec_decode(speculative_config)
@@ -870,6 +879,61 @@ class SamplingParams(
value=invalid_token_ids,
)
def _validate_trace_replay(
self,
model_config: ModelConfig,
speculative_config: SpeculativeConfig | None,
) -> None:
"""Validate trace replay request compatibility."""
if self.trace_decode_token_ids is None:
return
if len(self.trace_decode_token_ids) == 0:
raise ValueError("trace_decode_token_ids must be a non-empty list.")
if self.n != 1:
raise ValueError("trace_decode_token_ids requires n=1.")
if not all(isinstance(t, int) and t >= 0 for t in self.trace_decode_token_ids):
raise ValueError(
"trace_decode_token_ids must contain non-negative integers."
)
if self.prompt_logprobs is not None:
raise ValueError(
"trace_decode_token_ids is not supported with prompt_logprobs."
)
if speculative_config is not None:
raise ValueError(
"trace_decode_token_ids is not supported with speculative decoding."
)
if self.structured_outputs is not None:
raise ValueError(
"trace_decode_token_ids is not supported with structured outputs."
)
if self.repetition_detection is not None:
raise ValueError(
"trace_decode_token_ids is not supported with repetition_detection."
)
if self.thinking_token_budget is not None:
raise ValueError(
"trace_decode_token_ids is not supported with thinking_token_budget."
)
if self.bad_words:
raise ValueError("trace_decode_token_ids is not supported with bad_words.")
vocab_size = model_config.get_vocab_size()
invalid_token_ids = [
token_id
for token_id in self.trace_decode_token_ids
if token_id < 0 or token_id >= vocab_size
]
if invalid_token_ids:
raise VLLMValidationError(
f"token_id(s) {invalid_token_ids} in trace_decode_token_ids "
f"contain out-of-vocab token ids. Vocabulary size: {vocab_size}",
parameter="trace_decode_token_ids",
value=invalid_token_ids,
)
def _validate_logits_processors(self, model_config: ModelConfig) -> None:
from vllm.v1.sample.logits_processor import (
validate_logits_processors_parameters,
+40 -1
View File
@@ -51,7 +51,6 @@ class InputProcessor:
self.speculative_config = vllm_config.speculative_config
self.structured_outputs_config = vllm_config.structured_outputs_config
self.observability_config = vllm_config.observability_config
self.use_v2_model_runner = vllm_config.use_v2_model_runner
self.generation_config_fields = model_config.try_get_generation_config()
@@ -128,6 +127,15 @@ class InputProcessor:
"not configured. Please set --reasoning-parser "
"and/or --reasoning-config to use thinking_token_budget."
)
if (
params.trace_decode_token_ids
and not self.model_config.enable_trace_replay
):
raise VLLMValidationError(
"trace_decode_token_ids is set but trace replay is not "
"enabled. Start the engine with --enable-trace-replay "
"to use it."
)
elif isinstance(params, PoolingParams):
supported_pooling_tasks = [
task for task in supported_tasks if task in POOLING_TASKS
@@ -156,6 +164,30 @@ class InputProcessor:
f"but got {type(params).__name__}"
)
def _normalize_trace_replay_params(
self, sampling_params: SamplingParams, prompt_len: int
) -> None:
"""Apply trace replay's generation semantics to request-local params."""
trace_token_ids = sampling_params.trace_decode_token_ids
assert trace_token_ids
assert sampling_params.max_tokens is not None
max_trace_len = max(self.model_config.max_model_len - prompt_len, 1)
trace_token_ids = trace_token_ids[:max_trace_len]
sampling_params.trace_decode_token_ids = trace_token_ids
# Apply this after the generation config so its EOS token cannot stop
# replay before the trace is exhausted.
sampling_params.max_tokens = min(
len(trace_token_ids), sampling_params.max_tokens
)
sampling_params.min_tokens = 0
sampling_params.ignore_eos = True
sampling_params._eos_token_id = None
sampling_params.stop = []
sampling_params.stop_token_ids = []
sampling_params._all_stop_token_ids = set()
def _validate_lora(self, lora_request: LoRARequest | None) -> None:
if lora_request is None:
return
@@ -340,6 +372,13 @@ class InputProcessor:
)
if self.tokenizer is not None:
sampling_params.update_from_tokenizer(self.tokenizer)
if sampling_params.trace_decode_token_ids:
self._normalize_trace_replay_params(
sampling_params,
length_from_prompt_token_ids_or_embeds(
prompt_token_ids, prompt_embeds
),
)
else:
pooling_params = params.clone()
+1
View File
@@ -408,6 +408,7 @@ class GPUModelRunner(LoRAModelRunnerMixin):
logprobs_mode=self.model_config.logprobs_mode,
num_speculative_tokens=self.decode_query_len,
use_fp64_gumbel=self.model_config.use_fp64_gumbel,
enable_trace_replay=self.model_config.enable_trace_replay,
reasoning_config=self.vllm_config.reasoning_config,
return_sampling_mask=self.model_config.return_sampling_mask,
)
+14
View File
@@ -26,6 +26,7 @@ from vllm.v1.worker.gpu.sample.output import SamplerOutput, SamplingMaskTensors
from vllm.v1.worker.gpu.sample.penalties import PenaltiesState
from vllm.v1.worker.gpu.sample.states import NO_LOGPROBS, SamplingStates
from vllm.v1.worker.gpu.sample.thinking_budget import ThinkingBudgetState
from vllm.v1.worker.gpu.sample.trace_replay import TraceReplayState
from vllm.v1.worker.gpu.states import RequestState
@@ -39,6 +40,7 @@ class Sampler:
logprobs_mode: LogprobsMode = "raw_logprobs",
num_speculative_tokens: int = 1,
use_fp64_gumbel: bool = False,
enable_trace_replay: bool = False,
reasoning_config: ReasoningConfig | None = None,
return_sampling_mask: bool = False,
):
@@ -53,6 +55,9 @@ class Sampler:
self.bad_words_state = BadWordsState(req_states)
self.logprob_token_ids_state = LogprobTokenIdsState(max_num_reqs, device)
self.thinking_budget_state = ThinkingBudgetState(req_states, reasoning_config)
self.trace_replay_state = (
TraceReplayState(req_states) if enable_trace_replay else None
)
self.needs_logits_processing = np.zeros(max_num_reqs, dtype=bool)
self.num_speculative_tokens = num_speculative_tokens
self.return_sampling_mask = return_sampling_mask
@@ -69,6 +74,8 @@ class Sampler:
self.bad_words_state.add_request(req_idx, sampling_params)
self.logprob_token_ids_state.add_request(req_idx, sampling_params)
self.thinking_budget_state.add_request(req_idx, sampling_params)
if self.trace_replay_state is not None:
self.trace_replay_state.add_request(req_idx, sampling_params)
states = self.sampling_states
temperature = states.temperature.np[req_idx]
@@ -93,6 +100,8 @@ class Sampler:
self.bad_words_state.apply_staged_writes()
self.logprob_token_ids_state.apply_staged_writes()
self.thinking_budget_state.apply_staged_writes()
if self.trace_replay_state is not None:
self.trace_replay_state.apply_staged_writes()
def __call__(
self,
@@ -128,6 +137,11 @@ class Sampler:
return_logprobs=return_logprobs,
)
if self.trace_replay_state is not None:
# Overwrite sampled tokens with the replay trace up-front so that
# computed logprobs reflect the real distribution of the forced token.
self.trace_replay_state.apply_trace(sampled, idx_mapping)
if return_logprobs:
if self.logprobs_mode in PROCESSED_LOGPROBS_MODES:
logits = processed_logits
+113
View File
@@ -0,0 +1,113 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import torch
from vllm.sampling_params import SamplingParams
from vllm.triton_utils import tl, triton
from vllm.v1.worker.gpu.buffer_utils import StagedWriteTensor, UvaBackedTensor
from vllm.v1.worker.gpu.states import RequestState
class TraceReplayState:
"""Per-request state for inference trace-replay.
When a request carries ``SamplingParams.trace_decode_token_ids``, the
sampler overwrites the sampled token at each decode step with the
predetermined trace token, while real logprobs and ranks are still computed
from the unmodified logit distribution. The replay step for a request is
derived entirely from GPU state (``total_len - prompt_len``), so no CPU
synchronization or async placeholder handling is needed.
"""
def __init__(self, req_states: RequestState):
self.max_num_reqs = req_states.max_num_reqs
self.device = req_states.device
self.req_states = req_states
self.trace_token_ids = StagedWriteTensor(
(self.max_num_reqs, req_states.max_model_len),
dtype=torch.int32,
device=self.device,
uva_instead_of_gpu=True,
)
self.trace_len = UvaBackedTensor(self.max_num_reqs, dtype=torch.int32)
def add_request(self, req_idx: int, sampling_params: SamplingParams) -> None:
trace = sampling_params.trace_decode_token_ids
if trace is not None:
self.trace_len.np[req_idx] = len(trace)
self.trace_token_ids.stage_write(req_idx, 0, trace)
else:
self.trace_len.np[req_idx] = 0
def apply_staged_writes(self) -> None:
self.trace_len.copy_to_uva()
self.trace_token_ids.apply_write()
def apply_trace(
self,
sampled: torch.Tensor,
idx_mapping: torch.Tensor,
) -> None:
apply_trace_tokens(
sampled,
idx_mapping,
self.trace_token_ids.gpu,
self.trace_len.gpu,
self.req_states.total_len.gpu,
self.req_states.prompt_len.gpu,
)
@triton.jit
def _trace_replay_kernel(
sampled_ptr, # [num_reqs], int64, mutated in place
idx_mapping_ptr, # [num_reqs] batch_idx -> req_state_idx
trace_token_ids_ptr, # [max_num_reqs, max_model_len], int32
trace_token_ids_stride,
trace_len_ptr, # [max_num_reqs], int32
total_len_ptr, # [max_num_reqs], int32
prompt_len_ptr, # [max_num_reqs], int32
):
batch_idx = tl.program_id(0)
req_state_idx = tl.load(idx_mapping_ptr + batch_idx)
if req_state_idx < 0:
return
trace_len = tl.load(trace_len_ptr + req_state_idx)
if trace_len <= 0:
return
# The token being sampled now is output token number
# (total_len - prompt_len): total_len reflects tokens committed through the
# previous step (post_update runs after sampling).
step = tl.load(total_len_ptr + req_state_idx) - tl.load(
prompt_len_ptr + req_state_idx
)
if step < 0 or step >= trace_len:
return
token_id = tl.load(
trace_token_ids_ptr + req_state_idx * trace_token_ids_stride + step
)
tl.store(sampled_ptr + batch_idx, token_id.to(tl.int64))
def apply_trace_tokens(
sampled: torch.Tensor,
idx_mapping: torch.Tensor,
trace_token_ids: torch.Tensor,
trace_len: torch.Tensor,
total_len: torch.Tensor,
prompt_len: torch.Tensor,
) -> None:
"""Overwrite ``sampled`` in place with trace tokens for the current step."""
num_reqs = idx_mapping.shape[0]
_trace_replay_kernel[(num_reqs,)](
sampled,
idx_mapping,
trace_token_ids,
trace_token_ids.stride(0),
trace_len,
total_len,
prompt_len,
)