[Spec Decode][MRV2] Warm up top-k/top-p and greedy rejection kernels for speculative decoding (#59260)

Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
This commit is contained in:
qizixi
2026-10-14 14:59:21 -04:00
committed by GitHub
co-authored by Claude Opus 5.5 mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
parent ae94e9c9e8
commit c1407a7e86
3 changed files with 18 additions and 5 deletions
+5 -3
View File
@@ -458,9 +458,11 @@ class GPUModelRunner(LoRAModelRunnerMixin):
self.vllm_config, self.req_states, self.is_pooling_model, config_processors
)
if self.is_last_pp_rank and not self.is_pooling_model:
# V2 bypasses TopKTopPSampler, which registers native warmups.
# ROCm only: on CUDA these warmups add ~2 min to every engine start.
if current_platform.is_rocm():
# Speculative verification uses native top-k/top-p masking even
# when target-only sampling can use FlashInfer's fused sampler.
if current_platform.is_rocm() or (
current_platform.is_cuda() and self.speculative_config is not None
):
from vllm.v1.sample.ops.topk_topp_sampler import (
register_top_k_top_p_warmups,
)