[Bugfix][MRV2][Spec Decode] Honor dynamic K in autoregressive speculators, 1.2~1.3x kernel perf improvement (#57053)

Signed-off-by: yewentao256 <zhyanwentao@126.com>
This commit is contained in:
Wentao Ye
2026-10-05 14:54:11 -04:00
committed by GitHub
parent c32495d58f
commit 74c5cbcd7b
15 changed files with 190 additions and 18 deletions
+1
View File
@@ -272,6 +272,7 @@ class CudaGraphManager:
if (
speculative_config
and speculative_config.uses_dynamic_speculative_decoding()
and self.decode_query_len > self.vllm_config.num_speculative_tokens
):
# decode_query_len = num_speculative_steps + num_new_sampled_tokens
# _per_step. Recover num_new_sampled_tokens_per_step