[Bugfix][MRV2][Spec Decode] Honor dynamic K in autoregressive speculators, 1.2~1.3x kernel perf improvement (#57053)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
This commit is contained in:
@@ -272,6 +272,7 @@ class CudaGraphManager:
|
||||
if (
|
||||
speculative_config
|
||||
and speculative_config.uses_dynamic_speculative_decoding()
|
||||
and self.decode_query_len > self.vllm_config.num_speculative_tokens
|
||||
):
|
||||
# decode_query_len = num_speculative_steps + num_new_sampled_tokens
|
||||
# _per_step. Recover num_new_sampled_tokens_per_step
|
||||
|
||||
Reference in New Issue
Block a user