[Bugfix][Spec Decode]Preserve user --speculative-config overrides for speculators-format models (#42376)

Signed-off-by: leeyongjun <jqueen.astro@gmail.com>
Signed-off-by: Lee Yongjun <35302114+elwhyjay@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This commit is contained in:
Lee Yongjun
2026-08-21 09:06:47 -07:00
committed by GitHub
co-authored by gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
parent 574e6a00ef
commit 6f74337c47
2 changed files with 163 additions and 1 deletions
+7 -1
View File
@@ -673,8 +673,14 @@ def maybe_override_with_speculators(
speculative_config = SpeculatorsConfig.extract_vllm_speculative_config(
config_dict=config_dict
)
speculators_method = speculative_config["method"]
# Set the draft model to the speculators model
# Apply user --speculative-config overrides (e.g. attention_backend).
if isinstance(vllm_speculative_config, dict):
speculative_config.update(vllm_speculative_config)
# Lock fields dictated by the speculators format
speculative_config["method"] = speculators_method
speculative_config["model"] = model
# Override model and tokenizer with the verifier model from config