[Bugfix][Spec Decode]Preserve user --speculative-config overrides for speculators-format models (#42376)
Signed-off-by: leeyongjun <jqueen.astro@gmail.com> Signed-off-by: Lee Yongjun <35302114+elwhyjay@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
parent
574e6a00ef
commit
6f74337c47
@@ -673,8 +673,14 @@ def maybe_override_with_speculators(
|
||||
speculative_config = SpeculatorsConfig.extract_vllm_speculative_config(
|
||||
config_dict=config_dict
|
||||
)
|
||||
speculators_method = speculative_config["method"]
|
||||
|
||||
# Set the draft model to the speculators model
|
||||
# Apply user --speculative-config overrides (e.g. attention_backend).
|
||||
if isinstance(vllm_speculative_config, dict):
|
||||
speculative_config.update(vllm_speculative_config)
|
||||
|
||||
# Lock fields dictated by the speculators format
|
||||
speculative_config["method"] = speculators_method
|
||||
speculative_config["model"] = model
|
||||
|
||||
# Override model and tokenizer with the verifier model from config
|
||||
|
||||
Reference in New Issue
Block a user