[Spec Decode] Add kv_cache_dtype to speculative_config to control separately from target (#48787)

Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Michael Goin
2026-07-15 23:49:23 -06:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 7dc2698632
commit 8bfd683901
6 changed files with 42 additions and 1 deletions
+4
View File
@@ -1754,6 +1754,10 @@ class EngineArgs:
if self.speculative_config is None:
return None
self.speculative_config = {
k.replace("-", "_"): v for k, v in self.speculative_config.items()
}
# Note(Shangming): These parameters are not obtained from the cli arg
# '--speculative-config' and must be passed in when creating the engine
# config.