[Spec Decode] Add kv_cache_dtype to speculative_config to control separately from target (#48787)
Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
7dc2698632
commit
8bfd683901
@@ -1754,6 +1754,10 @@ class EngineArgs:
|
||||
if self.speculative_config is None:
|
||||
return None
|
||||
|
||||
self.speculative_config = {
|
||||
k.replace("-", "_"): v for k, v in self.speculative_config.items()
|
||||
}
|
||||
|
||||
# Note(Shangming): These parameters are not obtained from the cli arg
|
||||
# '--speculative-config' and must be passed in when creating the engine
|
||||
# config.
|
||||
|
||||
Reference in New Issue
Block a user