Runtime Draft Weight Update for Speculative Decoding (#46725)
Signed-off-by: vx120 <893600387@qq.com> Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com> Signed-off-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: crp0128 <191679376@qq.com> Co-authored-by: aoshen02 <aoshen@inferact.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
co-authored by
crp0128
aoshen02
Claude Opus 4.8
parent
481e481be7
commit
fc1c548093
@@ -49,6 +49,10 @@ update_request = WeightTransferUpdateRequest(
|
||||
)
|
||||
```
|
||||
|
||||
At the LLM/API layer, call `start_draft_weight_update()` instead of
|
||||
`start_weight_update()` to target the speculative draft model;
|
||||
`update_weights` / `finish_weight_update` are unchanged.
|
||||
|
||||
### WeightTransferUpdateInfo
|
||||
|
||||
The base `WeightTransferUpdateInfo` is a marker class for backend-specific update info:
|
||||
|
||||
Reference in New Issue
Block a user