[Bugfix][Spec Decode] Fix EAGLE drafter multimodal encoder cache misses (#46315)
Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Roger Wang <hey@rogerw.io>
This commit is contained in:
co-authored by
Claude Opus 4.8
Roger Wang
parent
ac614587f5
commit
3e6529cc0e
@@ -1409,8 +1409,9 @@ class GPUModelRunner(LoRAModelRunnerMixin):
|
||||
input_batch.num_scheduled_tokens,
|
||||
input_batch.query_start_loc_np,
|
||||
input_batch.prefill_len_np,
|
||||
# +1 to consider the skew in eagle
|
||||
input_batch.num_computed_prefill_tokens_np + 1,
|
||||
input_batch.num_computed_prefill_tokens_np,
|
||||
# The EAGLE/MTP drafter reads one position ahead of the target.
|
||||
draft_lookahead=1,
|
||||
)
|
||||
|
||||
# Postprocess results and update request states.
|
||||
|
||||
Reference in New Issue
Block a user