[Bugfix][MRV2] Treat padded prompt tails as spec-decode rows for hybrid models (#58434)

Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Nick Hill
2026-09-24 18:29:53 -07:00
committed by GitHub
co-authored by Claude Opus 5.5
parent 155b988373
commit 7dbd0a8d22
2 changed files with 88 additions and 1 deletions
@@ -281,7 +281,14 @@ class MambaHybridModelState(DefaultModelState):
# Test request state, not num_scheduled_tokens == draft_count+1:
# adaptive rewrites num_scheduled_tokens to an even split, so that
# equality rarely holds and would demote every verify row to decode.
is_decode = (~input_batch.is_prefilling_np) & (
# A one-token prompt tail over prior state that the scheduler padded
# with placeholder drafts is also a spec-decode row: the prefill
# kernels can't roll the placeholders back.
num_computed = input_batch.num_computed_prefill_tokens_np
is_prompt_tail = (num_computed > 0) & (
input_batch.prefill_len_np - num_computed == 1
)
is_decode = (~input_batch.is_prefilling_np | is_prompt_tail) & (
input_batch.num_scheduled_tokens > 0
)
spec_decode_mask = (num_draft_tokens_per_req > 0) & is_decode