[Spec Decode] Enable Gemma4 DSpark adaptive verification with FlashInfer (#57263)
Signed-off-by: zixi-qi <zixi@inferact.ai> Co-authored-by: OpenAI Codex <noreply@openai.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: Nick Hill <nickhill123@gmail.com>
This commit is contained in:
co-authored by
OpenAI Codex
Claude Opus 5.5
Nick Hill
parent
3b4566c5cf
commit
a4eb3f25d6
@@ -69,7 +69,8 @@ class BatchExecutionDescriptor:
|
||||
num_reqs: int | None # None means no request padding is needed (PIECEWISE graphs)
|
||||
uniform_token_count: int | None = None
|
||||
# Upper bound on per-request query length. Varlen decode graphs leave
|
||||
# uniform_token_count unset, so this is what keeps a prefill batch out of one.
|
||||
# uniform_token_count unset, so this is what keeps a prefill batch out of one:
|
||||
# the runner passes None for any batch with a prefill.
|
||||
max_query_len: int | None = None
|
||||
num_active_loras: int = 0
|
||||
# Number of microbatches the batch is split into (DBO). 1 means no splitting.
|
||||
|
||||
Reference in New Issue
Block a user