[Feat][Frontend] structured decisions: DiffusionGemma read strategy (#60466)
Signed-off-by: Matt Mastracci <matthew@mastracci.com> Signed-off-by: Simon Veitner <sveitner@redhat.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Simon Veitner <sveitner@redhat.com> Co-authored-by: Misha Goin <mgoin64@gmail.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
Simon Veitner
Misha Goin
parent
7d1d8660d6
commit
3c802af3cf
@@ -20,7 +20,10 @@ log probabilities.
|
||||
|
||||
Thinking is off unless `chat_template_kwargs` turns it on.
|
||||
|
||||
This endpoint does not serve diffusion models yet.
|
||||
Probabilities can differ between identical requests. They depend on which
|
||||
requests share a batch and on whether a read hits the prefix cache. Set
|
||||
`VLLM_BATCH_INVARIANT=1` for reads that repeat exactly, at a cost in
|
||||
throughput.
|
||||
|
||||
Every read of a request starts with the state, so with
|
||||
`--enable-prefix-caching` the state is prefilled once and each further
|
||||
|
||||
Reference in New Issue
Block a user