[Feat][Frontend] structured decisions: DiffusionGemma read strategy (#60466)

Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
This commit is contained in:
Matt Mastracci
2026-10-10 09:33:22 +08:00
committed by GitHub
co-authored by Claude Opus 5.5 Simon Veitner Misha Goin
parent 7d1d8660d6
commit 3c802af3cf
6 changed files with 193 additions and 40 deletions
@@ -20,7 +20,10 @@ log probabilities.
Thinking is off unless `chat_template_kwargs` turns it on.
This endpoint does not serve diffusion models yet.
Probabilities can differ between identical requests. They depend on which
requests share a batch and on whether a read hits the prefix cache. Set
`VLLM_BATCH_INVARIANT=1` for reads that repeat exactly, at a cost in
throughput.
Every read of a request starts with the state, so with
`--enable-prefix-caching` the state is prefilled once and each further