[Bugfix][Frontend] Validate reused prompt token ids; render messages for media and echo (#55771)

Signed-off-by: Shijie Lyu <sjlyu@amazon.com>
Co-authored-by: Shijie Lyu <sjlyu@amazon.com>
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Shijie Lyu
2026-09-29 10:36:22 +08:00
committed by GitHub
co-authored by Shijie Lyu Claude
parent 0af34418e9
commit c6843d397c
6 changed files with 239 additions and 2 deletions
+2
View File
@@ -77,6 +77,8 @@ decode = client.chat.completions.create(
)
```
If `messages` has non-text content or `echo` is set, the ids are ignored and `messages` is rendered instead, so it must match the prefill request. Otherwise `kv_transfer_params["prompt_token_ids"]` must be a non-empty list of non-negative integers, or the request fails with HTTP 400, as it always does on `/v1/chat/completions/batch`.
## Development
We implement disaggregated prefilling by running 2 vLLM instances. One for prefill (we call it prefill instance) and one for decode (we call it decode instance), and then use a connector to transfer the prefill KV caches and results from prefill instance to decode instance.