[Frontend] Expose multimodal metadata for disaggregated prefill (#54659)
Signed-off-by: zhouyou9505 <zhouyou9505@gmail.com> Co-authored-by: Cursor Grok 4.6 <noreply@cursor.com>
This commit is contained in:
co-authored by
Cursor Grok 4.6
parent
808f8cd3ac
commit
52358e6e19
@@ -27,3 +27,147 @@ VLLM_ENABLE_SCALE_OUT_ENDPOINTS=1 vllm serve <model>
|
||||
- Render chat completions
|
||||
|
||||
For the post processing counterpart that turns generated token IDs back into OpenAI compatible responses, see the [Derenderer APIs](derenderer.md).
|
||||
|
||||
## Multimodal Render Features
|
||||
|
||||
Multimodal render responses include a `features` object with per-modality
|
||||
hashes, placeholder ranges, and serialized processor data. When the model
|
||||
exposes placeholder-metadata or `keep_on_cpu` fields (for example
|
||||
`image_grid_thw`), the response also includes `mm_metadata`. Each
|
||||
`mm_metadata` entry is a base64-encoded `MultiModalKwargsItem` containing
|
||||
only those fields, not encoder inputs such as `pixel_values`.
|
||||
|
||||
The arrays in `mm_hashes`, `mm_placeholders`, `kwargs_data`, and
|
||||
`mm_metadata` use the same per-modality item order. Downstream workers
|
||||
should split those fields:
|
||||
|
||||
- Encode requests keep `kwargs_data`.
|
||||
- Prefill requests may omit `kwargs_data` and send `mm_metadata` only when
|
||||
`ec_transfer_params` is also set, so embeddings are loaded by the EC
|
||||
connector. Omitting `kwargs_data` without `ec_transfer_params` is rejected.
|
||||
- Legacy clients that ignore `mm_metadata` and keep sending `kwargs_data`
|
||||
continue to work.
|
||||
|
||||
## Example
|
||||
|
||||
The example below shows how a disaggregated encode / prefill coordinator can
|
||||
split a multimodal render response. The render step returns both
|
||||
`kwargs_data` (encoder tensors plus metadata) and `mm_metadata` (metadata
|
||||
only). Encode keeps the full payload; prefill drops `kwargs_data` after the
|
||||
EC connector has published embeddings.
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
MODEL = "Qwen/Qwen3-VL-2B-Instruct"
|
||||
RENDER = "http://localhost:8100" # vllm launch render ...
|
||||
ENCODE = "http://localhost:8200" # encode worker
|
||||
PREFILL = "http://localhost:8300" # prefill worker
|
||||
|
||||
chat_request = {
|
||||
"model": MODEL,
|
||||
"messages": [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {"url": "<data-url>"}},
|
||||
{"type": "text", "text": "Describe this image."},
|
||||
],
|
||||
}
|
||||
],
|
||||
}
|
||||
|
||||
with httpx.Client(timeout=120.0) as client:
|
||||
# 1. Render: preprocess into token IDs and multimodal features.
|
||||
render_response = client.post(
|
||||
f"{RENDER}/v1/chat/completions/render", json=chat_request
|
||||
).json()
|
||||
|
||||
features = render_response["features"]
|
||||
# features["kwargs_data"]["image"][0] -> pixel_values + image_grid_thw
|
||||
# features["mm_metadata"]["image"][0] -> image_grid_thw only
|
||||
|
||||
# 2. Encode: send full kwargs_data so the encoder can run vision towers.
|
||||
encode_response = client.post(
|
||||
f"{ENCODE}/inference/v1/generate",
|
||||
json={
|
||||
"token_ids": render_response["token_ids"],
|
||||
"features": {
|
||||
"mm_hashes": features["mm_hashes"],
|
||||
"mm_placeholders": features["mm_placeholders"],
|
||||
"kwargs_data": features["kwargs_data"],
|
||||
},
|
||||
"sampling_params": {"max_tokens": 1},
|
||||
},
|
||||
).json()
|
||||
ec_transfer_params = encode_response["ec_transfer_params"]
|
||||
|
||||
# 3. Prefill: omit kwargs_data; load embeddings via EC connector.
|
||||
prefill_response = client.post(
|
||||
f"{PREFILL}/inference/v1/generate",
|
||||
json={
|
||||
"token_ids": render_response["token_ids"],
|
||||
"features": {
|
||||
"mm_hashes": features["mm_hashes"],
|
||||
"mm_placeholders": features["mm_placeholders"],
|
||||
"mm_metadata": features["mm_metadata"],
|
||||
},
|
||||
"ec_transfer_params": ec_transfer_params,
|
||||
"sampling_params": {"max_tokens": 64},
|
||||
},
|
||||
).json()
|
||||
|
||||
print(prefill_response["choices"][0]["token_ids"])
|
||||
```
|
||||
|
||||
Single-process clients can keep passing the full render response to
|
||||
`/inference/v1/generate` unchanged; `mm_metadata` is optional and ignored
|
||||
when `kwargs_data` is present.
|
||||
|
||||
### Payload shape
|
||||
|
||||
#### Render response
|
||||
|
||||
`/v1/chat/completions/render` returns both `kwargs_data` and `mm_metadata`.
|
||||
The arrays share the same per-modality item order. Base64 blobs are truncated
|
||||
below for readability.
|
||||
|
||||
```json
|
||||
{
|
||||
"token_ids": [151644, 872],
|
||||
"features": {
|
||||
"mm_hashes": {"image": ["abc123..."]},
|
||||
"mm_placeholders": {"image": [{"offset": 0, "length": 256}]},
|
||||
"kwargs_data": {
|
||||
"image": ["<base64 MultiModalKwargsItem: pixel_values + image_grid_thw>"]
|
||||
},
|
||||
"mm_metadata": {
|
||||
"image": ["<base64 MultiModalKwargsItem: image_grid_thw only>"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Forward `kwargs_data` to the encode worker. Keep `mm_metadata` for prefill.
|
||||
|
||||
#### Prefill request
|
||||
|
||||
Prefill omits `kwargs_data` and sends `mm_metadata` with
|
||||
`ec_transfer_params` from the encode response:
|
||||
|
||||
```json
|
||||
{
|
||||
"token_ids": [151644, 872],
|
||||
"features": {
|
||||
"mm_hashes": {"image": ["abc123..."]},
|
||||
"mm_placeholders": {"image": [{"offset": 0, "length": 256}]},
|
||||
"mm_metadata": {
|
||||
"image": ["<base64 MultiModalKwargsItem: image_grid_thw only>"]
|
||||
}
|
||||
},
|
||||
"ec_transfer_params": {
|
||||
"ec_items": [{"mm_hash": "abc123...", "peer_host": "10.0.0.1"}]
|
||||
},
|
||||
"sampling_params": {"max_tokens": 64}
|
||||
}
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user