[Model] Add Moondream3 model support(only query and caption skills) (#32325)

Signed-off-by: Dong Wang <dongw2019@gmail.com>
This commit is contained in:
Dong W
2026-05-01 10:06:48 +08:00
committed by GitHub
parent 14043dfecd
commit 7198940b39
19 changed files with 3238 additions and 9 deletions
+61
View File
@@ -215,6 +215,67 @@ When loading RGBA images (images with transparency), vLLM converts them to RGB f
- This setting only affects RGBA images with transparency; RGB images are unchanged
- If not specified, the default white background `(255, 255, 255)` is used for backward compatibility
#### Moondream3 Prompt Recipes { #moondream3-prompt-recipes }
`Moondream3ForCausalLM` supports two task-specific prompt formats:
- `query`: ask a question about the image.
- `caption`: generate a caption for the image.
```python
from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset
llm = LLM(
model="moondream/moondream3-preview",
tokenizer="moondream/starmie-v1",
trust_remote_code=True,
max_model_len=2048,
limit_mm_per_prompt={"image": 1},
)
image = ImageAsset("stop_sign").pil_image
def make_query_prompt(question: str) -> str:
return (
"<|endoftext|><image><|md_reserved_0|>query<|md_reserved_1|>"
f"{question}<|md_reserved_2|>"
)
def make_caption_prompt(length: str = "normal") -> str:
return (
"<|endoftext|><image><|md_reserved_0|>"
f"describe<|md_reserved_1|>{length}<|md_reserved_2|>"
)
query_out = llm.generate(
{
"prompt": make_query_prompt("What is shown in this image?"),
"multi_modal_data": {"image": image},
},
SamplingParams(max_tokens=64, temperature=0),
)[0].outputs[0].text
caption_out = llm.generate(
{
"prompt": make_caption_prompt(),
"multi_modal_data": {"image": image},
},
SamplingParams(max_tokens=100, temperature=0),
)[0].outputs[0].text
print("query:", query_out)
print("caption:", caption_out)
```
!!! note
The native Moondream3 model also has `detect` and `point` skills. Those
require custom coordinate decoding and are not exposed by this vLLM
implementation.
### Video Inputs
You can pass a list of NumPy arrays directly to the `'video'` field of the multi-modal dictionary
+7
View File
@@ -599,6 +599,7 @@ These models primarily accept the [`LLM.generate`](./generative_models.md#llmgen
| `Mistral3ForConditionalGeneration` | Mistral3 (HF Transformers) | T + I<sup>+</sup> | `mistralai/Mistral-Small-3.1-24B-Instruct-2503`, etc. | ✅︎ | ✅︎ |
| `MolmoForCausalLM` | Molmo | T + I<sup>+</sup> | `allenai/Molmo-7B-D-0924`, `allenai/Molmo-7B-O-0924`, etc. | ✅︎ | ✅︎ |
| `Molmo2ForConditionalGeneration` | Molmo2 | T + I<sup>+</sup> / V | `allenai/Molmo2-4B`, `allenai/Molmo2-8B`, `allenai/Molmo2-O-7B` | ✅︎ | ✅︎ |
| `Moondream3ForCausalLM` | Moondream3 | T + I | `moondream/moondream3-preview` | | ✅︎ |
| `MusicFlamingoForConditionalGeneration` | MusicFlamingo | T + A | `nvidia/music-flamingo-2601-hf`, `nvidia/music-flamingo-think-2601-hf` | ✅︎ | ✅︎ |
| `NVLM_D_Model` | NVLM-D 1.0 | T + I<sup>+</sup> | `nvidia/NVLM-D-72B`, etc. | | ✅︎ |
| `OpenCUAForConditionalGeneration` | OpenCUA-7B | T + I<sup>E+</sup> | `xlangai/OpenCUA-7B` | ✅︎ | ✅︎ |
@@ -661,6 +662,12 @@ Some models are supported only via the [Transformers modeling backend](#transfor
!!! note
For `InternVLChatModel`, only InternVL2.5 with Qwen2.5 text backbone (`OpenGVLab/InternVL2.5-1B` etc.), InternVL3 and InternVL3.5 have video inputs support currently.
!!! note
`Moondream3ForCausalLM` uses task-specific prompt templates for `query`
and `caption`. The native `detect` and `point` skills require custom
coordinate decoding and are not exposed by this vLLM implementation.
See [Moondream3 prompt recipes](../features/multimodal_inputs.md#moondream3-prompt-recipes).
!!! note
To use `TIGER-Lab/Mantis-8B-siglip-llama3`, you have to pass `--hf_overrides '{"architectures": ["MantisForConditionalGeneration"]}'` when running vLLM.