diff --git a/docs/features/multimodal_inputs.md b/docs/features/multimodal_inputs.md index 5f684604e6..4656ee43ea 100644 --- a/docs/features/multimodal_inputs.md +++ b/docs/features/multimodal_inputs.md @@ -365,6 +365,8 @@ You must enable this feature via `enable_mm_embeds=True`. The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users! +#### Image Embeddings + ??? code ```python @@ -441,6 +443,36 @@ For Qwen2-VL and MiniCPM-V, we accept additional parameters alongside the embedd print(generated_text) ``` +#### Audio Embeddings + +You can pass pre-computed audio embeddings similar to image embeddings: + +??? code + + ```python + from vllm import LLM + import torch + + # Enable audio embeddings support + llm = LLM(model="fixie-ai/ultravox-v0_5-llama-3_2-1b", enable_mm_embeds=True) + + # Refer to the HuggingFace repo for the correct format to use + prompt = "USER: