[Misc] Reorganize inputs (#35182)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
This commit is contained in:
@@ -33,19 +33,20 @@ class MultiModalFeatures(BaseModel):
|
||||
"""Lightweight multimodal metadata produced by the render step.
|
||||
|
||||
Carries hashes (for cache lookup / identification) and placeholder
|
||||
positions so the downstream ``/generate`` service knows *where* in
|
||||
positions so the downstream `/generate` service knows *where* in
|
||||
the token sequence each multimodal item lives.
|
||||
|
||||
.. note:: Phase 1 — metadata only.
|
||||
Phase 2 should add ``mm_kwargs`` (processed tensor data) using a
|
||||
binary transport so the ``/generate`` side can skip re-processing.
|
||||
The ``/generate`` endpoint must also be updated to inject these
|
||||
features into ``ProcessorInputs`` before passing to
|
||||
``InputProcessor.process_inputs``.
|
||||
Note:
|
||||
Phase 1 — metadata only.
|
||||
Phase 2 should add `mm_kwargs` (processed tensor data) using a
|
||||
binary transport so the ``/generate` side can skip re-processing.
|
||||
The `/generate` endpoint must also be updated to inject these
|
||||
features into `EngineInput` before passing to
|
||||
`InputProcessor.process_inputs`.
|
||||
"""
|
||||
|
||||
mm_hashes: dict[str, list[str]]
|
||||
"""Per-modality item hashes, e.g. ``{"image": ["abc", "def"]}``."""
|
||||
"""Per-modality item hashes, e.g. `{"image": ["abc", "def"]}`."""
|
||||
|
||||
mm_placeholders: dict[str, list[PlaceholderRangeInfo]]
|
||||
"""Per-modality placeholder ranges in the token sequence."""
|
||||
|
||||
Reference in New Issue
Block a user