[MM] Further cleanup _apply_hf_processor_main (#53610)
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
This commit is contained in:
@@ -454,17 +454,17 @@ return a schema of the tensors outputted by the HF processor that are related to
|
||||
??? code
|
||||
|
||||
```python
|
||||
def _get_hf_processor_text(self, mm_counts: Mapping[str, int]) -> str:
|
||||
def _get_hf_mm_text(self, mm_counts: Mapping[str, int]) -> str:
|
||||
# Mistral3Processor requires text corresponding to the images
|
||||
return self.dummy_inputs.get_dummy_text(mm_counts)
|
||||
|
||||
def _postprocess_hf_mm_data(
|
||||
self,
|
||||
mm_data: Mapping[str, object],
|
||||
hf_processor_mm_kwargs: Mapping[str, object],
|
||||
hf_data: Mapping[str, object],
|
||||
hf_kwargs: Mapping[str, object],
|
||||
processed_data: BatchFeature,
|
||||
) -> BatchFeature:
|
||||
if not mm_data:
|
||||
if not hf_data:
|
||||
return processed_data
|
||||
|
||||
pixel_values = processed_data.get("pixel_values")
|
||||
@@ -486,25 +486,36 @@ return a schema of the tensors outputted by the HF processor that are related to
|
||||
calls the HF processor on the multi-modal data without passing any text.
|
||||
If the HF processor instead requires text corresponding to the multi-modal items,
|
||||
you should override
|
||||
[_get_hf_processor_text][vllm.multimodal.processing.BaseMultiModalProcessor._get_hf_processor_text]
|
||||
[_get_hf_mm_text][vllm.multimodal.processing.BaseMultiModalProcessor._get_hf_mm_text]
|
||||
to return the dummy text from
|
||||
[BaseDummyInputsBuilder.get_dummy_text][vllm.multimodal.processing.BaseDummyInputsBuilder.get_dummy_text]
|
||||
like in the example above. If the HF processor expects the multi-modal data
|
||||
under different keys than those provided by the multi-modal items
|
||||
(e.g. `audio` instead of `audios`), or requires additional keyword arguments
|
||||
under different keys than those provided by the multi-modal items,
|
||||
or requires additional keyword arguments
|
||||
(e.g. `sampling_rate`), you should override
|
||||
[_preprocess_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._preprocess_hf_mm_data].
|
||||
[_get_hf_mm_inputs][vllm.multimodal.processing.BaseMultiModalProcessor._get_hf_mm_inputs].
|
||||
If you need to modify the output of the HF processor,
|
||||
you should override
|
||||
[_postprocess_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._postprocess_hf_mm_data].
|
||||
Passthrough fields have already been merged into `processed_data` when this
|
||||
hook is called.
|
||||
If you need to change how the normalized inputs are passed to the HF
|
||||
processor (for example, to call a method other than `__call__`, or to use
|
||||
different kwargs to construct and invoke it), override
|
||||
[_call_hf_processor][vllm.multimodal.processing.BaseMultiModalProcessor._call_hf_processor].
|
||||
If the HF processor applies additional transformations to the prompt tokens
|
||||
that are not reflected in its multi-modal outputs (e.g. prepending a BOS token),
|
||||
you should override
|
||||
[_postprocess_prompt][vllm.multimodal.processing.BaseMultiModalProcessor._postprocess_prompt]
|
||||
to replicate them.
|
||||
For even more control over how the HF processor is called, you can override
|
||||
Only processors that require multiple HF calls or manually construct the
|
||||
complete output should override
|
||||
[_apply_hf_processor_main][vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_main]
|
||||
directly.
|
||||
directly. Such implementations must obtain their normalized inputs through
|
||||
[_get_hf_mm_inputs][vllm.multimodal.processing.BaseMultiModalProcessor._get_hf_mm_inputs]
|
||||
and return through
|
||||
[_finalize_hf_mm_data][vllm.multimodal.processing.BaseMultiModalProcessor._finalize_hf_mm_data]
|
||||
so passthrough fields and model-specific postprocessing are applied consistently.
|
||||
|
||||
Since `pixel_values` is now a list with one tensor per image, we can override
|
||||
[_get_mm_fields_config][vllm.multimodal.processing.BaseMultiModalProcessor._get_mm_fields_config] as follows:
|
||||
@@ -724,7 +735,7 @@ Examples:
|
||||
|
||||
### Custom HF processor
|
||||
|
||||
Some models don't define an HF processor class on HF Hub. In that case, you can define a custom HF processor that has the same call signature as HF processors and return it from [BaseProcessingInfo.get_hf_processor][vllm.multimodal.processing.BaseProcessingInfo.get_hf_processor]. It is then applied to the multi-modal data via [InputProcessingContext.call_hf_processor][vllm.multimodal.processing.InputProcessingContext.call_hf_processor] inside [_apply_hf_processor_main][vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_main].
|
||||
Some models don't define an HF processor class on HF Hub. In that case, you can define a custom HF processor that has the same call signature as HF processors and return it from [BaseProcessingInfo.get_hf_processor][vllm.multimodal.processing.BaseProcessingInfo.get_hf_processor]. The default [_call_hf_processor][vllm.multimodal.processing.BaseMultiModalProcessor._call_hf_processor] applies it to the normalized multi-modal data via [InputProcessingContext.call_hf_processor][vllm.multimodal.processing.InputProcessingContext.call_hf_processor]. Override `_call_hf_processor` if the custom processor instead exposes a different callable interface.
|
||||
|
||||
Examples:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user